Text Cleaning Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

10.5M 6K 392

harvesttext

文本挖掘和预处理工具（文本清洗、新词发现、情感分析、实体识别链接、关键词抽取、知识抽取、句法分析等），无监督或弱监督方法

3K 3K 340

pnlp

NLP预/后处理工具。

2K 30 6

squeakycleantext

Text preprocessing and PII anonymisation for NLP/ML. ONNX NER ensemble, language detection, stopword removal. Built for statistical ML and language models.

2K 8 0

nlp-preprocessing

Text Preprocessing Package includes cleaning, tokenization, dataset preparation ...etc

2K 18 7

extractnet

A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package

879 298 26

araclean

Offset-preserving Arabic text normalization & cleaning for NLP — map cleaned spans back to the original. Pure-Python, typed, reproducible profiles.

830 1 0

asciilint

ASCII and character policy checks for text files

742 0 0

textlasso

TextLasso is a Simple Python library for extracting structured data from raw text, with special focus on processing LLM (Large Language Model) responses.

672 4 0

mim-nlp

A Python package with ready-to-use models for various NLP tasks and text preprocessing utilities. The implementation allows fine-tuning.

644 2 0

farsflow

Deterministic, dependency-free, and semantic-safe Persian text preprocessing library

538 0 0

hnlp

NLP预/后处理工具。

538 30 6

wisup-e2m

E2M converts various file types (doc, docx, epub, html, htm, url, pdf, ppt, pptx, mp3, m4a) into Markdown. It’s easy to install, with dedicated parsers and converters, supporting custom configs. E2M offers an all-in-one, flexible, and open-source solution.

527 1K 73

valx

An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.

479 5 1