Text Cleaning Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

squeakycleantext

Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.

harvesttext

文本挖掘和预处理工具（文本清洗、新词发现、情感分析、实体识别链接、关键词抽取、知识抽取、句法分析等），无监督或弱监督方法

pnlp

NLP预/后处理工具。

extractnet

A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package

nlp-preprocessing

Text Preprocessing Package includes cleaning, tokenization, dataset preparation ...etc

textlasso

Simple packege for working with LLM text responses and prompts.

wisup-e2m

E2M converts various file types (doc, docx, epub, html, htm, url, pdf, ppt, pptx, mp3, m4a) into Markdown. It’s easy to install, with dedicated parsers and converters, supporting custom configs. E2M offers an all-in-one, flexible, and open-source solution.

valx

An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.

mim-nlp

A Python package with ready-to-use models for various NLP tasks and text preprocessing utilities. The implementation allows fine-tuning.

hnlp

NLP预/后处理工具。

textcl

Text preprocessing package for use in NLP tasks https://pypi.org/project/textcl/

topicrankpy

A Python package to get useful information from documents using TopicRank Algorithm.

rehuman

Python bindings for rehuman: Unicode-safe text cleaning & normalization

textprepro

👀 Everything Everyway All At Once Text Preprocessing for Natural Language Processing.

banglish-stopwords

A high-performance library to filter Banglish stopwords from text.

text-cleaner-yuvan

A lightweight Python package for cleaning, normalizing, and tokenizing text data.

takin

A Python toolkit for file processing, text cleaning and data splitting. 文件处理，文本清洗和数据划分的python工具包。

ternaus-cleantext

Cleans text as in the CLIP model