Preprocessing Python Packages

unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

jaconv

Pure-Python Japanese character interconverter for Hiragana, Katakana, Hankaku, and Zenkaku

nnaudio

Audio processing by using pytorch 1D convolution network

semhash

Fast Multimodal Semantic Deduplication & Filtering

neologdn

Japanese text normalizer for mecab-neologd

pyprep

PyPREP: A Python implementation of the Preprocessing Pipeline (PREP) for EEG data

indic-num2words

Python library for converting numbers to words for all Indian Languages.

nums-from-string

Extract numbers from a string

df2onehot

Convert a unstructured array into a stuctured dataframe.

autoreject

Automated rejection and repair of bad trials/sensors in M/EEG

autots

Automated Time Series Forecasting

contextualspellcheck

✔️Contextual word checker for better suggestions (not actively maintained)

kditransform

Kernel density integral transformation: feature preprocessing and univariate clustering (TMLR, 2023)

jionlp

中文 NLP 预处理、解析工具包，准确、高效、易用 A Chinese NLP Preprocessing & Parsing Package www.jionlp.com

eegprep

EEGPrep is an automated preprocessing tool for human EEG data built on a benchmarked EEGLAB pipeline

smashed

SMASHED is a toolkit designed to apply transformations to samples in datasets, such as fields extraction, tokenization, prompting, batching, and more. Supports datasets from Huggingface, torchdata iterables, or simple lists of dictionaries.

nvtabular

NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.

pyhealth

A Deep Learning Python Toolkit for Healthcare Applications.

logprep

log data pre processing, generation and shipping in python

xmip

Analysis ready CMIP6 data in python the easy way with pangeo tools.

maldiamrkit

Comprehensive toolkit for MALDI-TOF mass spectrometry data preprocessing for antimicrobial resistance (AMR) prediction purposes

sk-transformers

A collection of pandas & scikit-learn compatible transformers for preprocessing and feature engineering 🛠

sktutor

sktutor helps your machines learn

pnlp

NLP预/后处理工具。