Preprocessing Python Packages

unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

jaconv

Pure-Python Japanese character interconverter for Hiragana, Katakana, Hankaku, and Zenkaku

nnaudio

Audio processing by using pytorch 1D convolution network

semhash

Fast Multimodal Semantic Deduplication & Filtering

pyprep

PyPREP: A Python implementation of the Preprocessing Pipeline (PREP) for EEG data

neologdn

Japanese text normalizer for mecab-neologd

autoreject

Automated rejection and repair of bad trials/sensors in M/EEG

pyhealth

A Deep Learning Python Toolkit for Healthcare Applications.

df2onehot

Convert a unstructured array into a stuctured dataframe.

indic-num2words

Python library for converting numbers to words for all Indian Languages.

autots

Automated Time Series Forecasting

contextualspellcheck

✔️Contextual word checker for better suggestions (not actively maintained)

nums-from-string

Extract numbers from a string

eegprep

EEGPrep is an automated preprocessing tool for human EEG data built on a benchmarked EEGLAB pipeline

nvtabular

NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.

kditransform

Kernel density integral transformation: feature preprocessing and univariate clustering (TMLR, 2023)

jionlp

中文 NLP 预处理、解析工具包，准确、高效、易用 A Chinese NLP Preprocessing & Parsing Package www.jionlp.com

logprep

log data pre processing, generation and shipping in python

smashed

SMASHED is a toolkit designed to apply transformations to samples in datasets, such as fields extraction, tokenization, prompting, batching, and more. Supports datasets from Huggingface, torchdata iterables, or simple lists of dictionaries.

dsr-feature-eng-ml

Machine learning model evaluation and feature engineering framework with hyperparameter tuning, data balancing, and feature importance analysis.

maldiamrkit

Comprehensive toolkit for MALDI-TOF mass spectrometry data preprocessing for antimicrobial resistance (AMR) prediction purposes

xmip

Analysis ready CMIP6 data in python the easy way with pangeo tools.

seqtools

A library for transparent transformation of indexable containers (lists, etc.)

sktutor

sktutor helps your machines learn