Data Preprocessing Python Packages

skrub

Machine learning with dataframes

split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

desbordante

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

pyhelpers

PyHelpers: An open-source toolkit for facilitating Python users' data manipulation tasks.

lucifer-ml

Semi-Auto Machine Learning Library by d4rk-lucif3r

clearbox-preprocessor

A fast and felxible data preprocessor based on polars.

melpy

A NumPy-based deep learning library for building neural networks. It features an automatic differentiation engine and supports training architectures like LSTM, CNN, and FNN.

mzutils

Mohan Zhang's toolkit

loren-frank-data-processing

Python tools for reading in data from Loren Frank's lab

data-preprocessors

🛠️An easy to use tool for Data Preprocessing specially for Text Preprocessing

gators

Lightning-fast data preprocessing and feature engineering for Python, built on Polars. 108 sklearn-style transformers for imputation, encoding, scaling, outlier clipping, discretization and feature selection, with Pipeline support and ONNX export for low-latency inference.

freshdata-cleaner

Automated, explainable data cleaning for pandas and Polars, repair messy tabular data safely and see exactly what changed.

nutsml

Flow-based data pre-processing for deep learning

autoprepml

AutoPrepML - Multi-modal Python preprocessing and EDA toolkit for tabular, text, time-series, graph, and image data. CLI and Python API, PyPI package, configurable workflows, and cross-platform CI.

duplipy

DupliPy is a quick and easy-to-use package that can handle text formatting and data augmentation tasks for NLP in Python. It now offers support for image augmentation tasks as well.

deocr

A high-performance highly-customizable reverse OCR tool that renders text or huggingface-compatible datasets to images. Dimension, DPI, CSS configurable!

data-purifier

A Python library for Automated Exploratory Data Analysis, Automated Data Cleaning, and Automated Data Preprocessing For Machine Learning and Natural Language Processing Applications in Python.

hdsemg-select

A graphical user interface (GUI) application for selecting and analyzing HDsEMG channels. This tool helps identify and exclude faulty channels (e.g., due to electrode misplacement, corrosion or noisy channels) from HDsEMG recordings.

atlantic

Atlantic: Automated Data Preprocessing Framework for Machine Learning

phaeton

A high-performance preprocessing and ETL engine python library for sanitizing raw data streams, accelerated by Rust.

nonechucks

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

ptrail

PTRAIL is a state-of-the art parallel computation library for Mobility Data Preprocessing and feature extraction.

tab2seq

Transform tabular event data into sequences ready for Transformer and Sequential models: Life2Vec, BEHRT and more.

dptools

Python package with utilities for data processing, aggregation, feature engineering and data versioning