Data Preprocessing Python Packages

skrub

Machine learning with dataframes

split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

tab2seq

Transform tabular event data into sequences ready for Transformer and Sequential models: Life2Vec, BEHRT and more.

desbordante

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

freshdata-cleaner

Automated, explainable DataFrame cleaning for Python and pandas — smart missing-value handling, outlier detection, duplicate resolution, and AI-ready preprocessing in one call.

pyhelpers

PyHelpers: An open-source toolkit for facilitating Python users' data manipulation tasks.

ml-express

A Python library for day to day data analysis and machine learning. This aims to make data building, cleaning and machine learning much much faster. A library of extension and helper modules for Python's data analysis and machine learning libraries.

lucifer-ml

Semi-Auto Machine Learning Library by d4rk-lucif3r

melpy

A NumPy-based deep learning library for building neural networks. It features an automatic differentiation engine and supports training architectures like LSTM, CNN, and FNN.

clearbox-preprocessor

A fast and felxible data preprocessor based on polars.

loren-frank-data-processing

Python tools for reading in data from Loren Frank's lab

data-preprocessors

🛠️An easy to use tool for Data Preprocessing specially for Text Preprocessing

mzutils

Mohan Zhang's toolkit

nutsml

Flow-based data pre-processing for deep learning

wiz-craft

A CLI-based dataset preprocessing tool for machine learning tasks. Features include data exploration, null value handling, one-hot encoding, and feature scaling, and download the modified dataset effortlessly.

hdsemg-select

hdsemg-select package

duplipy

DupliPy is a quick and easy-to-use package that can handle text formatting and data augmentation tasks for NLP in Python. It now offers support for image augmentation tasks as well.

deocr

A high-performance highly-customizable reverse OCR tool that renders text or huggingface-compatible datasets to images. Dimension, DPI, CSS configurable!

atlantic

Atlantic: Automated Data Preprocessing Framework for Machine Learning

data-purifier

A Python library for Automated Exploratory Data Analysis, Automated Data Cleaning, and Automated Data Preprocessing For Machine Learning and Natural Language Processing Applications in Python.

dptools

Data Preprocessing Tools

ptrail

PTRAIL is a state-of-the art parallel computation library for Mobility Data Preprocessing and feature extraction.

phaeton

A high-performance preprocessing and ETL engine python library for sanitizing raw data streams, accelerated by Rust.

biosets

A bioinformatics extension of 🤗 Datasets library, built for ML applications on biological and omics data, offering easy integration of metadata and low-code data management tools.