Data Preprocessing Python Packages

skrub

Machine learning with dataframes

desbordante

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

mzutils

Mohan Zhang's toolkit

loren-frank-data-processing

Python tools for reading in data from Loren Frank's lab

lucifer-ml

Semi-Auto Machine Learning Library by d4rk-lucif3r

melpy

A NumPy-based deep learning library for building neural networks. It features an automatic differentiation engine and supports training architectures like LSTM, CNN, and FNN.

pyhelpers

PyHelpers: An open-source toolkit for facilitating Python users' data manipulation tasks.

clearbox-preprocessor

A fast polars based data pre-processor for ML datasets

hdsemg-select

A graphical user interface (GUI) application for selecting and analyzing HDsEMG channels. This tool helps identify and exclude faulty channels (e.g., due to electrode misplacement, corrosion or noisy channels) from HDsEMG recordings.

data-preprocessors

🛠️An easy to use tool for Data Preprocessing specially for Text Preprocessing

nutsml

Flow-based data pre-processing for Machine Learning

atlantic

Atlantic: Automated Data Preprocessing Framework for Machine Learning

deocr

A high-performance highly-customizable reverse OCR tool that renders text or huggingface-compatible datasets to images. Dimension, DPI, CSS configurable!

duplipy

DupliPy is a quick and easy-to-use package that can handle text formatting and data augmentation tasks for NLP in Python. It now offers support for image augmentation tasks as well.

data-purifier

A Python library for Automated Exploratory Data Analysis, Automated Data Cleaning and Automated Data Preprocessing For Machine Learning and Natural Language Processing Applications in Python.

vision-converter

VisionConverter is a Python library designed to convert object detection dataset annotations between the most widely used formats in computer vision. Allows developers to do conversions both from the command line and directly in Python code.

nonechucks

nonechucks is a library that provides wrappers for PyTorch's datasets, samplers, and transforms to allow for dropping unwanted or invalid samples dynamically.

ptrail

PTRAIL: A Mobility-data Preprocessing Library using parallel computation.

biosets

A bioinformatics extension of 🤗 Datasets library, built for ML applications on biological and omics data, offering easy integration of metadata and low-code data management tools.

dptools

Data Preprocessing Tools

fifa-preprocessing

Python package with tools to preprocess data, made for data analysis.

data-modori

LMOps Tool for Korean

machine-learning-data-pipeline

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.