Data Preparation Python Packages

skrub

Machine learning with dataframes

convmerge

Merge heterogeneous chat/text sources into a single LLM training format (JSONL)

optimuspyspark

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

datamind

All-in-one intelligent assistant powered by LlamaIndex — RAG, GraphRAG, NL2SQL, Skills & Memory with multimodal support.

jupyterlab-amphi

visual data prep powered by python

refinedoc

Python library for extracting headers, footers and body from PDF

focal

Data Access Objects for local files

eazydatafix

Deterministic-first Python framework for Agentic EDA, data quality, validation, cleaning, and reproducible reporting—no LLM required.

ciaren

Open-source, local-first visual ETL and data pipeline builder. Drag-and-drop data prep and ML workflows, preview every step, schedule runs, and export clean pandas or Polars Python code. Self-hosted, no lock-in.

ciaren-client

Open-source, local-first visual ETL and data pipeline builder. Drag-and-drop data prep and ML workflows, preview every step, schedule runs, and export clean pandas or Polars Python code. Self-hosted, no lock-in.

amphi-scheduler

visual data prep powered by python

aw

AI Agentic Workflows for data preparation and transformation

pyoptimus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

dataclean-csv

Dependency-free Python CLI that cleans messy CSV files — normalizes headers, removes duplicates, fills missing values — and writes a JSON data-quality report. Stdlib only, no pandas.

tokbin-core

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

tokbin

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

hdsemg-select

A graphical user interface (GUI) application for selecting and analyzing HDsEMG channels. This tool helps identify and exclude faulty channels (e.g., due to electrode misplacement, corrosion or noisy channels) from HDsEMG recordings.

datarec-lib

A Python Library for Standardized and Reproducible Data Management in Recommender Systems

label-maker

Data Preparation for Satellite Machine Learning

easydatafix

Deterministic-first Python framework for Agentic EDA, data quality, validation, cleaning, and reproducible reporting—no LLM required.

dptools

Python package with utilities for data processing, aggregation, feature engineering and data versioning

forestiler

Create Image Tiles From Large Input Rasters According to a Classified Mask Vector File

ml-express

A Python library for day to day data analysis and machine learning. This aims to make data building, cleaning and machine learning much much faster. A library of extension and helper modules for Python's data analysis and machine learning libraries.

invisible-rabbit

Scalable data pre processing and curation toolkit for LLMs