Data Preparation Python Packages

skrub

Machine learning with dataframes

tracebloc-ingestor

tracebloc data pipeline for training/test dataset setup

optimuspyspark

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

jupyterlab-amphi

visual data prep powered by python

aw

AI Agentic Workflows for data preparation and transformation

ml-express

A Python library for day to day data analysis and machine learning. This aims to make data building, cleaning and machine learning much much faster. A library of extension and helper modules for Python's data analysis and machine learning libraries.

amphi-scheduler

Amphi Scheduler (JupyterLab extension + Python backend)

convmerge

Convert Alpaca / ShareGPT datasets to unified messages JSONL for LLM supervised fine-tuning — fetch from HuggingFace or GitHub, normalize, convert, mix, and dedupe in one CLI pipeline

hdsemg-select

hdsemg-select package

refinedoc

Python library for extracting headers, footers and body from PDF

datarec-lib

A Python Library for Standardized and Reproducible Data Management in Recommender Systems

pyoptimus

No description available

label-maker

Data preparation for satellite machine learning

dptools

Data Preprocessing Tools

machine-learning-data-pipeline

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

dataclr

Feature selection for tabular datasets using advanced filter and wrapper methods

forestiler

Create Image Tiles From Large Input Rasters According to a Classified Mask Vector File

datamind

All-in-one intelligent assistant powered by LlamaIndex — RAG, GraphRAG, NL2SQL, Skills & Memory with multimodal support.

datarec

A Python Library for Standardized and Reproducible Data Management in Recommender Systems

invisible-rabbit

Scalable data pre processing and curation toolkit for LLMs

prosto

Data processing toolkit radically changing the way data is processed

spltr

`Spltr` is a simple PyTorch-based data loader and splitter. It may be used to load arrays and matrices or Pandas DataFrames and CSV files containing numerical data with subsequent split it into train, test (validation) subsets in the form of PyTorch DataLoader objects.

daxpy

A pre-machine-learning model package

invisible-unicorn

Scalable Data Preprocessing Tool for Training Large Language Models