Data Preparation Python Packages

skrub

Machine learning with dataframes

optimuspyspark

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

jupyterlab-amphi

visual data prep powered by python

convmerge

Merge heterogeneous chat/text sources into a single LLM training format (JSONL)

datarec-lib

A Python Library for Standardized and Reproducible Data Management in Recommender Systems

hdsemg-select

A graphical user interface (GUI) application for selecting and analyzing HDsEMG channels. This tool helps identify and exclude faulty channels (e.g., due to electrode misplacement, corrosion or noisy channels) from HDsEMG recordings.

tracebloc-ingestor

tracebloc data pipeline for training/test dataset setup

amphi-scheduler

Amphi Scheduler (JupyterLab extension + Python backend)

datarec

Standardized & reproducible data management for recommender systems.

pyoptimus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

refinedoc

Python library for extracting headers, footers and body from PDF

dptools

Data Preprocessing Tools

forestiler

Create Image Tiles From Large Input Rasters According to a Classified Mask Vector File

label-maker

Data Preparation for Satellite Machine Learning

machine-learning-data-pipeline

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

dataclr

Feature selection for tabular datasets using advanced filter and wrapper methods

ml-express

A Python library for day to day data analysis and machine learning. This aims to make data building, cleaning and machine learning much much faster. A library of extension and helper modules for Python's data analysis and machine learning libraries.

prosto

Data processing toolkit radically changing the way data is processed

invisible-rabbit

Scalable data pre processing and curation toolkit for LLMs

spltr

`Spltr` is a simple PyTorch-based data loader and splitter. It may be used to load arrays and matrices or Pandas DataFrames and CSV files containing numerical data with subsequent split it into train, test (validation) subsets in the form of PyTorch DataLoader objects.

daxpy

A pre-machine-learning model package

invisible-unicorn

Scalable data pre processing and curation toolkit for LLMs

lava-ray

Scalable data pre processing and curation toolkit for LLMs