Dataloader Python Packages

datarax

Differentiable data pipelines for JAX/Flax NNX: HuggingFace, TFDS and ArrayRecord sources, augmentation stages, scan-based epochs, exact mid-epoch resume.

conecta

SQL to Arrow made easy.

jax-dataloader

Pytorch-like dataloaders for JAX.

db-graphql-gateway

Automatically generate a secure, production-ready GraphQL API from a database connection.

continuum

A clean and simple data loading library for Continual Learning

omniload

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

datago

A natively parallel dataloader for Python, written in Rust. Serving data at GB/s speeds, while covering aspect ratio bucketing, crop and resize for image ML workloads.

cellmap-data

Utility for loading CellMap data for machine learning training, utilizing PyTorch, TensorStore, Xarray, and PyDantic.

dlt-filesystem

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

scdataloader

a dataloader to work with large single cell datasets from lamindb

mcap-data-loader

Load and process MCAP data files for machine learning and robotics training pipelines, with a LeRobot training integration.

fewsoldataloader

This repo hosts the PyTorch dataloader for FewSOL dataset | ICRA 2023

lavender-data

Load & manage evolving datasets efficiently

bruker

A modular toolkit for Bruker MRI raw-data handling.

pyroboframes

High-performance ML dataloader for robotics. LeRobot format, hardware-accelerated video decode, MLX arrays. Zero-copy pipeline for Apple Silicon.

torch-waymo

PyTorch dataloader for Waymo Open Dataset

audiotree

Audio data loading and augmentations in JAX

datatap

Focus on Algorithm Design, Not on Data Wrangling

foldpipe

Bounded-memory asynchronous streaming for native PyTorch/PyG molecular-data shards, with remote prefetch from Hugging Face and Google Drive.

ukv

Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️

hydrax

A zero-copy multiprocess dataloader for Jax.

bucketsampler

Aspect ratio bucketing for diffusion training. PyTorch native, DDP correct, no training-framework lock-in.

dloader

A Python implementation of the DataLoader pattern for data fetching with automatic batching.

asrad-reader

a tool aim to read the station data from ASRAD