Dataloader Python Packages

conecta

SQL to Arrow made easy.

9K 8 0

pyroboframes

Fast ML dataloader for robot learning. LeRobot datasets, hardware video decode, multi-output formats (NumPy/MLX/PyTorch/JAX).

7K 0 0

cellmap-data

Utility for loading CellMap data for machine learning training, utilizing PyTorch, TensorStore, Xarray, and PyDantic.

5K 8 3

datago

A natively parallel dataloader for Python, written in Rust. Serving data at GB/s speeds, while covering aspect ratio bucketing, crop and resize for image ML workloads.

3K 129 11

omniload

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

2K 0 2

jax-dataloader

Pytorch-like dataloaders for JAX.

2K 104 4

scdataloader

a dataloader to work with large single cell datasets from lamindb

2K 40 7

continuum

A clean and simple data loading library for Continual Learning

2K 448 44

lavender-data

Load & evolve datasets efficiently

1K 22 0

online-dynamic-batching

Online dynamic batching for faster LLM/VLM fine-tuning

1K 1 0

fewsoldataloader

Pytorch Dataloader for FewSOL

786 2 0

bruker

A modular toolkit for Bruker MRI raw-data handling.

687 56 30

torchvtk

PyTorch volume toolkit. Efficient data loading, dataset conversions, visualization tools

655 10 2

dynabatch

PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.

653 1 0

vdataset

Load video datasets to PyTorch DataLoader. (Custom Video Data set for PyTorch DataLoader)

579 0 0

torch-dataloader-utils

Efficient Data Loader Utils for loading data from structured sources into Pytorch

509 0 0

torch-waymo

PyTorch dataloader for Waymo Open Dataset

476 19 4

audiotree

Audio data loading and augmentations in JAX

457 7 1

datatap

Client library for dataTap

452 34 3

bucketsampler

Aspect ratio bucketing toolkit for diffusion model training. PyTorch native, DDP correct.

437 2 0

ukv

Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️

417 635 36

ffcv-pl

[FFCV-PL] manage fast data loading with ffcv and pytorch lightning

365 16 0

dloader

DataLoader implementation for Python asyncio

314 0 0

light-dataloader

Faster DataLoader for datasets that are fully loaded into memory as tensors.

303 3 0