Data Loading Python Packages

dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

datarax

Differentiable data pipelines for JAX/Flax NNX: HuggingFace, TFDS and ArrayRecord sources, augmentation stages, scan-based epochs, exact mid-epoch resume.

blobpack

Pack dataset media into plain STORED zip shards; read them fast anywhere.

dlt-core

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

omniload

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

dlt-cratedb

dlt destination adapter for CrateDB

dlt-filesystem

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

pytest-data-loader

Pytest plugin for loading test data for data-driven testing (DDT)

behave-kit

The Swiss-army knife for Behave — soft assertions, typed context, conditional skip, environment management, tag-based fixtures, data loading, context dump, and step suggestions. Adopt incrementally with 3 levels of integration.

minimal-dataset-pytorch

Lightweight experimental PyTorch data-loading library with Parquet-backed datasets, concurrent loading, benchmarking and performance instrumentation.

dlt-source-personio

DLT (www.github.com/dlt-hub/dlt) source for Personio (www.personio.com)

m2data

A Python package for working with GEC data in .m2 files

datavolt

Reusable data engineering toolkit My personal data infrastructure

spltr

`Spltr` is a simple PyTorch-based data loader and splitter. It may be used to load arrays and matrices or Pandas DataFrames and CSV files containing numerical data with subsequent split it into train, test (validation) subsets in the form of PyTorch DataLoader objects.

fastload-data

Efficient chunked loader for large CSV and Excel files

podium-nlp

Podium: a framework agnostic Python NLP library for data loading and preprocessing

bp-storage

Library to load various vision datasets from disk

dlt-dataops

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️