Webdataset Python Packages

webdataset

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

wids

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

blobpack

Pack dataset media into plain STORED zip shards; read them fast anywhere.

webdatasetng

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

tarproc

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

chug

Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.

webdataset-latch

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.