PyRank

Memmap Python Packages

Python packages with the GitHub topic memmap. Sorted by monthly downloads, with stars and forks.
lucidrains
memmap-replay-buffer

A simple numpy memmap replay buffer for RL and personal use-cases

2K 9 0
krysyuk
tokbin-core

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

374 2 0
krysyuk
tokbin

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

345 2 0