Huggingface Tokenizers Python Packages

tokbin-core

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

tokbin

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

transformers-domain-adaptation

:no_entry: [DEPRECATED] Adapt Transformer-based language models to new text domains