Bpe Tokenizer Python Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

hf-tokenizer-visualizer

Visualize HuggingFace Byte-Pair Encoding (BPE) Tokenizer encoding process

tokenizerz

Minimal BPE tokenizer in Zig

rbpe

R-BPE: Improving BPE-Tokenizers with Token Reuse

amharic-tokenizer

Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA