Bpe Tokenizer Python Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

amharic-tokenizer

Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

turbobpe

turboBPE is a fast, minimal Byte Pair Encoding (BPE) tokenizer built on minbpe. It features a C-accelerated backend and a batch-merging algorithm to drastically speed up training while maintaining a clean, hackable codebase. Perfect for training custom vocabularies from scratch.

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

tuetoken

The fastest tokenizer for LLMs, up to 30x+ faster.

hf-tokenizer-visualizer

Visualize HuggingFace Byte-Pair Encoding (BPE) Tokenizer encoding process

tokenizerz

BPE tokenizer for LLMs in Pure Zig

rbpe

R-BPE: Improving BPE-Tokenizers with Token Reuse

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA