Bpe Tokenizer Python Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

turbobpe

A fast BPE tokenizer, minbpe with CPython-accelerated core

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

tuetoken

The fastest tokenizer for LLMs, up to 30x+ faster.

tokenizerz

BPE tokenizer for LLMs in Pure Zig

hf-tokenizer-visualizer

Visualize HuggingFace Byte-Pair Encoding (BPE) Tokenizer encoding process

amharic-tokenizer

Amharic tokenizer with BPE-like merges over decomposed fidel (Cython)

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA

rbpe

R-BPE: Improving BPE-Tokenizers with Token Reuse