Byte Pair Encoding Python Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

tuetoken

The fastest tokenizer for LLMs, up to 30x+ faster.

bpe-summarizer

Auto summarization from BPE tokenization

pgn-tokenizer

A byte pair encoding (BPE) tokenizer for chess portable game notation (PGN)

vgram

Feature extraction from sequential data

textok

Text Tokenizer in C++