PyRank

Byte Pair Encoding Python Packages

Python packages with the GitHub topic byte-pair-encoding. Sorted by relevance, with stars and monthly downloads.
gweidart
rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

16K 41 5
tuetoken-org
tuetoken

The fastest tokenizer for LLMs, up to 30x+ faster.

2K 3 0
DVDAGames
pgn-tokenizer

A byte pair encoding (BPE) tokenizer for chess portable game notation (PGN)

669 1 1
crodriguez1a
bpe-summarizer

Auto summarization from BPE tokenization

528 3 1
akhvorov
vgram

V-gram builder library

332 7 0
jiauzhang
textok

Text Tokenizer in C++

127 0 0