Byte Pair Encoding Python Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

bpe-summarizer

This summarizer attempts to leverage Byte Pair Encoding (BPE) tokenization and the Bart vocabulary to filter text by semantic meaningfulness.

vgram

Feature extraction from sequential data

pgn-tokenizer

PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game Notiation (PGN).

textok

Text Tokenizer in C++