Bpe Python Packages

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

nfelib

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

subword-nmt

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

youtokentome

Unsupervised text tokenizer focused on computational efficiency

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

amharic-tokenizer

Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

bytetok

A fast, minimal BPE tokenizer for building LLMs.

turbobpe

turboBPE is a fast, minimal Byte Pair Encoding (BPE) tokenizer built on minbpe. It features a C-accelerated backend and a batch-merging algorithm to drastically speed up training while maintaining a clean, hackable codebase. Perfect for training custom vocabularies from scratch.

grouptok

A translation-grouped tokenizer: subwords that translate each other share a group

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

morpheme

Fast, pure-Rust subword tokenization (BPE, WordPiece, Unigram) compatible with Hugging Face tokenizers.

kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

tuetoken

The fastest tokenizer for LLMs, up to 30x+ faster.

filipino-tokenizer

The first, morphologically-aware subword tokenizer for Philippine languages, combining a structured linguistic affix data with BPE for language model pretraining.

piece-tokenizer

Implementation of Piece Tokenizer

puretiktoken

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for GPT-4/4o, no Rust extension, zero dependencies

bpe

Byte Pair Encoding for Python!

bpetokenizer

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

nlcodec

Natural Language EnCoder-Decoder: word, char, bpe etc

tokenization-scorer

Simple-to-use scoring function for arbitrarily tokenized texts.

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

bpe-summarizer

Auto summarization from BPE tokenization

lexibyte

A production-grade, algorithmically optimized Byte-Pair Encoding (BPE) tokenizer. Features GPT-4 style regex pre-splitting, Sennrich frequency optimization, and O(1) inference caching.

pgn-tokenizer

A byte pair encoding (BPE) tokenizer for chess portable game notation (PGN)