Bpe Python Packages

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

nfelib

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

subword-nmt

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

youtokentome

Unsupervised text tokenizer focused on computational efficiency

filipino-tokenizer

The first open-source, morphologically-aware subword tokenizer for Philippine languages, combining a structured linguistic affix data with BPE for language model pretraining.

kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

bpe

Byte pair encoding for graceful handling of rare words in NLP

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

bytetok

A fast, minimal BPE tokenizer for NLP research.

bpetokenizer

A Byte Pair Encoding (BPE) tokenizer, which algorithmically follows along the GPT tokenizer(tiktoken), allows you to train your own tokenizer. The tokenizer is capable of handling special tokens and uses a customizable regex pattern for tokenization(includes the gpt4 regex pattern). supports `save` and `load` tokenizers in the `json` and `file` format. The `bpetokenizer` also supports [pretrained](bpetokenizer/pretrained/) tokenizers.

bpe-summarizer

This summarizer attempts to leverage Byte Pair Encoding (BPE) tokenization and the Bart vocabulary to filter text by semantic meaningfulness.

tokenization-scorer

Simple-to-use scoring function for arbitrarily tokenized texts.

amharic-tokenizer

Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

nlcodec

nlcodec is a collection of encoding schemes for natural language sequences. nlcodec.db is a efficient storage and retrieval layer for integer sequences of varying lengths.

pgn-tokenizer

PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game Notiation (PGN).

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA

geobpe

Geometric Byte Pair Encoding of Protein Structure (ICLR 2026)

nfelib-xsdata

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e