Bpe Python Packages

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

nfelib

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

subword-nmt

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

youtokentome

Unsupervised text tokenizer focused on computational efficiency

turbobpe

A fast BPE tokenizer, minbpe with CPython-accelerated core

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

tuetoken

The fastest tokenizer for LLMs, up to 30x+ faster.

filipino-tokenizer

The first, morphologically-aware subword tokenizer for Philippine languages, combining a structured linguistic affix data with BPE for language model pretraining.

puretiktoken

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for GPT-4/4o, no Rust extension, zero dependencies

bpetokenizer

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

bytetok

A fast, minimal BPE tokenizer for NLP research.

amharic-tokenizer

Amharic tokenizer with BPE-like merges over decomposed fidel (Cython)

bpe

Byte Pair Encoding for Python!

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

tokenization-scorer

Simple-to-use scoring function for arbitrarily tokenized texts.

pgn-tokenizer

PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game Notiation (PGN).

bpe-summarizer

Auto summarization from BPE tokenization

nlcodec

nlcodec is a collection of encoding schemes for natural language sequences. nlcodec.db is a efficient storage and retrieval layer for integer sequences of varying lengths.

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA

geobpe

Protein Structure Tokenization via Geometric Byte Pair Encoding (GeoBPE)

nfelib-xsdata

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e