Sentencepiece Python Packages

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

tiny-tokenizer

No description available

meitei-senter

Neural sentence boundary detection for Meitei Mayek (Manipuri) using SentencePiece tokenization and a CNN-based spaCy pipeline.

escape-unk

Escape unknown symbols in SentecePiece vocabularies

toksight

Tokenizer analysis & comparison toolkit — compression, coverage, audit, cost estimation. Zero deps.