Sentencepiece Python Packages

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

morpheme

Fast, pure-Rust subword tokenization (BPE, WordPiece, Unigram) compatible with Hugging Face tokenizers.

kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

piece-tokenizer

Implementation of Piece Tokenizer

tiny-tokenizer

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

akshara-tokenizer

Akshara-aware tokenizer for six Brahmic scripts, with a trained SentencePiece model

escape-unk

Escape unknown symbols in SentecePiece vocabularies

meitei-senter

Neural sentence boundary detection for Meitei Mayek (Manipuri) using SentencePiece tokenization and a CNN-based spaCy pipeline.

toksight

Tokenizer analysis & comparison toolkit — compression, coverage, audit, cost estimation. Zero deps.