Tokenizer Python Packages

pytokens

A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

sacremoses

Python port of Moses tokenizer, truecaser and normalizer

fugashi

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

nagisa

A Japanese tokenizer based on recurrent neural networks

soynlp

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

quebra-frases

chunks strings into byte sized pieces

sentence-splitter

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

count-tokens

Count tokens in a text file.

tokenizer

A tokenizer for Icelandic text.

toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

javac-parser

Exposes OpenJDK's Java parser and scanner to Python

tokenicer

A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.

botok

🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python

nlpo3

Thai natural language processing library in Rust, with Python and Node bindings.

sctokenizer

A Source Code Tokenizer

hazm

Persian NLP Toolkit

pyrwkv-tokenizer

A fast RWKV Tokenizer written in Rust

rwkv-tokenizer

RWKV Tokenizer

turbobpe

A fast BPE tokenizer, minbpe with CPython-accelerated core

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

mlx-model-doctor

Validate an MLX / Hugging Face model repository before you load it.