Tokenizer Python Packages

pytokens

A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

sacremoses

Python port of Moses tokenizer, truecaser and normalizer

fugashi

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

nagisa

A Japanese tokenizer based on recurrent neural networks

soynlp

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

sentence-splitter

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

tokenizer

A tokenizer for Icelandic text.

natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

count-tokens

Count tokens in a text file.

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

sctokenizer

A Source Code Tokenizer

tokenicer

A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.

toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

botok

🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python

quebra-frases

chunks strings into byte sized pieces

nlpo3

Thai natural language processing library in Rust, with Python and Node bindings.

hazm

Persian NLP Toolkit

cereja

Cereja is a bundle of useful functions we don't want to rewrite and .. just pure fun!

javac-parser

Exposes OpenJDK's Java parser and scanner to Python

modelito

Lightweight Python abstractions and connectors for LLMs (OpenAI, Claude, Gemini, Ollama)

crossandra

A fast and simple tokenization library for Python operating on enums and regular expressions, with a decent amount of configuration.

pyrwkv-tokenizer

A fast RWKV Tokenizer written in Rust

somajo

A tokenizer and sentence splitter for German and English web and social media texts.