Tokenizer Python Packages

sqlparse

A non-validating SQL parser module for Python

pytokens

A fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

sacremoses

Python port of Moses tokenizer, truecaser and normalizer

fugashi

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

gigatoken

Language model tokenization at GB/s

nagisa

A Japanese tokenizer based on recurrent neural networks

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

soynlp

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

ja-sentence

Light-weight sentence tokenizer for Japanese.

zh-sentence

Light-weight sentence tokenizer for Chinese languages.

kr-sentence

Light-weight sentence tokenizer for Korean. Supports full-width and half-width punctuation marks.

natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

quebra-frases

chunks strings into byte sized pieces

sentence-splitter

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

javac-parser

Exposes OpenJDK's Java parser and scanner to Python

tokenizer

A tokenizer for Icelandic text.

count-tokens

Count tokens in a text file.

tokenicer

A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.

botok

🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python

sctokenizer

A Source Code Tokenizer

hazm

Persian NLP Toolkit

nlpo3

Thai natural language processing library in Rust, with Python and Node bindings.

cereja

Zero-dependency Python utilities for files, text, data processing, system tasks and developer workflows.