Tokenization Python Packages

spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

sudachipy

Sudachi in Rust 🦀 and new generation of SudachiPy

tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

ff3

FPE - Format Preserving Encryption with FF3 in Python

toonify

Toonify: Compact data format reducing LLM token usage by 30-60%

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

razdel

Rule-based token, sentence segmentation for Russian language

count-tokens

Count tokens in a text file.

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

h-net-dynamic-chunking

Implementation of the dynamic chunking mechanism in H-net by Hwang et al. of Carnegie Mellon

quebra-frases

chunks strings into byte sized pieces

nlpcloud

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...

youtokentome

Unsupervised text tokenizer focused on computational efficiency

attacut

A Fast and Accurate Neural Thai Word Segmenter

filipino-tokenizer

The first open-source, morphologically-aware subword tokenizer for Philippine languages, combining a structured linguistic affix data with BPE for language model pretraining.

icetk

A unified tokenization tool for Images, Chinese and English.

spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

tivars

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files

bitbadgespy-sdk

The most feature-rich tokenization standard ever built — TypeScript SDK for the BitBadges tokenization Cosmos SDK module

plane

A text processing tool including tag(HTML, URL, Email) extraction and removing, punctuation normalization, simple segmentation, and so on.

ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

rosette-api

Babel Street Analytics Client Library for Python

code-tokenize

Fast tokenization and structural analysis of any programming language