PyRank

Tokenization Python Packages

Python packages with the GitHub topic tokenization. Sorted by monthly downloads, with stars and forks.
explosion
spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

21.3M 34K 5K
WorksApplications
sudachipy

Sudachi in Rust 🦀 and new generation of SudachiPy

1.6M 486 62
mysto
ff3

FPE - Format Preserving Encryption with FF3 in Python

280K 104 20
marcelroed
gigatoken

Language model tokenization at GB/s

266K 4K 222
adbar
simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

184K 219 17
AgentOps-AI
tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

161K 2K 107
natasha
razdel

Rule-based token, sentence segmentation for Russian language

130K 288 34
OpenVoiceOS
quebra-frases

chunks strings into byte sized pieces

70K 1 3
ScrapeGraphAI
toonify

Toonify: Compact data format reducing LLM token usage by 30-60%

68K 364 29
OpenNMT
pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

43K 343 83
izikeros
count-tokens

Count tokens in a text file.

21K 15 0
nlpcloud
nlpcloud

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...

16K 86 8
vkcom
youtokentome

Unsupervised text tokenizer focused on computational efficiency

11K 978 106
davidpirogov
toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

4K 9 5
TI-Toolkit
tivars

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files

4K 29 3
PyThaiNLP
attacut

A Fast and Accurate Neural Thai Word Segmenter

3K 96 18
kaunghtetsan275
pyidaungsu-numpy2

Tiny language detector + tokenizer for 50+ SE/South Asian languages (Burmese, Karen, Chin, Shan, Mon, Khmer, Lao, Thai, Tamil, Hindi, …). Revamp of pyidaungsu.

3K 42 7
Deepchavda007
blazechunk

âš¡ The fastest semantic text chunking library a SIMD-accelerated Rust core with a sync + async Python API.

3K 4 1
BitBadges
bitbadgespy-sdk

The most feature-rich tokenization standard ever built — TypeScript SDK for the BitBadges tokenization Cosmos SDK module

3K 0 0
rosette-api
rosette-api

Babel Street Analytics Client Library for Python

2K 38 37
THUDM
icetk

A unified tokenization tool for Images, Chinese and English.

2K 152 16
redact-secret
redact-secret-vault

Reversible redaction for AI workflows: replace secrets with opaque tokens, then restore them only to authorized destinations. JavaScript and Python.

2K 1 0
explosion
spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

2K 863 112
daac-tools
vaporetto

🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. (Python wrapper)

2K 21 1