Tokenization Python Packages

spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

sudachipy

Sudachi in Rust 🦀 and new generation of SudachiPy

tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

ff3

FPE - Format Preserving Encryption with FF3 in Python

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

razdel

Rule-based token, sentence segmentation for Russian language

toonify

Toonify: Compact data format reducing LLM token usage by 30-60%

quebra-frases

chunks strings into byte sized pieces

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

count-tokens

Count tokens in a text file.

toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

nlpcloud

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...

youtokentome

Unsupervised text tokenizer focused on computational efficiency

attacut

A Fast and Accurate Neural Thai Word Segmenter

turbobpe

A fast BPE tokenizer, minbpe with CPython-accelerated core

h-net-dynamic-chunking

Implementation of the dynamic chunking mechanism in H-net by Hwang et al. of Carnegie Mellon

plane

A text processing tool including tag(HTML, URL, Email) extraction and removing, punctuation normalization, simple segmentation, and so on.

spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

pyidaungsu-numpy2

Tiny language detector + tokenizer for 50+ SE/South Asian languages (Burmese, Karen, Chin, Shan, Mon, Khmer, Lao, Thai, Tamil, Hindi, …). Revamp of pyidaungsu.

tab2seq

Transform tabular event data into sequences ready for Transformer and Sequential models: Life2Vec, BEHRT and more.

tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

icetk

A unified tokenization tool for Images, Chinese and English.

rosette-api

Babel Street Analytics Client Library for Python

ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).