Word Segmentation Python Packages

sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

pythainlp

Thai natural language processing in Python

symspellpy

Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm

nagisa

A Japanese tokenizer based on recurrent neural networks

kiwipiepy

Python API for Kiwi

kiwipiepy-model

Python API for Kiwi

pycantonese

Cantonese Linguistics and NLP

adaseq

AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models

bi-lstm-crf

A PyTorch implementation of the BI-LSTM-CRF model.

rustling

A high-performance library for computational linguistics

wordseg

Word segmentation models

youtokentome

Unsupervised text tokenizer focused on computational efficiency

bert-multitask-learning

BERT for Multitask Learning

kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

tf-sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

hellonlp

NLP tools, word segmentation, sentence segmentation， New-Word-Discovery，新词发现

lac

百度NLP：分词，词性标注，命名实体识别，词重要性

dksplit

DKSplit - fast word segmentation for Python. Split domain names and concatenated strings into words using BiLSTM-CRF + ONNX Runtime.

vncorenlp

A Python wrapper for VnCoreNLP using a bidirectional communication channel.

hanlperceptron

Native Python HanLP Perceptron Model: HanLPerceptron 中文斷詞 詞性標註 命名實體識別

cjieba

Python cffi binding for cjieba

symspellcpppy

A Fast SymSpell port for python written in C++ using pybind11.

hashformers

Accurate word segmentation for hashtags and text, powered by Transformers and Beam Search. A scalable alternative to heuristic splitters and massive LLMs.