Tokenize Python Packages

untokenize

Transforms tokens into original source code (while preserving whitespace)

quebra-frases

chunks strings into byte sized pieces

tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

tivars

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files

pynlp

A pythonic wrapper for Stanford CoreNLP.

tokenizesentences

Python module to tokenize english sentences.

tokelor

Visualize Python token stream produced by tokenize module.

jadoc

Tokenizes Japanese documents to enable CRUD operations.

witokit

A Python toolkit to generate a tokenized dump of Wikipedia for NLP