Tokenize Python Packages

untokenize

Transforms tokens into original source code (while preserving whitespace)

quebra-frases

chunks strings into byte sized pieces

tivars

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files

tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

pynlp

A pythonic wrapper for Stanford CoreNLP.

tokenizesentences

Python module to tokenize english sentences.

tokelor

Visualize Python token stream produced by tokenize module.

witokit

A Python toolkit to generate a tokenized dump of Wikipedia for NLP

jadoc

Tokenizes Japanese documents to enable CRUD operations.