Tokenize Python Packages

untokenize

Transforms tokens into original source code (while preserving whitespace)

quebra-frases

chunks strings into byte sized pieces

tivars

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files

tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

tokelor

Visualize Python token stream produced by tokenize module.

pynlp

A pythonic wrapper for Stanford CoreNLP.

jadoc

Tokenizes Japanese documents to enable CRUD operations.

witokit

A Python toolkit to generate a tokenized dump of Wikipedia for NLP