Tokenizers Python Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

ginza-transformers

Use custom tokenizers in spacy-transformers

llm-magnet

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

token2token

Easy-to-make token mappings using one or two tokenizers and a parallel corpus.

tokenizerchanger

Python script for manipulating the existing tokenizer.

tftokenizers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

precious-nlp

A tokenizer-free NLP library with T-FREE, CANINE, and byte-level approaches

perovskite-tokenizer

Tokenizer for Perovskites