Tokenizers Python Packages

ginza-transformers

Use custom tokenizers in spacy-transformers

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

llm-magnet

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

tokenizerchanger

Library for manipulating the existing tokenizer.

precious-nlp

A tokenizer-free NLP library with T-FREE, CANINE, and byte-level approaches

tftokenizers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels.