subwords
High-precision, Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.
Evaluation for unsupervised morphological analysis and segmentation
