Tiny language detector + tokenizer for 50+ SE/South Asian languages (Burmese, Karen, Chin, Shan, Mon, Khmer, Lao, Thai, Tamil, Hindi, …). Revamp of pyidaungsu.
Tiny language detector + tokenizer for 50+ SE/South Asian languages (Burmese, Karen, Chin, Shan, Mon, Khmer, Lao, Thai, Tamil, Hindi, …). Revamp of pyidaungsu.
Language identification and tokenization for Southeast Asian languages (Burmese, Karen variants, Chin variants, Eastern Kayah, Shan, etc.) — also exposes a `ricelang` CLI
