byte-level
🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.
A tiny byte-level multi-head content classifier (~1.5M params, ~200KB ONNX, <6ms). Classifies code, text, markup, config, images, binary, secrets, 62 code languages, 30 text languages, 90 MIME types from raw bytes — no tokenizer needed.
Non-learned byte-level signal encoder for PyTorch - one modality-agnostic 27-D exact base (anchor rule, Delta = Gray code), losslessly invertible. pip install hsl-embedding
Kronecker Embeddings: byte-level structured token representations for parameter-efficient language models. Reference implementation.
