Byte Level Python Packages

tinybpe

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

purebyte

Tiny byte-level AI on CPU: a zero-dependency C++17 runtime and neural decision models (0.8 ms per 64-byte decision, 9.4 MB RAM). Reads raw bytes, no tokenizer. Specialists for secret scanning (F1 0.797 vs 0.337 for gitleaks on CredData) and PII redaction. Code under Apache-2.0; model weights under their own license.

pico-type

A tiny byte-level multi-head content classifier (~1.5M params, ~200KB ONNX, <6ms). Classifies code, text, markup, config, images, binary, secrets, 62 code languages, 30 text languages, 90 MIME types from raw bytes — no tokenizer needed.

hsl-embedding

Deterministic, non-learned byte-to-HSL signal encoder for PyTorch: 27-D interpretable features, lossless byte codec, tests, controls, and GPU paths.

hsl-embedding-zero

PyTorch transformer input from raw bytes with zero learned input parameters: no tokenizer, embedding table, or learned input projection.

kronecker-embeddings

Kronecker Embeddings: byte-level structured token representations for parameter-efficient language models. Reference implementation.