Cpu Inference Python Packages

shad0wllm

System 0 for your LLM: learns the fixed-choice answers your LLM gives (intents, routing, moderation) and serves the repeats in microseconds from a small CPU table, at most 5% disagreement, certified. Zero-code proxy for the OpenAI Decisions API, System One and chat models.

vllm-cpu

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

nanogemm

Minimalist, bare-metal SIMD & Assembly GEMM engine for Python. Sub-microsecond CPU matrix multiplication & Native MCP Server (nanogemm-mcp).

bit-jev

Kev-style structured decisions on BitNet with a native I2_S CPU path (source preview)

diarize

Speaker diarization for Python — "who spoke when?" CPU-only, no API keys, Apache 2.0. ~10.8% DER on VoxConverse, 8x faster than real-time.

edgejev

离线可用的本地类型化决策：4 核 CPU 单题 15.6ms。Local & offline Jev / System One inference on CPU — ONNX + INT8, no torch at runtime. 支持 laya / kev / PlayJev

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

ovkit

One line of Python for 20 AI capabilities on Intel CPU/GPU/NPU — detection, OCR, text-to-speech, pose. Offline, Apache-2.0, no AGPL.

phocinae-server

Phocinae decision-model inference server: TypeSafe /v1/systemone-compatible local API (Largha 150M)

simthinkd

A tiny decision model (265k parameters) that picks one action in about 2 ms on one CPU core, inside real-time loops.

vllm-cpu-avx512vnni

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

vllm-cpu-amxbf16

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

vllm-cpu-avx512bf16

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

dnaty

Compress PyTorch models for edge devices — CPU-only, no GPU, no retraining. One function call.

vllm-cpu-avx512

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

purebyte

Tiny byte-level AI on CPU: a zero-dependency C++17 runtime and neural decision models (0.8 ms per 64-byte decision, 9.4 MB RAM). Reads raw bytes, no tokenizer. Specialists for secret scanning (F1 0.797 vs 0.337 for gitleaks on CredData) and PII redaction. Code under Apache-2.0; model weights under their own license.

velocirag

Lightning-fast RAG for AI agents. ONNX-powered, 4-layer fusion, MCP server. No PyTorch.

bnn-lab

Extreme low-bit inference lab: pack weights to 1–1.58 bits and run real XNOR/popcount kernels on CPU/edge — honest STE training, not fake GPU 32× from sign().

adaptshot

Few-shot image classification on an ordinary CPU, offline, with conformal prediction sets that say how much to trust each answer and a human-in-the-loop correction path. Torch-free ONNX inference, 120 MB, Python 3.10+.

fast-lightonocr

High-performance Rust inference engine for LightOnOCR-2-1B using ONNX Runtime.

memvanta

Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducible llama.cpp benchmarks.

inferlast

inferlast — CPU-first LLM inference optimizer: prove a real win on your CPU before you pay for a GPU. Profiles the true bottleneck, picks a change, and refuses to call it a win unless the benchmark is honestly real.

gguf-llama

Wrapper for simplified use of Llama2 GGUF quantized models.

arey

Simple large language model playground app