Cpu Inference Python Packages

vllm-cpu

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

dnaty

Compress PyTorch models for edge devices — CPU-only, no GPU, no retraining. One function call.

diarize

Speaker diarization for Python — "who spoke when?" CPU-only, no API keys, Apache 2.0. ~10.8% DER on VoxConverse, 8x faster than real-time.

vllm-cpu-avx512vnni

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

vllm-cpu-amxbf16

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

vllm-cpu-avx512bf16

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

vllm-cpu-avx512

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

velocirag

Lightning-fast RAG for AI agents. ONNX-powered, 4-layer fusion, MCP server. No PyTorch.

gguf-llama

Wrapper for simplified use of Llama2 GGUF quantized models.

arey

Simple large language model playground app