Inference Optimization Python Packages

llm-autotune

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

sollol

Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics

contextpilot

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

torch-quant

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

rabbitllm

Run 70B+ LLMs on a single 4GB GPU — no quantization required.

pavo-bench

A 50K-turn voice pipeline benchmark and an 85K-param meta-controller that cuts P95 latency 10.3% and energy 71% vs fixed cloud. TMLR 2026.

wildedge-sdk

Python SDK for WildEdge

kito

Optimize layers structure of Keras model to reduce computation time

thinkrouter

Cut LLM reasoning-token costs by 60% with one line of code

llm-autobatch

Turn single LLM calls into fast micro-batches. Rust core, Python API.