Inference Optimization Python Packages

tessera-hypernetwork

Generate per-session LoRA adapters in <1s for agentic inference efficiency.

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

llm-autotune

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

flashspec

Adaptive speculative-decoding inference engine with Triton-optimised verification and online bandit draft selection.

sollol

Intelligent Load Balancer for Ollama Clusters - Adaptive Parallelism + VRAM-Aware GPU Routing + Ray/Dask/llama.cpp Distribution

contextpilot

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

wildedge-sdk

ML inference monitoring for Python: on-device and remote models

rabbitllm

Run 70B+ LLMs on a single 4GB GPU — no quantization required.

torch-quant

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

supercompress

SuperCompress — learned context compression for LLMs.

thinkrouter

Cut LLM reasoning-token costs by 60% with one line of code.

kito

Keras inference time optimizer

llm-autobatch

Turn single LLM calls into fast micro-batches. Rust core, Python API.

pavo-bench

PAVO-Bench: 50K-turn voice pipeline benchmark and 85K-param meta-controller for ASR->LLM->TTS routing.