Inference Optimization Python Packages

kernel-lens

Deploy any research model right away

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

gst-plugin-iceoryx2

GStreamer iceoryx2 shared-memory zero-copy src/sink with python bindings

airun-profiler

An open-source AI infrastructure profiler that connects AI workload telemetry to physical infrastructure behavior and economics.

contextpilot

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

sollol

Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics

microloop

Behavior JIT for production AI. Turn verified repeated decisions into qualified local software.

llm-autotune

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

inferenceos

Adaptive AI inference runtime built on llama.cpp with intelligent scheduling, self-optimizing execution, runtime learning, and advanced memory management.

ucci-router

UCCI: calibrated cascade routing for LLMs. Reference implementation of arXiv:2605.18796 (Python, CLI, Rust).

wildedge-sdk

Python SDK for WildEdge

ace-experiment-framework

Contract and claim-boundary tooling for empirical AI-system experiments. Turns retained evidence into falsifiable, condition-specific decisions.

turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

inferpilot

Evidence-first vLLM screening and controlled capacity analysis. Screen live metrics, verify config changes, and abstain when evidence is weak.

custom-dl-optimizer

Regression-aware PyTorch inference optimizer using FX, Triton, and TorchInductor

rabbitllm

Run 70B+ LLMs on a single 4GB GPU — no quantization required.

berth-placement

Adaptive placement infrastructure for AI inference. berth predicts, sounding measures, pilot adapts.

torch-quant

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

flashspec

Adaptive speculative-decoding inference engine with Triton-optimised verification and online bandit draft selection.

llm-autobatch

Turn single LLM calls into fast micro-batches. Rust core, Python API.

univis

UniVis — Transformer 推理冗余诊断与可视化工具：一行代码挂载 hook，逐层冗余画像 · 实时热力图 · 自包含 HTML 报告；已在沐曦曦云 C500 与 NVIDIA L20 上实证冗余画像跨硬件一致（r=1.0000）。

kito

Optimize layers structure of Keras model to reduce computation time

hypermcts

🚀 Autonomous Test-Time Compute Scaling & Dynamic KV-Cache Sparsification for Edge & Local LLMs | Pure PyTorch / Ollama MCTS Reasoning Engine (-65% VRAM)

thinkrouter

Cut LLM reasoning-token costs by 60% with one line of code