Kv Cache Python Packages

lmcache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

veloxquant-mlx

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

glyd

Lossless AI compression: 33% less GPU memory, bit for bit.

quantcpp

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

tardigrade-db

Experimental LLM-native memory-kernel prototype for persistent agent memory in quantized KV-cache latent space.

narwhal-inference

Adaptive Disaggregated Inference on a Role-free Fleet!

pion-vllm-mlx

Pion — the memory engine for AI inference

mnemolink

Curated, injectable personas and artificial memories for AI agents.

lmcache-cli

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

turboquant-mlx-full

Extreme weight + KV cache compression for LLMs on Apple Silicon (MLX implementation of Google's TurboQuant)

turboquant-pro

Consumer-aware compression for embedding indexes and LLM KV caches — compress by the metric the downstream consumer actually uses. PCA-Matryoshka + TurboQuant (27x @ 99.8% recall@10), asymmetric K/V, CUDA/Triton kernels, vLLM plugin, replayable CI-gated claims. MIT.

bigsmall

Lossless AI model compression - ~34% smaller with bit-identical weights; the autopilot profiles your machine, picks the highest fidelity that runs, and streams models bigger than your RAM.

semblend-vllm-connector

Out-of-tree vLLM KVConnector for SemBlend semantic KV donor discovery

cappr

Completion After Prompt Probability. Make your LLM make a choice

metria

Research, reference implementations, and evaluation tools for efficient LLM inference, KV-cache compression, quantization, and behavioral fidelity.

kvfit

Will a Hugging Face LLM fit your GPU or DGX? Plan weights, KV cache or recurrent state, context, TP/DP, and measured calibration

processfork

git for AI agents — snapshot, fork, and merge live LLM sessions in 8 ms. Drop-in for Claude Code, LangGraph, vLLM, SGLang.

mlx-quant-fidelity

Measure quantization quality loss on Apple Silicon MLX — KL divergence, top-token flip rate and perplexity delta for KV-cache and weight quantization

workingset

GPU serving-capacity study: a scenario model and interactive explorer for multi-GPU, MoE, and prompt-caching trade-offs in agentic coding workloads

llm-autotune

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

adaptq

High-performance C++17 KV cache compression engine for LLM inference with Python bindings, SIMD acceleration, and multi-backend support.

squish-ai

⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.

avp

Python SDK for Agent Vector Protocol – transfer KV-cache between LLM agents instead of text

thaw-vllm

git for live agent sessions: snapshot, branch, and diff a running vLLM/SGLang session as a durable file. inspect & diff on a laptop, no GPU; restore skips prefill. Rust + CUDA, Apache-2.0. pip install thaw-vllm