Inference Acceleration Python Packages

sageattention

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

mlx-teacache

TeaCache step-skipping for FLUX, Qwen-Image & Z-Image diffusion on Apple Silicon, in pure MLX

autonomi-nos

⚡️ A fast and flexible PyTorch inference server that runs locally, on any cloud or AI HW.

turbodiffusion

TurboDiffusion: 100–200× Acceleration for Video Diffusion Models

torch-nos

⚡️ A fast and flexible PyTorch inference server that runs locally, on any cloud or AI HW.

rwkv-quant-nv

RWKV-7 FP8 quantized inference - 6.4x decode speedup on Blackwell GPUs with <0.3% accuracy loss. Full FP8 E4M3 weight quantization with fused Triton kernels.