Speculative Decoding Python Packages

rapid-mlx

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

speculators

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

mtplx

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

gmlx

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

ax-engine

One Mac process. Many models. Real speed. Multi-model LLM serving with prefix reuse, MTP acceleration, and OpenAI APIs — built for Apple Silicon, measured against mlx-lm and llama.cpp.

mlx-dspark

Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

inferhost

Self-hosted multi-modal AI server: chat/vision LLMs (Qwen, Llama, Gemma, DeepSeek), text-to-speech (OuteTTS, Qwen3-TTS) and image generation (SDXL, Flux, Z-Image, Qwen-Image) behind one OpenAI-compatible endpoint. Wraps llama.cpp + stable-diffusion.cpp, auto-fetches GGUF models from Hugging Face, VRAM hot-swap, speculative decoding, no compiling.

aphrodite-engine

Large-scale LLM inference engine

intel-extension-for-transformers

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

transformers-mblt

Python toolkit for running pre-quantized Hugging Face Transformers models (LLMs, VLMs, speech recognition) on Mobilint NPUs.

quantprobe

Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGUF on your exact hardware, self-calibrates, emits the exact llama.cpp command — or 'quantprobe auto' does it all. Falsification-tested laws; misses published at full size. pip install quantprobe

angelslim

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

yunshu

Fast local LLM / VLM inference engine for Apple Silicon (MLX). OpenAI- and Anthropic-compatible, lossless speculative decoding (DFlash2 / MTP), hybrid-model prefix cache with RAM/SSD tiers, native Claude Code / Codex / opencode support. Qwen3.8-27B tuned first.

msmf

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.

model-serving-minefield

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.

tightwad

Mixed-vendor GPU inference cluster manager with speculative decoding

eagle-llm

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

flashspec

Adaptive speculative-decoding inference engine with Triton-optimised verification and online bandit draft selection.

m5-infer

Extraordinary speed, extraordinary quality — MLX-based inference engine for Apple Silicon.

dejavuu

Training-free speculative decoding for ONNX Runtime and Hugging Face PyTorch models

dfly-mlx

Native DFly speculative decoding for Apple Silicon with MLX

mlxturbo

Speculative-decoding inference engine and OpenAI/Anthropic/Responses-compatible server for MLX on Apple Silicon

opencac

Multi-agent orchestration CLI for AI coding tools. Chain Claude Code, Antigravity, and Codex with validated handoffs, JSONL audit logging, hybrid cloud/local routing, and speculative decoding for local LLMs.

angelslim-fork

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.