Inference Engine Python Packages

astroid

A common base representation of python source code for pylint and other projects

sigit-code

A modular local inference LLM coding agent. Core component of siGit Factory.

ai4i-core

A comprehensive microservices based platform to handle language based AI models inferences at scale.

qai-hub-models-cli

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

qai-hub-models

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

mtplx

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

gmlx

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

onediff

OneDiff: An out-of-the-box acceleration library for diffusion models.

experta

Expert Systems for Python

mlx-dspark

Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

onediffx

OneDiff: An out-of-the-box acceleration library for diffusion models.

nobodywho

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

aphrodite-engine

Large-scale LLM inference engine

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

needle-rs

600 KB WASM runtime for Cactus Compute's Needle AI tool-calling models, Needle 3, 2 and 1 from one build. Browser, Cloudflare Workers, Node.js, Python, C FFI and no_std. Token-exact with the JAX reference. No backend, no API key.

kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

yunshu

Fast local LLM / VLM inference engine for Apple Silicon (MLX). OpenAI- and Anthropic-compatible, lossless speculative decoding (DFlash2 / MTP), hybrid-model prefix cache with RAM/SSD tiers, native Claude Code / Codex / opencode support. Qwen3.8-27B tuned first.

neurobrix

Universal AI Runtime — Execute any model on any hardware

friendli-client

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

qai-hub-apps

The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

fedml

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

exxa

Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and minimal learning curve.

para-attn

https://wavespeed.ai/ Context parallel attention that accelerates DiT model inference with dynamic caching

bigrig

Run large Mixture-of-Experts models on an Apple Silicon Mac that does not have room for them, and know what it cost