Inference Server Python Packages

baseten-performance-client

The simplest way to serve AI/ML models in production

truss

The simplest way to serve AI/ML models in production

truss-transfer

The simplest way to serve AI/ML models in production

inference-sdk

Turn any computer or edge device into a command center for your computer vision projects.

inference

Turn any computer or edge device into a command center for your computer vision projects.

rapid-mlx

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

inference-gpu

Turn any computer or edge device into a command center for your computer vision projects.

inference-cli

Turn any computer or edge device into a command center for your computer vision projects.

inference-cpu

Turn any computer or edge device into a command center for your computer vision projects.

inference-core

Turn any computer or edge device into a command center for your computer vision projects.

vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

decisio

A serving layer for decisions: typed questions in, a probability for every option out, one forward pass each. Three bases: Qwen3.6-35B-A3B, Gemma 4 12B, Gemma 4 31B.

ramalama

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

inferhost

Self-hosted multi-modal AI server: chat/vision LLMs (Qwen, Llama, Gemma, DeepSeek), text-to-speech (OuteTTS, Qwen3-TTS) and image generation (SDXL, Flux, Z-Image, Qwen-Image) behind one OpenAI-compatible endpoint. Wraps llama.cpp + stable-diffusion.cpp, auto-fetches GGUF models from Hugging Face, VRAM hot-swap, speculative decoding, no compiling.

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.

phocinae-server

Phocinae decision-model inference server: TypeSafe /v1/systemone-compatible local API (Largha 150M)

baseten-inference-client

The simplest way to serve AI/ML models in production

friendli-client

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

mila-llm

A C++23 library for local LLMs at the metal — inference and training, built from explicit neural-network components you can read and understand.

mship

Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.

mship-engine

Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.

jev-compatible-server

Open inference runtime for decision models with a Jev-compatible API.

eeane

Run text embedding and reranking models on the Apple Neural Engine of Apple Silicon Macs. OpenAI-compatible server; ModernBERT, BERT, and XLM-RoBERTa.

laya-decision-api

把本地 System-1 决策模型（Laya）包成 HTTP 服务：意图分类 / 工单分派 / LLM 路由 / 内容审核，离线零 token、不生成文字；Apple Silicon(MLX) 与 Linux(torch) 双后端；可导出 OpenAPI 3.0/3.1；Java/TS/Python 零依赖 SDK。