Model Serving Python Packages

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

baseten-performance-client

The simplest way to serve AI/ML models in production

truss

The simplest way to serve AI/ML models in production

truss-transfer

The simplest way to serve AI/ML models in production

bentoml

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

a11-kit

A streaming action runtime for AI agents, model serving, and multimodal APIs

predikit

The missing bridge between your ML models and your AI agents.

envd

🏕️ Reproducible development environment for humans and agents

mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

vllm-cpu-nightly

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-omni

A framework for efficient model inference with omni-modality models

mlrun-pipelines-kfp-v1-8

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mlrun-pipelines-kfp-common

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

flama

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

experts4bit-qlora

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

vllm-tpu

A high-throughput and memory-efficient inference and serving engine for LLMs

grouped-nf4-gemm

4-bit GPU kernels that compute directly on packed Mixture-of-Experts weights, for training and inference.

hugpy-video

Self-hosted LLM console, OpenAI-compatible API and GPU worker fleet: 13 hugpy-* Python distributions and the React consoles.

mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

hugpy

Self-hosted LLM console, OpenAI-compatible API and GPU worker fleet: 13 hugpy-* Python distributions and the React consoles.

ax-engine

One Mac process. Many models. Real speed. Multi-model LLM serving with prefix reuse, MTP acceleration, and OpenAI APIs — built for Apple Silicon, measured against mlx-lm and llama.cpp.

hugpy-engine

Self-hosted LLM console, OpenAI-compatible API and GPU worker fleet: 13 hugpy-* Python distributions and the React consoles.

hugpy-oracle

Self-hosted LLM console, OpenAI-compatible API and GPU worker fleet: 13 hugpy-* Python distributions and the React consoles.