Model Serving Python Packages

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

truss

The simplest way to serve AI/ML models in production

baseten-performance-client

The simplest way to serve AI/ML models in production

truss-transfer

The simplest way to serve AI/ML models in production

bentoml

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

vllm-tpu

A high-throughput and memory-efficient inference and serving engine for LLMs

predikit

The missing bridge between your ML models and your AI agents.

mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

flama

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

mlrun-pipelines-kfp-v1-8

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mlrun-pipelines-kfp-common

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

envd

🏕️ Reproducible development environment for humans and agents

vllm-omni

A framework for efficient model inference with omni-modality models

vllm-cpu-nightly

A high-throughput and memory-efficient inference and serving engine for LLMs

ovmsclient

A scalable inference server for models optimized with OpenVINO™

vllm-ascend

Community maintained hardware plugin for vLLM on Ascend

lorax-client

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

nbox

The official python package for NimbleBox. Exposes all APIs as CLIs and contains modules to make ML 🌸

hsml

Hopsworks Machine Learning Api 🚀 Model management with a model registry and model serving

clearml-serving

ClearML - Model-Serving Orchestration and Repository Solution

jrjmodelregistry

Simple and secure model registry for storing, versioning, and serving ML models using S3 and MongoDB.

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.