Serving Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

litserve

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

torch-model-archiver

Serve, optimize and scale PyTorch models in production

ray-cpp

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torchserve

Serve, optimize and scale PyTorch models in production

ant-ray-cpp-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

ovmsclient

A scalable inference server for models optimized with OpenVINO™

torch-model-archiver-nightly

Serve, optimize and scale PyTorch models in production

ant-ray-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torchserve-nightly

Serve, optimize and scale PyTorch models in production

torch-workflow-archiver

Serve, optimize and scale PyTorch models in production

torch-workflow-archiver-nightly

Serve, optimize and scale PyTorch models in production

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.

secretflow-serving-lib

SecretFlow-Serving is a serving system for privacy-preserving machine learning models.

ant-ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

paddle-serving-app

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

paddle-serving-client

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

bodywork

ML pipeline orchestration and model deployments on Kubernetes.

paddle-serving-server-gpu

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

clearml-serving

ClearML - Model-Serving Orchestration and Repository Solution

friendli-client

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

evadb

Database system for AI-powered apps