Serving Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torch-model-archiver

Serve, optimize and scale PyTorch models in production

litserve

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

torchserve

Serve, optimize and scale PyTorch models in production

ant-ray-cpp-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-cpp

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torch-model-archiver-nightly

Serve, optimize and scale PyTorch models in production

seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

ovmsclient

A scalable inference server for models optimized with OpenVINO™

clearml-serving

ClearML - Model-Serving Orchestration and Repository Solution

torch-workflow-archiver

Serve, optimize and scale PyTorch models in production

ant-ray-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torch-workflow-archiver-nightly

Serve, optimize and scale PyTorch models in production

torchserve-nightly

Serve, optimize and scale PyTorch models in production

omniback

Serving Inside Pytorch

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

friendli-client

Friendli Suite Client

secretflow-serving-lib

SecretFlow-Serving is a serving system for privacy-preserving machine learning models.

ant-ray

Ray provides a simple, universal API for building distributed applications.

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.

paddle-serving-server

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

bodywork

ML pipeline orchestration and model deployments on Kubernetes, made really easy.

paddle-serving-client

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

fastdeploy-python

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle