Serving Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torch-model-archiver

Serve, optimize and scale PyTorch models in production

litserve

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

torchserve

Serve, optimize and scale PyTorch models in production

ant-ray-cpp-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

torch-workflow-archiver

Serve, optimize and scale PyTorch models in production

ray-cpp

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torch-model-archiver-nightly

Serve, optimize and scale PyTorch models in production

ovmsclient

A scalable inference server for models optimized with OpenVINO™

ant-ray-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

friendli-client

Friendli Suite Client

clearml-serving

ClearML - Model-Serving Orchestration and Repository Solution

ant-ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torchserve-nightly

Serve, optimize and scale PyTorch models in production

torch-workflow-archiver-nightly

Serve, optimize and scale PyTorch models in production

secretflow-serving-lib

SecretFlow-Serving is a serving system for privacy-preserving machine learning models.

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.

omniback

Serving Inside Pytorch

evadb

Database system for AI-powered apps

fastdeploy-python

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

bodywork

ML pipeline orchestration and model deployments on Kubernetes.

paddle-serving-client

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）