Inference Python Packages

sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

filetype

Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

torchao

PyTorch native quantization and sparsity for training and inference

mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

inference-gpu

Turn any computer or edge device into a command center for your computer vision projects.

inference-cli

Turn any computer or edge device into a command center for your computer vision projects.

onnx-graphsurgeon

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

python-toon

🐍 TOON for Python (Token-Oriented Object Notation) Encoder/Decoder - Reduce LLM token costs by 30-60% with structured data.

mooncake-transfer-engine

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

any-llm-sdk

Communicate with an LLM provider using a single interface

sglang-kernel

SGLang is a high-performance serving framework for large language models and multimodal models.

tensorrt-cu12-bindings

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

multi-model-server

Multi Model Server is a tool for serving neural net models for inference

model-archiver

Multi Model Server is a tool for serving neural net models for inference

sgl-kernel

SGLang is a high-performance serving framework for large language models and multimodal models.

sagemaker-inference

Serve machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.

tensorrt-cu12

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

pnnx

ncnn is a high-performance neural network inference framework optimized for the mobile platform