Inference Python Packages

sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

filetype

Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

torchao

PyTorch native quantization and sparsity for training and inference

mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

onnx-graphsurgeon

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

owlrl

A simple implementation of the OWL2 RL Profile on top of RDFLib: it expands the graph with all possible triples that OWL RL defines. It can be used together with RDFLib to expand an RDFLib Graph object, or as a stand alone service with its own serialization.

tensorrt-cu12-bindings

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

inference-gpu

Turn any computer or edge device into a command center for your computer vision projects.

sglang-kernel

SGLang is a high-performance serving framework for large language models and multimodal models.

python-toon

🐍 TOON for Python (Token-Oriented Object Notation) Encoder/Decoder - Reduce LLM token costs by 30-60% with structured data.

multi-model-server

Multi Model Server is a tool for serving neural net models for inference

model-archiver

Multi Model Server is a tool for serving neural net models for inference

inference-cli

Turn any computer or edge device into a command center for your computer vision projects.

tensorrt-cu12

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

tensorrt-cu12-libs

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

sagemaker-inference

Serve machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.

text-generation

Large Language Model Text Generation Inference

inference-sdk

Turn any computer or edge device into a command center for your computer vision projects.