Inference Python Packages

filetype

Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

torchao

PyTorch native quantization for training and inference

mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

owlrl

A simple implementation of the OWL2 RL Profile on top of RDFLib: it expands the graph with all possible triples that OWL RL defines. It can be used together with RDFLib to expand an RDFLib Graph object, or as a stand alone service with its own serialization.

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

sglang-kernel

SGLang is a high-performance serving framework for large language models and multimodal models.

tensorrt-cu12

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

python-toon

🐍 TOON for Python (Token-Oriented Object Notation) Encoder/Decoder - Reduce LLM token costs by 30-60% with structured data.

tensorrt-cu12-libs

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

sagemaker-inference

Serve machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.

tensorrt-cu12-bindings

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

onnx-graphsurgeon

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

tensorrt

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

mooncake-transfer-engine-cuda13

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

tensorrt-cu13

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

tensorrt-cu13-libs

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

multi-model-server

Multi Model Server is a tool for serving neural net models for inference

model-archiver

Multi Model Server is a tool for serving neural net models for inference