Llm Inference Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

flashinfer-python

FlashInfer: Kernel Library for LLM Serving

flashinfer-cubin

FlashInfer: Kernel Library for LLM Serving

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

yunchang

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

bentoml

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

monocle-apptrace

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.

ai-dynamo-runtime

A Datacenter Scale Distributed Inference Serving Framework

openvino-dev

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

ai-dynamo

A Datacenter Scale Distributed Inference Serving Framework

veloxquant-mlx

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

ray-cpp

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

xllamacpp

xllamacpp - a Python wrapper of llama.cpp

ant-ray-cpp-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

experts4bit-qlora

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

rapid-mlx

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

chimeraforge

PyPI capacity-planning CLI for LLM deployment. pip install chimeraforge.

prompt-poet

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

vllm-cpu

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

grouped-nf4-gemm

4-bit GPU kernels that compute directly on packed Mixture-of-Experts weights, for training and inference.

mlx-kquant

Native K-quant support for MLX, with a quantization and fine-tuning toolchain for Apple Silicon

kvbm

A Datacenter Scale Distributed Inference Serving Framework