Distributed Inference Python Packages

flashinfer-python

FlashInfer: Kernel Library for LLM Serving

flashinfer-cubin

FlashInfer: Kernel Library for LLM Serving

tightwad

Mixed-vendor GPU inference cluster manager with speculative decoding

sollol

Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics

mlx-sharding

Distributed Inference for mlx LLm

forge-sdk

Python SDK for the Forge compute economy — where AI agents earn and spend Compute Units

forge-cu-mcp

MCP server for Forge compute economy — lets Claude, Cursor, and AI agents interact with CU

terradev-mcp

An imperative command-line-interface for AI workload orchestration

vimin-core

Source-available local AI inference orchestration — broadcast dispatch and multi-step pipelines across up to 10 nodes