PyRank

Distributed Inference Python Packages

Python packages with the GitHub topic distributed-inference. Sorted by monthly downloads, with stars and forks.
flashinfer-ai
flashinfer-python

FlashInfer: Kernel Library for LLM Serving

3.1M 7K 2K
flashinfer-ai
flashinfer-cubin

FlashInfer: Kernel Library for LLM Serving

1.2M 7K 2K
sgl-project
sglang-omni

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

3K 1K 565
BenevolentJoker-JohnL
sollol

Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics

987 6 3
drc10101
sawyer-core

Distributed MoE inference network — the load is split, friends help.

533 0 0
youngharold
tightwad

Mixed-vendor GPU inference cluster manager with speculative decoding

325 38 5
clearclown
forge-cu-mcp

Tirami — distributed LLM inference where compute is currency. 1 TRM = 10^9 FLOPs. 21B supply cap, yield halving, staking, collusion resistance. 100% Rust, OpenAI-compatible, no token, no ICO. "tira mi su" = pull me up.

237 1 0
mzbac
mlx-sharding

Distributed Inference for mlx LLm

120 105 10
clearclown
forge-sdk

Tirami — distributed LLM inference where compute is currency. 1 TRM = 10^9 FLOPs. 21B supply cap, yield halving, staking, collusion resistance. 100% Rust, OpenAI-compatible, no token, no ICO. "tira mi su" = pull me up.

96 1 0
pberlizov
vimin-core

Run open-source LLMs on your own machines, offline. Nothing leaves your network: no cloud, no telemetry.

48 0 0