Moe Python Packages

sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

flashinfer-python

FlashInfer: Kernel Library for LLM Serving

flashinfer-cubin

FlashInfer: Kernel Library for LLM Serving

nvidia-cudnn-frontend

cudnn_frontend provides a c++ wrapper for the cudnn backend API and samples on how to use it

sglang-kernel

SGLang is a high-performance serving framework for large language models and multimodal models.

sgl-kernel

SGLang is a high-performance serving framework for large language models and multimodal models.

llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

vllm-tpu

A high-throughput and memory-efficient inference and serving engine for LLMs

ms-swift

Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-R1, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).

tensorrt-llm

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

terradev-cli

NUMA-aware GPU provisioning and orchestration for stateless MoE workloads of all sizes

sglang-kt

SGLang is a high-performance serving framework for large language models and multimodal models.

awex

A high-performance RL training-inference weight synchronization framework, designed to enable second-level parameter updates from training to inference in RL workflows

uccl

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

abliterix

Automated alignment adjustment for LLMs — direct steering, LoRA, and MoE expert-granular abliteration, optimized via multi-objective Optuna TPE.

switch-transformers

Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

mlx-flash

Run AI models too large for your Mac's memory — expert caching, speculative execution, and 15+ research techniques for MoE inference on Apple Silicon

llmtuner

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

plato-edge

Edge-optimized Cocapn fleet packages for ARM64 devices with limited resources

lazyllm-llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

ai-dynamo-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

dblcsgen

SGLang is a high-performance serving framework for large language models and multimodal models.

vllm-acc

A high-throughput and memory-efficient inference and serving engine for LLMs