Mixture Of Experts Python Packages

nvidia-cudnn-frontend

cudnn_frontend provides a c++ wrapper for the cudnn backend API and samples on how to use it

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

mixture-of-experts

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

egobox

Efficient global optimization toolbox in Rust: bayesian optimization, mixture of gaussian processes, sampling methods

smt

Surrogate Modeling Toolbox

optillm

Optimizing inference proxy for LLMs

terradev-cli

NUMA-aware GPU provisioning and orchestration for stateless MoE workloads of all sizes

hivemind

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

peer-pytorch

PEER - Pytorch

krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

st-moe-pytorch

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

abliterix

Automated alignment adjustment for LLMs — direct steering, LoRA, and MoE expert-granular abliteration, optimized via multi-objective Optuna TPE.

dissenter

Multi-LLM debate engine for complex questions — surface disagreement, synthesize decisions

switch-transformers

Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

mlx-flash

Run AI models too large for your Mac's memory — expert caching, speculative execution, and 15+ research techniques for MoE inference on Apple Silicon

pytorch-mixtures

The one-stop solution to easily integrate MoE & MoD layers into custom PyTorch code.

soft-moe-pytorch

Soft MoE - Pytorch

sinkhorn-router-pytorch

Self contained pytorch implementation of a sinkhorn based router, for mixture of experts or otherwise

mixture-of-attention

Mixture of Attention

mlxlmprobe

Universal probing and interpretability tool for MLX language models on Apple Silicon

lmxlab

Transformer language models on Apple Silicon with MLX

neural-trees

sklearn-compatible PyTorch implementations of Soft Decision Trees, HMoE, and classifier comparison tests (5×2cv F-test). pip install neural-trees

mergoo

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

mixturelib

The implementation of mixtures for different tasks.