Model Parallelism Python Packages

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

substrax

JAX/Flax NNX training infrastructure: device meshes and SPMD sharding, an Orbax checkpoint store, early stopping and callbacks, W&B and MLflow logging.

colossalai

Making large AI models cheaper, faster and more accessible

colossalai-nightly

Making large AI models cheaper, faster and more accessible

redco

NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference

torchgpipe

A GPipe implementation in PyTorch

fleet-x

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

custom-colossalai

Making large AI models cheaper, faster and more accessible

fleet-lightning

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

adeepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

torchshard

Slicing a PyTorch Tensor Into Parallel Shards

veloce

WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.

paddle-fleet

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。