#Model-parallelism

deepspeedai

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

hpcaitech

ColossalAI

Making large AI models cheaper, faster and more accessible

kakaobrain

torchgpipe

A GPipe implementation in PyTorch

hkproj

pytorch-transformer-distributed

Distributed training (multi-node) of a Transformer model