Building a PyTorch DDP training pipeline that scales
Wrapping a model in DDP is the easy part. The rest of the pipeline is where distributed runs go wrong: process group setup, calling set_epoch on the sampler every epoch, and keeping checkpoint writes on rank 0.