Distributed training data transfer: fixing the multi-GPU stall
Multi-GPU training usually stalls on communication, not compute. What nvidia-smi topo and Nsight show you about NCCL overhead, and the DDP settings I change: static graph, bucket views, PowerSGD compression and smaller buckets.