Gradient accumulation before you buy more GPUs
Getting a larger effective batch size out of the GPUs you already have: gradient accumulation in a PyTorch loop, DDP with no_sync(), why the loss has to be scaled, and the bucket size and data loading settings that decide whether scaling stays linear.