Adding hardware fixes a lot of scaling problems. In WordPress work, a high-traffic store gets object caching and a load balancer and it holds. Training or fine-tuning a large language model does not behave that way: standard Distributed Data Parallelism (DDP) runs into a limit that extra GPUs cannot lift. The limit is VRAM redundancy, and it eats the budget before you finish the first epoch.
The memory redundancy tax in DDP
Under standard DDP, every GPU in the cluster holds a full replica of the model parameters, the gradients and the optimizer states. For a 7B-parameter model with Adam in FP32, that is not just the weight size. It comes to about 112 GB of VRAM per card before any training starts, which is how an 80GB A100 manages to throw an Out-of-Memory error on the first micro-batch.
ZeRO memory optimization (Zero Redundancy Optimizer) goes after exactly that. Rather than replicating everything everywhere, it partitions those states across the GPUs. The shift is the one databases went through years ago, from every node keeping a full copy to a sharded setup.
ZeRO stages: sharding the infrastructure
Microsoft’s ZeRO comes in three stages, each freeing more VRAM than the last by taking on a different piece of the redundancy:
- ZeRO-1 shards the optimizer states only. Parameters and gradients are still replicated on every GPU.
- ZeRO-2 shards the optimizer states and the gradients. For mid-sized training runs this is usually where people settle.
- ZeRO-3 shards the parameters too, so the full model only exists during the forward and backward pass.
If communication overhead between your cards is already hurting, read my guide on solving GPU-to-gpu communication bottlenecks in AI before you go near stage 3.
Implementing ZeRO-1/2 logic
The sharding logic is easiest to follow at the parameter level. Each rank manages its own slice instead of running a global update, and that puts a lot of weight on collective operations such as all_reduce and all_gather.
# Simplified parameter sharding logic
def bbioon_shard_parameters(model, world_size, rank):
for param in model.parameters():
numel = param.data.numel()
shard_size = (numel + world_size - 1) // world_size
start = rank * shard_size
end = min(start + shard_size, numel)
# In a real ZeRO-3 implementation, we would free the non-local data here
local_shard = param.data.view(-1)[start:end].clone()
return local_shardPyTorch FSDP (Fully Sharded Data Parallel)
You can build ZeRO yourself on top of torch.distributed, but in practice ZeRO memory optimization now means PyTorch FSDP. It wraps your model layers and overlaps the communication for you, so you are not hand-orchestrating the all_gather that has to land right before the forward pass.
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.fully_shard import fully_shard
# Wrap your transformer layers to enable JIT parameter gathering
for layer in model.layers:
fully_shard(layer)
# Wrap the full model
fsdp_model = FSDP(model)That just-in-time parameter gathering is what lets a 7B model run in 14 GB of VRAM rather than 112 GB. FSDP2, the current version, also leaves more room to tune, including CPU Offloading to push optimizer states out to system RAM when the GPUs really are full.
If ZeRO memory optimization is eating your dev hours, I can take it on. I have been working with WordPress since the 4.x days, and I know how to join heavy backend infrastructure to AI services that have to scale.
The trade-off: latency against memory
ZeRO is not a free lunch. You are trading network bandwidth for VRAM, so on a slow interconnect, NVLink or plain Ethernet, the GPUs sit waiting for parameter shards instead of computing. ZeRO-3 raises communication volume by 50% compared with standard DDP. The Microsoft ZeRO paper and the official PyTorch FSDP documentation both go further than this.
