A lot of the performance audits I take on lately look the same: thousands of dollars of cloud credits going into H100s or A100s, and GPU utilization that traces a sawtooth wave. The client wants to know how much more hardware to buy. The answer is usually none, because the data pipeline is the part that needs work. After 14 years in high-load environments, whether that is a WooCommerce checkout under load or a RAG-based AI assistant, the bottleneck is rarely where anyone expects it.
Hardware is not the fix
Treating hardware as a bandage for slow code has become normal in AI work. A GPU sitting at 10% while every CPU core is pinned at 100% is rented compute you are not using. The GPU is a very fast calculator and it depends on the CPU to feed it batches across the PCIe bridge, so slow data loading means the card spends most of its life waiting for the next tensor to show up.
I hit the same throughput problem writing about scaling LLMs and prompt caching. A pipeline runs at the speed of its slowest stage, and that stage is almost never the GPU.
Solving the CPU to GPU data bottleneck
Better GPU utilization starts with parallelizing the feeder. A PyTorch DataLoader runs in the main process by default, so the training loop goes load, preprocess, move to GPU, compute, one step at a time. The GPU idles while the CPU loads, and the CPU idles while the GPU computes. Both halves of the machine spend half their time waiting on the other.
# The "Naive" Approach - Sequential and Slow
train_loader = DataLoader(dataset, batch_size=32, num_workers=0)
# The "Senior Dev" Approach - Parallelized
train_loader = DataLoader(
dataset,
batch_size=64, # Power of 2
num_workers=4, # Parallelize data prep
pin_memory=True, # Fast lane for PCIe transfer
prefetch_factor=2 # Queue up batches in advance
)pin_memory=True matters more than it looks. It allocates your data in page-locked memory so the GPU can pull it with Direct Memory Access (DMA) instead of routing everything through the CPU. num_workers spawns background processes that keep VRAM stocked. One caveat: set workers above your physical core count and performance drops, because the box spends the difference on context switching.
The same pattern showed up in what I wrote on host memory bottlenecks. Memory management is usually what decides whether a project ships or falls over under load.
Mixed precision and kernel efficiency
With the data on the device, the next question is whether the compute kernels are doing useful work. Most people still default to FP32, which is more precision than 99% of machine learning tasks need. Modern NVIDIA cards ship Tensor Cores built for 16-bit math, and without mixed precision (FP16 or BF16) those cores sit unused.
The tool for that is torch.autocast. It picks which operations keep high precision, Softmax among them, and which can run faster in half precision. That alone can double your throughput without touching a line of model logic. The NVIDIA Mixed Precision Training Guide goes through the details.
Then there is torch.compile(). Every operation carries kernel overhead: the GPU reads from VRAM, computes, writes back. PyTorch 2.0 and later can fuse those kernels and cut the round trips to VRAM. It costs you almost nothing to try, and a string of sequential operations turns into one optimized block.
If GPU utilization is eating your dev hours, hand it over. I have been working with WordPress since the 4.x days and have scaled systems across a few different architectures.
What to change first
If GPU utilization is the problem, look at the data pipeline before you look at the card. num_workers parallelizes the loading, pin_memory speeds up the transfer, and torch.compile fuses the operations. The PyTorch Performance Tuning Guide covers the API side in more depth than I can here.
