The default advice for slow training has become “just throw a bigger A100 at it,” and it quietly wrecks your project ROI. AI/ML data transfer bottlenecks are usually the real cause. I have spent 14 years debugging race conditions and database locks on high-traffic WooCommerce sites, and the shape of the problem is the same: when the GPU sits idle while the CPU sweats over a DataLoader, the architecture is wrong, not the hardware.
Performance talk usually means GPU utilization. The thing that actually costs you money is GPU starvation, where the accelerator waits on the CPU to hand over the next batch. It is a very good chef standing around because the delivery driver is late with the ingredients.
Finding AI/ML data transfer bottlenecks with Nsight
Most devs reach for the PyTorch Profiler first. It handles framework-level debugging well, but it cannot see the whole system. That is what NVIDIA Nsight Systems (nsys) is for: PyTorch shows you kernels, nsys shows you PCIe activity, OS interrupts and DMA transfers. Same gap as staring at one slow WordPress query versus profiling the whole Nginx, PHP-FPM and MySQL stack with Xdebug.
A baseline trace usually shows wide gaps of whitespace along the GPU timeline. That is what AI/ML data transfer bottlenecks look like. The CPU is stuck in a sequential loop: load batch, copy to GPU, run kernels, repeat. Nothing overlaps, so the GPU waits its turn at every step.
Make sure you are chasing the right culprit first. I covered that in a guide on finding the real bottleneck.
Step 1: multi-process data loading
The first mistake is running the DataLoader in a single process. PyTorch loads data in the same process as your training loop unless you tell it otherwise, and that alone will cap your throughput. Set num_workers and you get separate CPU processes fetching batches in parallel, which leaves the training process free to keep feeding the GPU.
# The "Naive" Way (Sequential)
train_loader = DataLoader(dataset, batch_size=64, num_workers=0)
# The Professional Way (Parallel)
# Match this to your vCPU count, but test for diminishing returns.
train_loader = DataLoader(dataset, batch_size=64, num_workers=8)Step 2: pinned memory and async transfers
Even with several workers, the .to(device) call can still stall you. Host memory is pageable by default, meaning the OS is free to move it around. Copying pageable memory to the GPU makes the driver stage it in a temporary pinned buffer first, so every batch gets copied twice.
Set pin_memory=True on the DataLoader and pass non_blocking=True on the transfer. The CPU then queues the copy and carries on with the next task instead of waiting for the GPU to confirm receipt.
# Enabling the fast lane
train_loader = DataLoader(
dataset,
batch_size=64,
num_workers=8,
pin_memory=True
)
def copy_data(batch):
data, targets = batch
# Non-blocking allows the CPU to stay ahead of the GPU
return data.to("cuda", non_blocking=True), targets.to("cuda", non_blocking=True)Pipelining with CUDA streams
Sending data is one thing; pipelining it is where the win is. Modern NVIDIA GPUs have separate engines for memory copy (DMA) and compute kernels (SMs). CUDA Streams let you tell the GPU to copy batch N+1 while it is still crunching batch N, so the two engines work at once instead of taking turns.
If execution is still slow after that, read my notes on fixing slow Python code before you blame the hardware.
Implementation: the prefetcher pattern
Wrapping the loader in a prefetcher class is the tidiest way to do this. The stream synchronization stays in one place and the training loop stays readable.
class bbioon_DataPrefetcher:
def __init__(self, loader):
self.loader = iter(loader)
self.stream = torch.cuda.Stream()
self.next_batch = None
self.preload()
def preload(self):
try:
self.batch = next(self.loader)
with torch.cuda.stream(self.stream):
self.next_batch = [x.to("cuda", non_blocking=True) for x in self.batch]
except StopIteration:
self.next_batch = None
def __next__(self):
torch.cuda.current_stream().wait_stream(self.stream)
batch = self.next_batch
self.preload()
return batchIf AI/ML data transfer bottlenecks are eating your dev hours, I can take it off your plate. I have been wrestling with WordPress since the 4.x days.
What this buys you
A tuned data pipeline is partly about speed and mostly about not paying for idle cloud hardware. Sequential to parallel, pageable to pinned, synchronous to pipelined: those changes have given me over 2x throughput without touching a single hyperparameter. Profile with nsys instead of guessing. The official references are the NVIDIA Nsight documentation and the PyTorch DataLoader specs.
