The default answer to scaling AI is to throw more hardware at the problem, and it quietly wrecks performance. I have watched teams burn thousands in cloud credits while their A100s sat idle, because nobody respected the Host and Device Paradigm. If you don’t know how the CPU and the GPU talk to each other, you are guessing at configuration and calling it engineering.
The relationship is strictly hierarchical. The Host is your CPU: it runs the OS and your Python script, and it gives the orders. The Device is your GPU, a parallel processor that sits there doing nothing until the Host hands it work. That handoff is the basis of high-performance computing, and it is also where most bottlenecks live.
The handoff is asynchronous
The mistake I see most often is treating GPU calls like synchronous PHP functions. Run tensor.to('cuda') and the CPU does not wait for the data to land before moving to the next line. It drops the command in a queue and carries on. That is asynchronous execution, and without it your hardware spends most of its life stalled.
CUDA Streams manage that queue. A stream is a conveyor belt of tasks, and by default everything rides the same belt. Overlapping work means running more than one: while the GPU chews through Batch A, the CPU should already be copying Batch B from RAM to VRAM.
Implementing multiple streams
In PyTorch it looks like this. Leave out non_blocking=True and the transfer blocks the CPU thread, which throws away cycles you paid for.
# The "Architect's" way to overlap data and compute
compute_stream = torch.cuda.Stream()
transfer_stream = torch.cuda.Stream()
# Using separate streams for true parallelism
with torch.cuda.stream(transfer_stream):
# Enqueue the transfer without blocking the CPU
next_batch = next_batch_cpu.to('cuda', non_blocking=True)
with torch.cuda.stream(compute_stream):
# This runs simultaneously on the GPU while Batch N+1 is being copied
output = model(current_batch)The synchronization trap
Call print(gpu_tensor), or branch on something like if tensor.item() > 0, and you force a host-device synchronization. The CPU stops where it is and waits for the GPU to drain its entire queue, all to get one value back into RAM. Inside a tight training loop that can cost you half your throughput or more.
I have mentored devs who could not work out why their “optimized” model ran slower than the plain CPU version. It was accidental sync points every time. You want the GPUs to go brrrrr, and they only do that when the queue stays full and the CPU stops interrupting to ask how things are going.
For how logic-heavy operations behave in a far less exotic setting, there is my take on WordPress Core Performance and AI scripts.
Ranks and distributed logic
Past a single GPU you start dealing with ranks. In the Host and Device Paradigm, a rank is one CPU process bound to one GPU, so four GPUs means four processes. They share no memory. Each is an independent worker, and they talk to each other over the network through a backend like NCCL.
That changes how you build the data pipeline. You are not sending data to a GPU, you are keeping several processes in step through collective operations like AllReduce, and one lagging rank holds up the whole cluster.
If this host and device wrangling is eating your dev hours, I can take it on. I have been working with WordPress, high-scale APIs and awkward architectures since the 4.x days.
What to do about it
The GPU is not a black box. What sits between the Host (CPU) and the Device (GPU) is an ordered queue, so treat it like one:
- Use multiple CUDA streams so I/O and compute overlap.
- Keep
.item()andprint()out of performance-critical loops. - Pass
non_blocking=Trueon host-to-device transfers.
