I thought I had seen every way a distributed system can choke, then the early benchmarks for Intel’s Gaudi accelerators on AWS DL1 instances landed. The numbers came back terrible. When you scale AI models across dozens of nodes, a 50% performance drop is not a tuning problem. Sitting under it was a Host Memory Bottleneck, the product of a pragmatic business decision that ignored how systems engineering works.
The hardware itself is impressive. Gaudi chips carry ten 100 Gbps network interfaces in the silicon, built for native RDMA over RoCE v2 so the chips can talk to each other without waking the CPU. Cloud environments rarely match that architecture. To keep DL1 instances affordable, AWS wired them with standard host NICs instead of Gaudi’s integrated networking, so every byte took a detour through the host CPU and DRAM. Anyone who read my earlier notes on scaling large models knows that this kind of overhead is what kills distributed training.
The architecture of a host memory bottleneck
In a native RDMA setup, memory moves from card A to card B. With the Host Memory Bottleneck in the way, the path was Gaudi Memory > Host DRAM > CPU > TCP/IP stack > Host NIC > Network, then the same mess again on the receiving end. That detour ate CPU cycles and added latency, and the scalability of collective operations like AllReduce, which GPT-scale training leans on constantly, evaporated with it.
Peer Direct was our answer: simulate RDMA performance on host NICs that were never built for it. That meant integrating the AWS Elastic Fabric Adapter (EFA), libfabric, and Habana’s Collective Communication Library (HCCL), then using the Linux kernel DMA-BUF framework to share device buffers straight with the network layer.
Refactoring the control path
The first thing that bit us was the cost of memory registration. You cannot just point a NIC at data, you have to register the memory region first, and that means expensive kernel calls. Do it on every transfer and the registration overhead alone buries your throughput. We ended up with a dirty but effective LRU (Least Recently Used) cache for memory registrations.
// Conceptual Logic for the HCCL Registration Cache
// We side-stepped the Host Memory Bottleneck by re-using libfabric handles
void* bbioon_get_registered_handle(void* gaudi_ptr, size_t size) {
if (registration_cache.exists(gaudi_ptr)) {
return registration_cache.get(gaudi_ptr); // Instant hit
}
// Expensive kernel registration via DMA-BUF
int dma_fd = gaudi_driver_get_fd(gaudi_ptr, size);
fi_mr_regattr attr = { .mr_fd = dma_fd, .flags = FI_MR_DMABUF };
return bbioon_register_with_libfabric(&attr);
}Caching those mappings took the control path out of the critical loop. The same thing shows up in my performance audits: the biggest gains usually come from deleting unnecessary detours, not from micro-optimizing the fast path.
Lessons from the war room
Building Peer Direct was as much an operational problem as a technical one. The AWS engineers we worked with were 12 hours ahead, which gave every debugging iteration a 24-hour turnaround. libfabric was not yet mainstream in the AI accelerator world either, and the documentation was thin, so we spent nights reading its source. Messy, but it worked. Once Peer Direct went live, throughput on large message sizes doubled.
If this host memory bottleneck work or any messy backend optimization is eating your dev hours, hand it to me. I have been wrestling with high-performance systems and WordPress since the 4.x days.
What to check first
If your distributed system is not scaling, do not start with the model architecture. Micro-benchmark the network topology instead. Efficiency that drops as you add nodes points at the data path. In the cloud, assumptions about direct access are usually wrong, and you end up building the bridge yourself, the way we did with Peer Direct.
