What Is RDMA?

RDMA stands for Remote Direct Memory Access. It lets one machine read or write another machine’s memory directly over the network, without involving either machine’s CPU and without copying the data through the operating system kernel. The network adapter moves the bytes straight between application memory on both ends. The result is very low latency and very high throughput with almost no CPU overhead, which is why RDMA is the standard way GPUs exchange data during AI training.

RoCEv2 (RDMA over Converged Ethernet, version 2) encapsulates RDMA in UDP and IP, using destination UDP port 4791, so RDMA can be routed across a standard Layer 3 Ethernet data center rather than needing a separate network. The RDMA operation still runs adapter to adapter with the same zero-copy, CPU-bypass behavior; RoCEv2 just gives it a routable Ethernet path. This is what makes a multi-vendor Ethernet fabric a viable alternative to InfiniBand for GPU clusters.

AI training runs collective operations such as AllReduce, where every GPU exchanges gradients with every other GPU after each step, moving enormous amounts of data on a tight schedule. If the CPU had to copy every message through the kernel, it would become the bottleneck and burn power doing it. RDMA removes the CPU and the copies from the data path, so a GPU’s data lands in the peer GPU’s memory with minimal latency, which shortens the exchange every GPU is waiting on.

RDMA was designed assuming the network does not drop packets. When a packet is lost, recovery is expensive and spikes tail latency, which can stall a whole GPU collective. So RDMA over Ethernet runs on a lossless fabric: Priority Flow Control and ECN keep the network no-drop. OcNOS-DC provides that fabric with PFC, ECN, and dynamic load balancing on Broadcom Tomahawk 5 hardware. See the lossless Ethernet and RoCEv2 pages for detail.