什么是 RDMA？

RDMA stands for Remote Direct Memory Access. It lets one machine read or write another machine's memory directly over the network, without involving either machine's CPU and without copying the data through the operating system kernel. The network adapter moves the bytes straight between application memory on both ends. The result is very low latency and very high throughput with almost no CPU overhead, which is why RDMA is the standard way GPUs exchange data during AI training.

RoCEv2（RDMA over Converged Ethernet 第 2 版）把 RDMA 封装进 UDP 和 IP，使用目的 UDP 端口 4791，因此 RDMA 可跨标准的三层以太网数据中心路由，而无需单独的网络。RDMA 操作仍以适配器到适配器的方式运行，保持同样的零拷贝、CPU 旁路行为；RoCEv2 只是为它提供一条可路由的以太网路径。正是这一点，让多厂商以太网 fabric 成为 GPU 集群中 InfiniBand 的可行替代方案。

AI training runs collective operations such as AllReduce, where every GPU exchanges gradients with every other GPU after each step, moving enormous amounts of data on a tight schedule. If the CPU had to copy every message through the kernel, it would become the bottleneck and burn power doing it. RDMA removes the CPU and the copies from the data path, so a GPU's data lands in the peer GPU's memory with minimal latency, which shortens the exchange every GPU is waiting on.

RDMA 的设计前提是网络不丢包。当一个数据包丢失时，恢复代价高昂并使尾延迟骤增，可能拖停整个 GPU 集合通信。因此基于以太网的 RDMA 运行在一张无损 fabric 上：优先级流控和 ECN 让网络不丢包。OcNOS-DC 在 Broadcom Tomahawk 5 硬件上通过 PFC、ECN 和动态负载均衡提供这张 fabric。详情参见无损以太网和 RoCEv2 页面。