RoCEv2: Lossless Ethernet for AI Fabrics
RoCEv2 is how RDMA runs on standard Ethernet: it encapsulates RDMA in UDP/IP and uses UDP destination port 4791, so GPU collective traffic is routable across a normal leaf-spine fabric. It needs a lossless fabric built from PFC and ECN, with DCQCN as the congestion-control loop, and OcNOS delivers that full toolkit today on supported 400G and 800G open hardware.
The same GPUs on a lossless leaf-spine fabric
A compact rail slice: two spines and two leaves carrying RoCEv2 between four GPUs. PFC pause frames travel hop by hop on congestion, while ECN marks elephant flows for DCQCN reaction at the source. RDMA is encapsulated in UDP/IP on destination port 4791, so the same traffic is routable across a standard Ethernet fabric.
Near-zero loss keeps GPU collectives efficient
GPU collectives (all-reduce, all-gather, all-to-all) generate elephant flows that saturate single fabric paths and demand near-zero loss to keep training jobs efficient. Drop a single packet on a 400G RoCEv2 link and the affected NIC re-transmits the entire RDMA send window, measurable as seconds of GPU idle time. RoCEv2 turns a leaf-spine fabric into a lossless transport for these workloads by encapsulating RDMA in UDP/IP on destination port 4791, so the same traffic is routable across Layer 3.
| Parameter | RoCEv1Layer 2 | RoCEv2UDP/IP, routable |
|---|---|---|
| Encapsulation | RDMA carried directly over Ethernet (dedicated EtherType). | RDMA encapsulated in UDP/IP, using UDP destination port 4791. |
| Network layer | Layer 2 only; confined to a single broadcast domain. | Layer 3 routable across a standard leaf-spine fabric. |
| ECMP entropy | No IP or UDP header to hash; limited path spreading. | UDP source port varied as a flow identifier to spread traffic across ECMP paths. |
| Lossless requirement | Needs a lossless fabric built from PFC and ECN. | Needs a lossless fabric built from PFC and ECN, with DCQCN as the congestion-control loop. |
| Reach and scale | Within a rack or single subnet. | Routed data center fabrics; reference designs for 256 to 4,096 GPU clusters. |
The controls that keep the fabric lossless
A RoCEv2 fabric stays lossless through three cooperating mechanisms: priority flow control pauses the right traffic class, ECN and DCQCN hold throughput without loss, and adaptive routing keeps the few large flows off congested uplinks.
Per-priority pause
802.1Qbb PFC pauses a single traffic class hop by hop so the queues carrying RDMA never drop. OcNOS pairs it with a PFC deadlock watchdog that detects a stalled priority and auto-recovers before it propagates.
ECN marking and the DCQCN loop
WRED-based ECN marks packets as queues build, and DCQCN is the congestion-control loop that reacts at the source to hold throughput without loss. Tuned defaults for xCCL collectives, with parametric override for custom RDMA stacks.
Adaptive flowlet routing
Static ECMP hashing collides on the few large elephant flows AI collectives produce. DLB re-bins flowlets on local link saturation in sub-millisecond windows, removing the static hashing collisions that hurt symmetric topologies.
RoCEv2 as OcNOS delivers it
Beyond the lossless controls, OcNOS ships the telemetry, reference designs, and a clean upgrade path that turn a lossless config into an operable fabric.
Per-priority queue stats
gNMI streaming sensors for queue depth, PFC pause counters, ECN-marked packets, and microburst detection, exported on a 10-second sample interval for fabric-wide observability.
Rail-optimized fabrics
Reference designs for rail-aligned and scheduled-fabric topologies, covering 256 to 4,096 GPU clusters on off-the-shelf 400G and 800G open switches. CLI diagnostics verify a known-good lossless config end to end.
A clean path to Ultra Ethernet
Build lossless RoCEv2 today and keep an open path to Ultra Ethernet, which adds packet spray and multi-path RDMA to standard Ethernet as UEC NICs ship. One NOS carries both.
One NOS image across open hardware
The RoCEv2 toolkit is part of the base OcNOS-DC license, not a set of paid add-ons, and it runs unchanged across a multi-vendor hardware choice.
- Open hardware choice. Run RoCEv2 on UfiSpace, Edgecore, or Celestica platforms with the same NOS image, so the fabric layer carries no vendor lock-in.
- Day-one feature parity. Adaptive load-balancing, DCQCN tuning, and ASIC-native telemetry are part of the base OcNOS-DC license, not paid add-ons.
- Reference designs. Reference configs for popular AI fabric topologies, with the configs and the test results published.
- Engineering access. The premium support tier includes direct dialog with the OcNOS RoCEv2 team during fabric bring-up.
Standard Ethernet, RDMA performance, open hardware
RoCEv2 lets an AI fabric reuse the Ethernet operations and gear the rest of the data center already runs, and OcNOS is the enabler that makes it lossless on open switches.
Standard Ethernet, RDMA speed
RoCEv2 carries RDMA over routable UDP/IP, so GPU collectives get low-latency, low-CPU data movement without a separate purpose-built fabric.
Open hardware choice
The same lossless config runs across UfiSpace, Edgecore, and Celestica switches at 400G and 800G, keeping the fabric layer multi-vendor.
OcNOS is the enabler
PFC, ECN, DCQCN, DLB, and per-priority telemetry ship as one NOS on open switches, so the complete lossless fabric is a build, not an integration project.
RoCEv2, answered
What is RoCEv2?
What is the difference between RoCEv2 and RoCEv1?
Does RoCEv2 need a lossless network?
What UDP port does RoCEv2 use?
How does RoCEv2 compare to InfiniBand?
Go deeper. Take it with you.
The product datasheet plus short, technical downloads that go further than this page.
OcNOS-DC Datasheet
Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.
Get the datasheetOcNOS 800G Lossless AI Fabric
Non-blocking RoCEv2 fabric on Broadcom Tomahawk 4/5 spines: SKU tiers, validated platforms, and deployment architecture.
Get the briefEVPN-VXLAN Data Center Fabric
Carrier-grade leaf-spine data center fabric: symmetric IRB, Type-2/Type-5 routes, and distributed anycast gateway.
Get the briefOcNOS-DC Datasheet
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
OcNOS 800G Lossless AI Fabric
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
EVPN-VXLAN Data Center Fabric
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
Building or scaling an AI fabric? Get a workload-specific review
Tell us the GPU scale and the collective pattern, and an IP Infusion engineer will size the switch tiers and tune the lossless config with you, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.
Design the whole AI fabric with OcNOS
From the business case to the port-count maths, pick up wherever you are in the build.