RoCEv2 · PFC / ECN · DCQCN

RoCE vs InfiniBand for AI clusters

RoCEv2 runs RDMA over open, multi-vendor Ethernet; InfiniBand is a single-vendor lossless fabric. RoCEv2 stays lossless with PFC and ECN and controls congestion with DCQCN, while InfiniBand is lossless by design through credit-based flow control. For most AI workloads, Ethernet with RoCEv2 now matches InfiniBand while keeping hardware choice. This page compares them axis by axis.

Book an architecture review Open the AI fabric design suite
UDP 4791RDMA routable at Layer 3
up to 800Gopen RoCEv2 fabrics
900+operator networks on OcNOS

The same GPUs, two ways to stay lossless

Left: RoCEv2 over Ethernet keeps queues shallow with hop-by-hop PFC pause, ECN marking on the switch, and DCQCN rate control back at the sending NIC. Right: InfiniBand uses credit-based flow control, sending only when the receiver has advertised buffer credits. Both avoid drops; one on open, multi-vendor Ethernet, the other on a single-vendor fabric.

RoCEv2 PFC/ECN/DCQCN loop versus InfiniBand credit-based flow control Two side-by-side diagrams. Left: a sending GPU NIC and a receiving GPU NIC connected through an Ethernet switch running OcNOS-DC. PFC pause acts hop-by-hop, the switch marks ECN on congestion, and a DCQCN signal returns to the sending NIC to lower its rate. Right: an InfiniBand switch and adapter exchanging buffer credits, so a sender transmits only when the receiver has advertised credit. Bottom band contrasts open multi-vendor Ethernet with a single-vendor InfiniBand fabric. RoCEv2 · OPEN ETHERNET INFINIBAND · SINGLE-VENDOR GPU NICsender SwitchOcNOS-DC · TH5 GPU NICreceiver ECN mark PFC pause (hop-by-hop) DCQCN rate control to sender PFC + ECN keep the fabric lossless · DCQCN controls congestion IB Switchsubnet manager GPU HCAIB adapter data (only if credits > 0) buffer credits Credit-based flow control · lossless by design · single ecosystem OPEN MULTI-VENDOR ETHERNET vs SINGLE-VENDOR INFINIBAND FABRIC

RoCE and InfiniBand, axis by axis

RoCE is RDMA over Converged Ethernet; the version that matters for AI is RoCEv2, which encapsulates RDMA in UDP/IP on destination port 4791 so the traffic is routable across a Layer 3 Ethernet fabric. InfiniBand is a purpose-built interconnect from the InfiniBand Trade Association, lossless by architecture through credit-based flow control. The practical question for an AI cluster is which transport meets the workload while keeping cost, supply chain, and operations sane.

Axis RoCERoCEv2 over Ethernet, open InfiniBandsingle-vendor fabric
Transport & standardRDMA over UDP/IP (RoCEv2, IBTA), routable at Layer 3 on destination UDP port 4791. Rides on standard IEEE Ethernet.Purpose-built InfiniBand link and transport layers (IBTA), with dedicated switches and host channel adapters.
How it stays losslessPFC for hop-by-hop flow control plus ECN with DCQCN for end-to-end congestion control, keeping switch queues shallow so RDMA avoids drops.Credit-based flow control: a sender transmits only against advertised receiver buffer credits, so the fabric is lossless by design.
EcosystemOpen, multi-vendor at every layer: switch ASIC, switch, NIC, NOS, and optics from independent suppliers.Effectively single-vendor for NIC and switch silicon; the fabric, adapters, and management come as one stack.
Switch silicon choiceBroadcom Tomahawk 5 (51.2 Tbps, 64x800G) and Tomahawk 4 (25.6 Tbps, 64x400G) across Edgecore and UfiSpace platforms, among others.Silicon from the InfiniBand vendor only; no independent ASIC or switch supply.
Congestion controlDCQCN end-to-end: the switch marks ECN and the sending NIC lowers its rate. Dynamic load balancing (DLB) spreads flows and rebalances on local congestion.Credit-based flow control with adaptive routing built into the specification.
ScaleOn Tomahawk 5, a 2-tier fabric reaches 4,096 GPUs at 800G per GPU, 1:1 non-blocking; adding a super-spine tier extends designs up to 16,384 GPUs.Scales through fat-tree topologies managed by a centralized subnet manager within one vendor's fabric.
Operations & toolingStandard Ethernet operations: BGP, EVPN, gNMI/OpenConfig telemetry, and the same automation (Ansible, NETCONF) used across the data center.Specialized InfiniBand tooling and a subnet manager, separate from the rest of the data center's operational model.
Vendor lock-inNone inherent: mix and match hardware and software, and second-source the supply chain.High: NIC, switch, cabling, and management typically come from a single vendor.

InfiniBand is a trademark of the InfiniBand Trade Association. NVIDIA and Mellanox are trademarks of their respective owners. This comparison is provided for informational purposes and does not imply affiliation or endorsement.

Where each one wins

The right answer is workload-specific. Pick InfiniBand where an absolute latency floor is the requirement; pick RoCE where the operating model, cost, or reach across sites carries the decision.

InfiniBand fits when

The latency floor is the spec

Tightly coupled HPC simulation where the absolute NIC-to-NIC latency floor matters more than total cost of ownership, and captive single-tenant clusters where a single-vendor stack is acceptable.

RoCE fits when

Operations and supply chain matter

Multi-tenant GPU-as-a-Service and AI back-ends that share one operational model, one tooling stack, and a multi-vendor supply chain with the rest of the data center.

RoCE fits when

Cost per GPU is the gate

OcNOS-DC runs on open hardware from multiple vendors, so the switch hardware can be sourced competitively.

RoCE fits when

The fabric spans data centers

Ethernet extends naturally across halls and regions with 400G ZR/ZR+ coherent optics, without a separate long-haul gateway layer.

How OcNOS-DC delivers the RoCEv2 path

OcNOS-DC provides the lossless Ethernet substrate that RoCEv2 requires. The switch supplies the loss-avoidance and congestion primitives; RoCEv2 is the transport that runs on the NIC.

Losslessness

PFC and ECN

Priority Flow Control, including PFC-with-QoS over Layer 3 with DCBX/LLDP, plus ECN, and Dynamic ECN on Tomahawk 5, so RDMA flows keep shallow queues without drops.

RoCEv2 lossless fabric
Congestion

DCQCN marking

The switch marks ECN so the end-to-end NIC-plus-switch DCQCN loop can rate-limit senders. ETS and WRED shape and manage the queues that carry RDMA.

DCQCN congestion control
Load balancing

DLB and RTAG7

Dynamic Load Balancing (including Reactive Path Rebalance and Random Flow on Tomahawk 5) with RTAG7 hashing spreads collectives and rebalances on local congestion.

Dynamic load balancing (DLB)
Resilience

PFC deadlock protection

PFC Deadlock Detection and Recovery plus buffer tuning keep the lossless fabric stable under the pause propagation that PFC can create.

PFC deadlock recovery
Telemetry

gNMI / OpenConfig

Streaming gNMI and OpenConfig telemetry give visibility for closed-loop tuning during cluster bring-up and steady-state operation.

OcNOS feature matrix
Standards

Ultra Ethernet Consortium member

IP Infusion is a member of the Ultra Ethernet Consortium (UEC). UEC 1.0 defines a multipath Ethernet transport that spreads one flow across many paths, running on UEC-capable NICs.

Ultra Ethernet (UEC)

Silicon choice is open: OcNOS-DC runs on the Edgecore AIS800-64D and UfiSpace S9321-64E / S9321-64EO (the -64EO adds 400G ZR+) on Broadcom Tomahawk 5, and the Edgecore AS9736-64D on Tomahawk 4, all on-chip and shared-buffer platforms. One IP Infusion contract covers the software and the qualified hardware on OcNOS Systems, with one TAC and one SLA.

Why many operators move to Ethernet for AI back-ends

The shift is operational and economic more than it is about raw latency. Three things add up fast on a large build: one network model shared with the rest of the data center, a multi-vendor supply chain, and lower cost per port at 400G and 800G.

The scale is proven in the field. Meta has described training its largest models over a RoCE Ethernet fabric on a 24,576-GPU cluster, a size once assumed to require InfiniBand. Dynamic load balancing on Tomahawk 4 and Tomahawk 5 rebinds flows on real-time congestion, so a well-run fabric can target utilization above 90 percent where static hashing would strand it.

InfiniBand's remaining edge, a lower absolute latency floor, still matters for a set of tightly coupled HPC workloads. For most distributed-training and inference collectives at scale, a correctly tuned RoCEv2 fabric sits below the threshold that changes job completion time, which is why the operational and economic case now decides most new AI back-end builds.

  • One network model shared with the rest of the data center, so skills and tooling carry over.
  • A multi-vendor supply chain that lets you second-source silicon, switches, NICs, and optics.
  • Lower cost per port at 400G and 800G, freeing capital for more GPUs, a larger storage tier, or a second site.

Both are valid, and most new AI back-ends land on Ethernet

InfiniBand keeps a lower absolute latency floor that a set of tightly coupled HPC workloads will keep paying for. For most AI back-end builds, the operational and economic case decides in favor of RoCEv2 over open Ethernet.

Both are valid

InfiniBand keeps a lower absolute latency floor that a set of tightly coupled HPC workloads will keep paying for.

RoCEv2 has closed the gap for AI

With PFC, ECN, DCQCN, and dynamic load balancing configured correctly, Ethernet matches InfiniBand on most distributed-training collectives while keeping hardware choice.

The decision is usually operational

One network model, a multi-vendor supply chain, and lower cost per port at 400G and 800G decide most new AI back-end builds.

OcNOS-DC is the open path

RoCEv2 primitives today on qualified open hardware from Edgecore and UfiSpace, with one contract, one TAC, and one SLA on OcNOS Systems.

Resource

Get the AI Fabric Reference Design

The full PDF: lossless RoCEv2 topology, switch and transceiver counts, qualified platforms, and a DCQCN starting profile on OcNOS-DC.

Download PDF

RoCE vs InfiniBand FAQ

What is the difference between RoCE and InfiniBand?
RoCE (in practice RoCEv2) carries RDMA over standard, routable Ethernet and IP, so it reuses the switches, NICs, optics, and operational tooling any vendor can supply. InfiniBand is a separate purpose-built fabric with its own switches, adapters, cabling, and subnet manager from a single vendor. RoCEv2 keeps the fabric lossless with PFC and ECN and controls congestion with DCQCN, while InfiniBand relies on credit-based flow control that is lossless by design.
Is Ethernet as fast as InfiniBand for AI?
For most distributed-training collectives at scale, yes. InfiniBand still has a lower absolute latency floor by hundreds of nanoseconds, but once RoCEv2 is configured with PFC, ECN, DCQCN, and dynamic load balancing, that gap sits below the level that changes job completion time for the majority of AI workloads. Many operators now run large AI back-end networks on Ethernet for that reason.
What is RoCEv2?
RoCEv2 (RDMA over Converged Ethernet version 2) encapsulates RDMA in UDP/IP on destination port 4791, so RDMA traffic is routable across Layer 3 Ethernet fabrics. It lets GPUs and storage move data directly between memory with low latency and low CPU overhead, which is why it is the mainstream way to run RDMA on an open, multi-vendor Ethernet network.
Does RoCE need a lossless fabric?
Yes. RoCEv2 performance drops sharply when packets are dropped, so it needs a lossless or near-lossless fabric. Operators provide that with PFC for hop-by-hop flow control and ECN with DCQCN for end-to-end congestion control, keeping switch queues shallow so RDMA flows avoid drops and retransmits. The switch supplies the lossless substrate; RoCEv2 is the transport that runs on the NIC.
Can I run RoCE on open switches?
Yes. RoCEv2 runs on standard Ethernet silicon. OcNOS-DC delivers the RoCEv2 building blocks (PFC, PFC-with-QoS over Layer 3, ECN, ETS, WRED, DLB, PFC deadlock detection and recovery, and gNMI/OpenConfig telemetry) on qualified open hardware such as Edgecore and UfiSpace platforms built on Broadcom Tomahawk 4 and Tomahawk 5, with Dynamic ECN on Tomahawk 5. One IP Infusion contract covers the software and the qualified hardware on OcNOS Systems.
What about Ultra Ethernet?
Ultra Ethernet (UEC) is the open standard that brings packet spray, multi-path RDMA, out-of-order delivery, and modern congestion control to Ethernet, narrowing what InfiniBand once did uniquely. IP Infusion is a member of the Ultra Ethernet Consortium, and OcNOS-DC runs lossless RoCEv2 with PFC, ECN, and DLB in production today on Broadcom Tomahawk 4 and Tomahawk 5 switches.

Go deeper. Take it with you.

The product datasheet plus short, technical downloads that go further than this page.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Lossless RoCEv2 fabric on Tomahawk 5 spines: SKU tiers, qualified platforms, and deployment architecture.

Get the brief
Solution Brief

EVPN-VXLAN Data Center Fabric

Leaf-spine data center fabric on EVPN-VXLAN: a VRF and VNIs per tenant, EVPN multihoming, BGP unnumbered and route-target filtering.

Get the brief

Sizing a RoCEv2 fabric against InfiniBand? Let's run the workload-specific maths

Tell us the workload and the GPU scale, and an IP Infusion engineer will run the numbers with you, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.

Book an architecture review Open the AI fabric design suite

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs AI Fabric Topologies Rail-optimized two-tier and three-tier fabrics, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA? AI networking switches
The technology inside RoCEv2 lossless Rail-optimized network DLB GLB (7.1) Ultra Ethernet Tomahawk 6 at 1.6T DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI