RoCEv2 · DLB · Ultra Ethernet

InfiniBand vs Ethernet for AI Fabrics

For most AI fabrics, Ethernet. For latency-critical HPC, InfiniBand. Modern Ethernet with RoCEv2 now runs production fabrics at 400G and 800G on open, multi-vendor hardware, and OcNOS-DC delivers it today. This guide covers where each one still fits.

Book an architecture review Open the AI fabric design suite
Two-thirdsEthernet share, AI scale-out
up to 800Gproduction RoCEv2 fabrics
1 NOSOcNOS-DC on open hardware
900+operator networks on OcNOS

The same GPUs, two very different networks

On the left, a single-vendor InfiniBand fabric: one silicon vendor, one switch line, one NIC ecosystem, and a subnet manager separate from the rest of the data center. On the right, a multi-vendor open Ethernet fabric: RoCEv2 NICs from any vendor, Broadcom switch silicon, OcNOS-DC as the NOS, and the same protocols you already run.

InfiniBand single-vendor fabric vs multi-vendor Ethernet fabric Two side-by-side fabric topologies. Left: a single-vendor InfiniBand fabric with two IB switches and three GPUs. Right: a multi-vendor Ethernet fabric with two OcNOS-DC spines and three GPUs with RoCEv2 NICs. Bottom labels contrast a single-vendor stack with an open multi-vendor stack. INFINIBAND · SINGLE-VENDOR ETHERNET · OPEN MULTI-VENDOR IB Switch-1InfiniBand IB Switch-2InfiniBand GPUIB NIC GPUIB NIC GPUIB NIC SINGLE NIC + SWITCH VENDOR SUBNET MANAGER · LIMITED MULTI-TENANCY Spine-1OcNOS-DC · TH5 Spine-2OcNOS-DC · TH5 GPURoCEv2 GPURoCEv2 GPURoCEv2 MULTI-VENDOR · OPEN HARDWARE RoCEv2 · DLB · EVPN-VXLAN · gNMI SINGLE-VENDOR STACK vs OPEN MULTI-VENDOR STACK

InfiniBand and Ethernet, axis by axis

InfiniBand was purpose-built for low-latency, lossless RDMA, and for two decades that gave it a real edge for tightly coupled HPC. Modern Ethernet, built on the DCB stack, RoCEv2, and increasingly DLB and UEC, has spent the last several years closing that gap. Which gaps still matter depends on the workload.

Axis InfiniBandsingle-vendor EthernetRoCEv2 / UEC, open
Latency floorVery low end-to-end NIC-to-NIC; typical hundreds of nanoseconds switch hop.Higher floor than IB by hundreds of nanoseconds, but well below the threshold that affects most distributed-training collectives at scale.
Loss toleranceLossless by architecture (credit-based flow control).Lossless via PFC + ECN + DCQCN. Production-grade today; UEC further reduces dependence on PFC pause.
Multi-path / load balancingAdaptive routing built into the spec.Static ECMP, plus DLB for congestion-aware single-hop balancing, GLB (OcNOS 7.1) for end-to-end, UEC packet-spray for next-gen.
Vendor ecosystemEffectively single-vendor for both NIC and switch silicon.Multi-vendor at every layer: ASIC, switch, NIC, NOS, optics. UEC is explicitly designed for vendor-neutral interop.
Operational modelSubnet manager. Different from rest of DC. Separate skills, separate tooling.Same BGP, EVPN, gNMI you already run. Same automation tools (Ansible, NETCONF, OpenConfig) as the rest of DC.
Multi-tenancyLimited; partitioning exists but is not a first-class concept.First-class via EVPN-VXLAN. GPU-as-a-Service, multi-team clusters, shared infra all natural.
Long-haul DCINot designed for it; needs IB-over-WAN gateways.Native via 400G ZR/ZR+ coherent pluggables and EVPN inter-DC.
Storage convergenceStorage runs alongside compute; needs IB-attached storage.NVMe-oF, NFS, S3 all over the same Ethernet fabric.
Roadmap velocityDriven by one vendor's release cadence.The UEC consortium, whose members span silicon, NIC, switch, and cloud companies, drives openly published spec evolution.

Where each one wins

The right answer is workload-specific. Pick InfiniBand where an absolute latency floor is the requirement; pick Ethernet where the operating model, cost, or reach across sites carries the decision.

Pick InfiniBand when

The latency floor is contractual

HPC simulation where the absolute latency floor matters more than total cost of ownership, and tight, captive single-tenant clusters where lock-in is acceptable.

Pick Ethernet when

The operating model matters

Multi-tenant GPU-as-a-Service and clusters that share infrastructure with the rest of the data center, where one operating model, one tooling stack, and a multi-vendor supply chain win.

Pick Ethernet when

Cost per GPU-flop is the gate

OcNOS-DC runs on open hardware from multiple vendors, so the switch hardware can be sourced competitively.

Pick Ethernet when

The fabric spans data centers

If a training run will ever cross two halls or two regions, coherent DCI, EVPN inter-DC, and standard multi-vendor optics extend the same Ethernet fabric across sites.

What modern Ethernet added

Three developments moved production AI fabrics onto Ethernet: genuinely lossless behaviour, load balancing that keeps flows off congested uplinks, and a spray-friendly next-generation transport.

Lossless behaviour

RoCEv2 with PFC, DCQCN, and PFC deadlock detection and recovery delivers the lossless RDMA transport AI collectives need, production-grade today on standard Ethernet.

RoCEv2 lossless fabric

Dynamic load balancing

Static ECMP collisions on AI workloads are real, but DLB rebins flowlets on local congestion, and GLB in OcNOS 7.1 extends that to end-to-end path scoring.

Dynamic load balancing (DLB)

Spray-friendly transport

Ultra Ethernet (UEC 1.0, June 2025) brings packet spray, multi-path RDMA, and out-of-order delivery to standard Ethernet. Build on RoCEv2 now, and plan for UEC as UEC-capable NICs ship.

Ultra Ethernet (UEC)

The network is a small line item, so spend the comparison on what it frees up

  • OcNOS-DC runs on open hardware from multiple vendors, so the switch hardware can be sourced competitively.
  • The saved capital funds more GPUs, a larger storage tier, or a second site for resilience.
  • Where the fabric is multi-tenant or shared with the rest of the data center, one network model is worth more than the line-item difference.

Both have a place, and most AI fabrics belong on Ethernet

Ethernet does not win every workload, but the operational and economic case is decisive for most, and it strengthens as the technical gap closes.

Both have a place

Ethernet does not win every workload. Tight HPC clusters with absolute-floor latency requirements still favor InfiniBand.

Most AI fabrics belong on Ethernet

Production AI training and inference at hyperscale is moving to Ethernet as the operational and economic case grows and the technical gap keeps closing.

OcNOS-DC is the open path

RoCEv2 and DLB today, GLB in OcNOS 7.1. One NOS on qualified open hardware from Edgecore, UfiSpace, and others.

InfiniBand vs Ethernet, answered

Is Ethernet fast enough for AI training versus InfiniBand?
For most distributed-training collectives at scale, yes. Ethernet has a higher latency floor than InfiniBand by hundreds of nanoseconds, but that sits below the threshold that affects most workloads once RoCEv2 with PFC, DCQCN, and DLB is configured correctly.
When should I still pick InfiniBand?
Choose InfiniBand for tight HPC simulation where the absolute latency floor matters more than total cost of ownership, and for captive single-tenant clusters where single-vendor lock-in is acceptable.
When does Ethernet win for an AI fabric?
Ethernet wins for multi-tenant GPU-as-a-Service, clusters that share one operational model and tooling with the rest of the data center, cost-sensitive multi-thousand-GPU builds, and fabrics that extend across data centers via coherent DCI and EVPN.
How does OcNOS close the gap with InfiniBand?
OcNOS-DC delivers RoCEv2 lossless transport, dynamic load balancing (DLB), and PFC deadlock detection and recovery today, with GLB in 7.1 for end-to-end path scoring, all on qualified open hardware from vendors such as Edgecore and UfiSpace.
Are hyperscalers replacing InfiniBand with Ethernet for AI?
Largely yes, though InfiniBand is not going away. According to Dell'Oro Group, Ethernet passed InfiniBand in AI back-end (scale-out) switching and reached about two-thirds of the market by early 2026, up from InfiniBand's roughly 80 percent share in late 2023. Meta has described training its largest models over a RoCE Ethernet fabric on one of two 24,576-GPU clusters. The Ultra Ethernet Consortium, whose members span silicon, NIC, switch, and cloud companies, standardizes this direction. The reasons are operational and economic: one network model shared with the rest of the data center, a multi-vendor supply chain, and lower cost per port at 400G and 800G.
What is the best alternative to InfiniBand for AI cluster networking?
Ethernet with RoCEv2 is the mainstream alternative. It carries RDMA losslessly using PFC, ECN and DCQCN and adds dynamic load balancing (DLB), and OcNOS-DC delivers this on open, multi-vendor hardware rather than a single-vendor fabric. Ultra Ethernet (UEC) adds packet spray and multi-path RDMA on UEC-capable NICs.
What is the difference between Ultra Ethernet and InfiniBand?
Ultra Ethernet (UEC) adds a transport built for AI and HPC to standard Ethernet: packet spray across all paths, multi-path RDMA, out-of-order delivery, and selective retransmission. The UEC 1.0 specification, published in June 2025, defines this transport. The difference is the ecosystem: InfiniBand is effectively single-vendor for NIC and switch silicon, while Ultra Ethernet is an open, multi-vendor specification any vendor can build to.
How much does an Ethernet AI fabric cost compared to InfiniBand?
OcNOS-DC runs on open hardware from multiple vendors, so the switch hardware can be sourced competitively. The more useful question is what any saved capital funds, whether that is more GPUs, a larger storage tier, or a second site for resilience. Contact IP Infusion for pricing on your platform.

Go deeper. Take it with you.

The product datasheet plus short, technical downloads that go further than this page.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Lossless RoCEv2 fabric on Tomahawk 5 spines: SKU tiers, qualified platforms, and deployment architecture.

Get the brief
Solution Brief

EVPN-VXLAN Data Center Fabric

Leaf-spine data center fabric on EVPN-VXLAN: a VRF and VNIs per tenant, EVPN multihoming, BGP unnumbered and route-target filtering.

Get the brief

Sizing your next cluster? Get a workload-specific review

Tell us the workload and the GPU scale, and an IP Infusion engineer will run the maths with you, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.

Book an architecture review Open the AI fabric design suite

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs AI Fabric Topologies Rail-optimized two-tier and three-tier fabrics, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA? AI networking switches
The technology inside RoCEv2 lossless Rail-optimized network DLB GLB (7.1) Ultra Ethernet Tomahawk 6 at 1.6T DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI