Networking Definition · Scale-out · RoCEv2

What Is a GPU Fabric?

A GPU fabric is the back-end network that interconnects GPUs across servers and pods so they can work together on one training or inference job. It carries the RDMA traffic GPUs exchange during collective operations, separate from the front-end network used for management and storage. It is the scale-out network that begins where the in-server scale-up domain, such as NVLink, ends, built from rail-optimized and Clos topologies running lossless RoCEv2, and OcNOS-DC provides it on Broadcom Tomahawk 5 hardware.

Book an architecture review See the topologies
~2,048 GPUs2-tier, 1:1 non-blocking
16,000+ GPUs3-stage Clos scale
51.2 TbpsTomahawk 5 · 64x800G
1 NOSOcNOS-DC on open hardware
Where the fabric begins

Scale-up ends, scale-out begins

A GPU fabric sits on one side of a clear boundary. Inside a server, the scale-up domain runs at terabit speeds; between servers, the scale-out GPU fabric carries the remainder. A collective uses scale-up first inside the domain, then this fabric carries the cross-server traffic.

Inside the server

Scale-up domain

The very high bandwidth interconnect inside a GPU server and its tightly coupled domain, for example NVLink within an NVL72 rack. GPUs here talk as if they were one large accelerator. This is not the GPU fabric.

Between servers

Scale-out GPU fabric

The Ethernet network between servers and pods across rail, leaf, spine, and Clos tiers. This is the GPU fabric. Because same-rail traffic is often absorbed by scale-up first, this plane carries the cross-rail and cross-pod remainder, which is why 1:1 non-blocking is worth paying for here specifically.

How the fabric is shaped

Rails and Clos

Most GPU fabrics are rail-optimized leaf-spine networks: each of a GPU server's 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf and never crosses the spine. Past a single pod, the design extends to a 3-stage Clos with a super-spine tier.

Rail-optimized leaves

Each of a server's 8 NICs maps to its own rail leaf, keeping the dominant same-rail AllReduce traffic on one leaf and off the spine.

Rail-optimized network

3-stage Clos scale

On radix-64 Tomahawk 5, a 2-tier fabric reaches about 2,048 GPUs at 1:1 non-blocking; a 3-stage Clos scales to 16,000+ GPUs.

AI fabric topologies

Lossless RoCEv2 transport

The GPU plane carries RDMA over RoCEv2, engineered lossless so a collective never stalls on a dropped packet.

RoCEv2 lossless fabric
On the GPU plane

Lossless RoCEv2, and OcNOS-DC delivers it

GPUs move data with RDMA over RoCEv2, which performs poorly under packet loss. So the GPU fabric is engineered lossless and kept evenly loaded, so no link becomes a hot spot during a collective. OcNOS-DC supplies exactly this on Broadcom Tomahawk 5, as an enabler for open, multi-vendor hardware.

  • Engineered lossless with Priority Flow Control (including over L3) and ECN, so a training step never stalls on a dropped packet.
  • Kept evenly loaded with Dynamic ECN and DLB, so no single link becomes a hot spot during a collective.
  • Delivered by OcNOS-DC on Broadcom Tomahawk 5 (51.2 Tbps, 64x800G) platforms such as the Edgecore AIS800-64D and UfiSpace S9321-64E.
FAQ

What Is a GPU Fabric FAQ

What is a GPU fabric?
A GPU fabric is the back-end network that interconnects GPUs across servers and pods so they can work together on a single training or inference job. It carries the RDMA traffic GPUs exchange during collective operations, and it is separate from the front-end network used for management and storage. In practice a GPU fabric is a leaf-spine (Clos) Ethernet network, usually rail-optimized, running lossless RoCEv2. OcNOS-DC provides it on Broadcom Tomahawk 5 hardware.
What is the difference between scale-up and scale-out?
Scale-up is the very high bandwidth interconnect inside a GPU server and its tightly coupled domain, for example NVLink within an NVL72 rack, where GPUs talk at terabit speeds as if they were one large accelerator. Scale-out is the network between servers and pods, the GPU fabric this page describes, built from rail, leaf, spine, and Clos tiers over Ethernet. A collective first uses scale-up inside the domain, then the scale-out GPU fabric carries the remainder between servers.
What topology does a GPU fabric use?
Most GPU fabrics use a rail-optimized leaf-spine design: each of a GPU server’s 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf. As the cluster grows past a single pod, the design extends to a 3-stage Clos with a super-spine tier. On radix-64 Tomahawk 5, a 2-tier fabric reaches about 2,048 GPUs at 1:1 non-blocking and a 3-stage Clos scales to 16,000+ GPUs. See the rail-optimized network and AI fabric topologies pages.
Does a GPU fabric have to be lossless?
Yes, on the GPU plane. GPUs move data with RDMA over RoCEv2, which performs poorly under packet loss, so the GPU fabric is engineered lossless with Priority Flow Control and ECN, and kept evenly loaded with dynamic load balancing. That no-drop behavior is what keeps a collective from stalling. OcNOS-DC supplies PFC, ECN, Dynamic ECN, and DLB on Broadcom Tomahawk 5 for exactly this.
Datasheet and Solution Briefs

Go deeper. Take it with you.

Two short, technical downloads that go further than this page: the full OcNOS-DC datasheet and the lossless 800G AI fabric architecture.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Non-blocking RoCEv2 fabric on Broadcom Tomahawk 4/5 spines: SKU tiers, validated platforms, and deployment architecture.

Get the brief

Standing up a GPU fabric? Let's size the plane together.

Tell us the GPU count and the workload, and an IP Infusion engineer will run the port maths with you, or start from a first-pass rail-optimized layout in the AI Fabric Design Suite.

Book an architecture review Open the AI fabric design suite
AI Fabric

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs AI Fabric Topologies Rail-optimized, scheduled 3-stage Clos, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA?
The technology inside RoCEv2 lossless Rail-optimized network DLB adaptive routing GLB (7.1) Ultra Ethernet DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI