Startseite · Dokumentation · Technologie · What Is a GPU Fabric
Networking Definition

What Is a GPU Fabric?

A GPU fabric is the back-end network that interconnects GPUs across servers and pods for training and inference. It is the scale-out network beyond the in-server scale-up domain, and it carries the RDMA traffic GPUs exchange.

A GPU fabric is the back-end network that interconnects GPUs across servers and pods so they can work together on one training or inference job. It carries the RDMA traffic GPUs exchange during collective operations, separate from the front-end network used for management and storage. It is the scale-out network that begins where the in-server scale-up domain (such as NVLink) ends, built from rail-optimized and Clos topologies running lossless RoCEv2. OcNOS-DC provides this fabric on Broadcom Tomahawk 5 hardware.

Scale-up ends, scale-out begins

A GPU fabric sits on one side of a clear boundary.

Inside the server

Scale-up domain

The very high bandwidth interconnect inside a GPU server and its tightly coupled domain, for example NVLink within an NVL72 rack. GPUs here talk at terabit speeds as if they were one large accelerator. This is not the GPU fabric.

Between servers

Scale-out GPU fabric

The Ethernet network between servers and pods: rail, leaf, spine, and Clos tiers. This is the GPU fabric. A collective uses scale-up first inside the domain, then this fabric carries the cross-server remainder.

Getting this boundary right matters for sizing. Because same-rail traffic is often absorbed by scale-up first, the GPU fabric carries the cross-rail and cross-pod remainder, which is why 1:1 non-blocking is worth paying for on the GPU plane specifically and not on storage or CPU racks.

How the fabric is shaped: rails and Clos

Most GPU fabrics are rail-optimized leaf-spine networks: each of a GPU server's 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf and never crosses the spine. Past a single pod, the design extends to a 3-stufiges Clos (leaf, spine, super-spine). On radix-64 Tomahawk 5 switches, a 2-tier fabric reaches roughly 2,048 GPUs at 1:1 non-blocking and a 3-stage Clos scales to 16,000+ GPUs, up to about 65,535 at the fat-tree limit. See rail-optimized network und AI-Fabric-Topologien for the port maths.

Lossless RoCEv2 on the GPU plane

GPUs move data with RDMA over RoCEv2, which performs poorly under packet loss. So the GPU fabric is engineered lossless with Priority Flow Control and ECN, and kept evenly loaded with dynamic load balancing so no link becomes a hot spot during a collective. OcNOS-DC supplies PFC (including over L3), ECN, Dynamic ECN, and DLB on Broadcom Tomahawk 5 (51.2 Tbps, 64x800G) platforms such as the Edgecore AIS800-64D and UfiSpace S9321-64E. That no-drop, evenly-loaded behavior is what keeps a training step from stalling.

Rail-optimized network AI-Fabric-Topologien Verlustfreie RoCEv2-Fabric What is an AI fabric AI-Fabric-Lösung OcNOS-DC

Standing up a GPU fabric? Let's size the plane together.

Architecture Review buchen →
FAQ

What Is a GPU Fabric FAQ

Was ist ein GPU-Fabric?
A GPU fabric is the back-end network that interconnects GPUs across servers and pods so they can work together on a single training or inference job. It carries the RDMA traffic GPUs exchange during collective operations, and it is separate from the front-end network used for management and storage. In practice a GPU fabric is a leaf-spine (Clos) Ethernet network, usually rail-optimized, running lossless RoCEv2. OcNOS-DC provides it on Broadcom Tomahawk 5 hardware.
What is the difference between scale-up and scale-out?
Scale-up is the very high bandwidth interconnect inside a GPU server and its tightly coupled domain, for example NVLink within an NVL72 rack, where GPUs talk at terabit speeds as if they were one large accelerator. Scale-out is the network between servers and pods, the GPU fabric this page describes, built from rail, leaf, spine, and Clos tiers over Ethernet. A collective first uses scale-up inside the domain, then the scale-out GPU fabric carries the remainder between servers.
What topology does a GPU fabric use?
Most GPU fabrics use a rail-optimized leaf-spine design: each of a GPU server's 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf. As the cluster grows past a single pod, the design extends to a 3-stage Clos with a super-spine tier. On radix-64 Tomahawk 5, a 2-tier fabric reaches about 2,048 GPUs at 1:1 non-blocking and a 3-stage Clos scales to 16,000+ GPUs. See the rail-optimized network and AI fabric topologies pages.
Does a GPU fabric have to be lossless?
Yes, on the GPU plane. GPUs move data with RDMA over RoCEv2, which performs poorly under packet loss, so the GPU fabric is engineered lossless with Priority Flow Control and ECN, and kept evenly loaded with dynamic load balancing. That no-drop behavior is what keeps a collective from stalling. OcNOS-DC supplies PFC, ECN, Dynamic ECN, and DLB on Broadcom Tomahawk 5 for exactly this.
AI-Fabric

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Lösung Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Referenzdesigns AI-Fabric-Topologien Rail-optimized, scheduled 3-stage Clos, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
Neu bei AI-Fabrics? Hier starten Was ist ein AI-Fabric? Was ist ein GPU-Fabric? Was ist verlustfreies Ethernet? Was ist RDMA?
The technology inside RoCEv2 lossless Rail-optimized network DLB adaptive routing GLB (7.1) Ultra Ethernet DCQCN PFC-Deadlock InfiniBand vs. Ethernet RoCE vs. InfiniBand AI fabric architecture Coherent DCI
Datenblatt und Solution Briefs

Gehen Sie tiefer. Nehmen Sie es mit.

Two short, technical downloads that go further than this page: the full OcNOS-DC datasheet and the lossless 800G AI fabric architecture.

Datenblatt

OcNOS-DC Datenblatt

Vollständige OcNOS-DC Spezifikation: Funktionsumfang für EVPN-VXLAN und Ethernet for AI, Software-SKUs, unterstützte Hardware-Plattformen und der Solution Ordering Guide.

Datasheet abrufen
Solution Brief

OcNOS 800G verlustfreie AI Fabric

Non-blocking RoCEv2-Fabric auf Broadcom Tomahawk 4/5 Spines: SKU-Stufen, validierte Plattformen und Deployment-Architektur.

Brief anfordern