Inicio · Documentación · Tecnología · What Is a GPU Fabric
Networking Definition

What Is a GPU Fabric?

A GPU fabric is the back-end network that interconnects GPUs across servers and pods for training and inference. It is the scale-out network beyond the in-server scale-up domain, and it carries the RDMA traffic GPUs exchange.

A GPU fabric is the back-end network that interconnects GPUs across servers and pods so they can work together on one training or inference job. It carries the RDMA traffic GPUs exchange during collective operations, separate from the front-end network used for management and storage. It is the scale-out network that begins where the in-server scale-up domain (such as NVLink) ends, built from rail-optimized and Clos topologies running lossless RoCEv2. OcNOS-DC provides this fabric on Broadcom Tomahawk 5 hardware.

Scale-up ends, scale-out begins

A GPU fabric sits on one side of a clear boundary.

Inside the server

Scale-up domain

The very high bandwidth interconnect inside a GPU server and its tightly coupled domain, for example NVLink within an NVL72 rack. GPUs here talk at terabit speeds as if they were one large accelerator. This is not the GPU fabric.

Between servers

Scale-out GPU fabric

The Ethernet network between servers and pods: rail, leaf, spine, and Clos tiers. This is the GPU fabric. A collective uses scale-up first inside the domain, then this fabric carries the cross-server remainder.

Getting this boundary right matters for sizing. Because same-rail traffic is often absorbed by scale-up first, the GPU fabric carries the cross-rail and cross-pod remainder, which is why 1:1 non-blocking is worth paying for on the GPU plane specifically and not on storage or CPU racks.

How the fabric is shaped: rails and Clos

Most GPU fabrics are rail-optimized leaf-spine networks: each of a GPU server's 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf and never crosses the spine. Past a single pod, the design extends to a Clos de 3 etapas (leaf, spine, super-spine). On radix-64 Tomahawk 5 switches, a 2-tier fabric reaches roughly 2,048 GPUs at 1:1 non-blocking and a 3-stage Clos scales to 16,000+ GPUs, up to about 65,535 at the fat-tree limit. See rail-optimized network and Topologías de AI fabric for the port maths.

Lossless RoCEv2 on the GPU plane

GPUs move data with RDMA over RoCEv2, which performs poorly under packet loss. So the GPU fabric is engineered lossless with Priority Flow Control and ECN, and kept evenly loaded with dynamic load balancing so no link becomes a hot spot during a collective. OcNOS-DC supplies PFC (including over L3), ECN, Dynamic ECN, and DLB on Broadcom Tomahawk 5 (51.2 Tbps, 64x800G) platforms such as the Edgecore AIS800-64D and UfiSpace S9321-64E. That no-drop, evenly-loaded behavior is what keeps a training step from stalling.

Rail-optimized network Topologías de AI fabric Tejido sin pérdidas RoCEv2 What is an AI fabric Solución AI Fabric OcNOS-DC

Standing up a GPU fabric? Let's size the plane together.

Reserve una revisión de arquitectura →
Preguntas frecuentes

What Is a GPU Fabric FAQ

¿Qué es una GPU fabric?
A GPU fabric is the back-end network that interconnects GPUs across servers and pods so they can work together on a single training or inference job. It carries the RDMA traffic GPUs exchange during collective operations, and it is separate from the front-end network used for management and storage. In practice a GPU fabric is a leaf-spine (Clos) Ethernet network, usually rail-optimized, running lossless RoCEv2. OcNOS-DC provides it on Broadcom Tomahawk 5 hardware.
What is the difference between scale-up and scale-out?
Scale-up is the very high bandwidth interconnect inside a GPU server and its tightly coupled domain, for example NVLink within an NVL72 rack, where GPUs talk at terabit speeds as if they were one large accelerator. Scale-out is the network between servers and pods, the GPU fabric this page describes, built from rail, leaf, spine, and Clos tiers over Ethernet. A collective first uses scale-up inside the domain, then the scale-out GPU fabric carries the remainder between servers.
What topology does a GPU fabric use?
Most GPU fabrics use a rail-optimized leaf-spine design: each of a GPU server's 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf. As the cluster grows past a single pod, the design extends to a 3-stage Clos with a super-spine tier. On radix-64 Tomahawk 5, a 2-tier fabric reaches about 2,048 GPUs at 1:1 non-blocking and a 3-stage Clos scales to 16,000+ GPUs. See the rail-optimized network and AI fabric topologies pages.
Does a GPU fabric have to be lossless?
Yes, on the GPU plane. GPUs move data with RDMA over RoCEv2, which performs poorly under packet loss, so the GPU fabric is engineered lossless with Priority Flow Control and ECN, and kept evenly loaded with dynamic load balancing. That no-drop behavior is what keeps a collective from stalling. OcNOS-DC supplies PFC, ECN, Dynamic ECN, and DLB on Broadcom Tomahawk 5 for exactly this.
Fabric de IA

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solución Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs Topologías de AI Fabric Rail-optimized, scheduled 3-stage Clos, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
¿Nuevo en las AI fabrics? Empieza aquí ¿Qué es una AI fabric? ¿Qué es una GPU fabric? ¿Qué es Ethernet sin pérdidas? ¿Qué es RDMA?
The technology inside RoCEv2 lossless Rail-optimized network DLB adaptive routing GLB (7.1) Ultra Ethernet DCQCN bloqueo por PFC InfiniBand frente a Ethernet RoCE frente a InfiniBand AI fabric architecture Coherent DCI
Datasheet y resúmenes de solución

Profundice. Llévelo consigo.

Two short, technical downloads that go further than this page: the full OcNOS-DC datasheet and the lossless 800G AI fabric architecture.

Datasheet

Datasheet de OcNOS-DC

Especificación completa de OcNOS-DC: el conjunto de funciones EVPN-VXLAN y Ethernet for AI, los SKU de software, las plataformas de hardware compatibles y la guía de pedido de la solución.

Obtener el datasheet
Solution brief

AI Fabric sin pérdidas de 800G con OcNOS

Fabric RoCEv2 sin bloqueo sobre spines Broadcom Tomahawk 4/5: niveles de SKU, plataformas validadas y arquitectura de despliegue.

Obtener el brief