Networking Definition · RDMA · RoCEv2

What Is an AI Fabric?

An AI fabric is the GPU back-end (scale-out) network that carries RDMA traffic between GPUs during distributed training and inference. It connects every GPU NIC across servers and pods using leaf and spine switches in rail-optimized and Clos topologies, and it runs a lossless Ethernet transport (RoCEv2 with PFC and ECN) so no packet is dropped under load. Because GPUs exchange data in lockstep collectives, the fabric’s tail latency sets job completion time. This is the networking sense, not the enterprise "data fabric" used in data management. OcNOS-DC provides this fabric on Broadcom Tomahawk 5 hardware.

Book an architecture review See the topologies

RoCEv2lossless Ethernet transport

51.2 TbpsTomahawk 5, 64x800G

16,000+GPUs, 3-stage Clos

OcNOS-DCone NOS on open hardware

Two meanings, one word

The networking sense, not "data fabric"

Two different fields use the word "fabric". A data fabric is a software architecture for unifying and governing data across an enterprise. An AI fabric, the subject of this page, is a physical network: the wires, switches, and transport that move RDMA traffic between GPUs. When an AI infrastructure team says "the fabric was the bottleneck", they mean this network, the one that decided how long the training run took.

Data fabric. An enterprise data-management architecture for unifying and governing data across systems. Software, not wires.
AI fabric. The physical GPU back-end network that carries RDMA between GPUs during training and inference. This page is about this meaning.
Why it matters. The AI fabric is the one that sets job completion time, so it is engineered to stay lossless and evenly loaded.

How the fabric is built

Rails, Clos, and a transport that cannot drop

An AI fabric is a leaf-spine (Clos) network tuned for one traffic pattern: many GPUs sending large flows to each other at once. Three design choices define it, and each one exists to protect job completion time.

Topology

Rail-optimized, then 3-stage Clos

Each of a GPU server’s 8 NICs connects to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf. Past a single pod, the design extends to a 3-stage Clos of leaf, spine, and super-spine.

AI fabric topologies

Transport

Lossless RoCEv2

GPUs move data with RDMA, which performs poorly when packets are dropped. PFC stops a switch queue from overflowing and ECN marks packets early so the sending NIC slows before loss happens. That combination carries RoCEv2 over Ethernet.

RoCEv2 lossless fabric

Behaviour

Why it has to be lossless

OcNOS-DC ships this fabric with PFC (including over L3), ECN, Dynamic ECN, WRED, and dynamic load balancing to keep every path evenly loaded, because the tail flow decides when the whole job finishes.

Lossless Ethernet

Where this fits

Ethernet is enough, and the switch sets the ceiling

The AI fabric is an Ethernet network today. What it can reach is set by the switch silicon underneath it and the GPU count you design for.

Ethernet is enough. RoCEv2 with PFC and ECN delivers lossless RDMA on standard, multi-vendor hardware. Meta has published a 24,000-GPU training cluster built on Ethernet.
The switch sets the ceiling. OcNOS-DC runs on Broadcom Tomahawk 5 (51.2 Tbps, 64x800G) platforms such as the Edgecore AIS800-64D and UfiSpace S9321-64E, the density an 800G GPU fabric needs.
Design by GPU count. Rail-optimized single pod up to roughly 1,000 GPUs, then a 3-stage Clos to 16,000+ and up to about 65,535 at the fat-tree limit.