Rail-optimized · 3-stage Clos · Coherent DCI

AI Fabric Topologies: Rail-Optimized & Scheduled Designs

Match the topology to the GPU count: a rail-optimized leaf-spine pod up to about 1,000 GPUs, a 3-stage Clos to 16,000+, and coherent DCI to span sites. Pick the smallest non-blocking design that keeps every GPU's link saturated during collectives. All three run on OcNOS-DC over a lossless RoCEv2 (PFC + ECN) L3 fabric, sized here in concrete port-counts on Broadcom Tomahawk 4 and Tomahawk 5.

Book an architecture review Open the AI fabric design suite
256 to 16k+GPUs, one design system
1:1non-blocking GPU plane
up to 800GTH5 leaf and spine
1 imageOcNOS-DC, every tier
Choose by GPU count, not by buzzword

The smallest non-blocking design that fits your GPU count

An AI fabric topology has one job: keep every GPU's outbound link saturated during a collective without creating tail-latency outliers. The right topology is the smallest one that does this for your GPU count, with a fall-back path for the next size up. Four reference points, with concrete port maths on Broadcom Tomahawk silicon.

One image, every tier of scale
256 GPUs
Entry non-blocking pod
One rack-row of rail-aligned leaves over a small spine tier. Two-tier folded Clos, 1:1 non-blocking.
8 leaves · 4 spines · TH4 · 400G
1,024 GPUs
Rail-optimized 2-tier pod
Rail-aligned leaves with a 1:1 non-blocking spine. Intra-rail AllReduce stays on the leaf; cross-rail traffic uses the spine. The standard single-pod scalable unit.
32 leaves · 16 spines · TH5 · 800G
4,096 GPUs
3-stage Clos
Leaf, spine, super-spine. Each 1,024-GPU pod is 1:1 non-blocking; a super-spine plane scales across pods. DLB at every tier; GLB end-to-end on the OcNOS 7.1 train.
128 leaves · 64 spines · 32 super-spines · TH5 · 800G
16,384 GPUs
Scaled 3-stage Clos
Multi-pod 3-stage Clos with a super-spine plane. Sized for the trillion-parameter training class.
512 leaves · 256 spines · 128 super-spines · TH5 · 800G

The two-tier leaf-spine ceiling is about 2,048 GPUs at one 800G fabric port per GPU; beyond it, add the super-spine tier. The same OcNOS-DC image runs every tier, so the fabric scales out with the cluster instead of being redesigned at each step.

Sizing your own cluster? The AI Fabric Design Suite gives a fast first cut: it sizes a non-blocking two-tier leaf-spine pod assuming one fabric NIC per GPU, and flags when you cross into three-tier scale. The reference designs below use the same non-blocking leaf/spine math and extend it to a 3-stage Clos at scale, so the switch counts line up with the tool. Rail-optimized here is the wiring discipline of an 8-NIC GPU server (one rail per leaf, so intra-rail AllReduce stays on the leaf) layered on that non-blocking fabric: it changes traffic locality, not the switch count. Use the tool for a ballpark; use these designs for the build.

Three reference designs

Rail-optimized, scheduled Clos, and coherent DCI

OcNOS-DC ships against three reference topologies that cover single-pod, multi-pod, and multi-site AI fabrics. Each is a concrete build, sized on HCL-listed Broadcom hardware, not a whiteboard sketch.

Reference design 1

Rail-optimized single pod

Each GPU server's 8 NICs land on 8 dedicated rail leaves, so same-rail AllReduce stays on one leaf and never traverses the spine. A non-blocking spine carries cross-rail flows. The single-pod scalable unit, up to about 1,000 GPUs.

Rail-optimized network
Reference design 2

Scheduled 3-stage Clos

Leaf, spine, and super-spine tiers scale from 4,096 to 16,384 GPUs. Non-blocking within each pod, with the super-spine plane setting the cross-pod ratio. DLB at every hop, GLB end-to-end on the OcNOS 7.1 train.

GLB fabric-wide balancing
Reference design 3

Coherent multi-DC DCI

When a training run spans data halls, extend the fabric with 400G ZR / ZR+ coherent optics on the spine. Transponder-free DCI between sites, with the in-site 3-stage Clos unchanged.

Coherent DCI
Reference design 1

Rail-optimized single pod

Each GPU server has 8 NICs, one per rail (a dedicated xCCL (NCCL / RCCL / oneCCL) collective channel). Each rail is its own dedicated leaf, so all 8 NICs on every server land on a different leaf. AllReduce across rail-N stays inside leaf-N, so there is no east-west pressure on the spine for the dominant collective pattern.

Rail-optimized AI fabric topology: four 800G Tomahawk 5 spines above eight rail leaves, with four GPU servers each mapping one of its eight NICs to a different rail leaf, so intra-rail AllReduce stays on one leaf.
Rail-optimized single pod: each GPU server's 8 NICs map one per rail to 8 dedicated leaves, so same-rail AllReduce stays on the leaf and only cross-rail traffic reaches the spine.

OcNOS pieces: BGP-unnumbered L3 underlay, RoCEv2 lossless (PFC + ECN) on every leaf, DLB at the spine tier. Built on HCL-listed hardware: the 800G scalable unit uses TH5 64x800G leaves and spines (Edgecore AIS800-64D or UfiSpace S9321-64E); the entry 256-GPU pod uses Edgecore AS9736-64D (TH4, 64x400G).

Reference design 2

3-stage Clos scheduled fabric: 4,096 to 16,384 GPUs

Rail-optimized stops scaling somewhere between 1k and 2k GPUs: you run out of leaf radix, or the spine tier becomes too oversubscribed. Above that, most modern AI fabrics move to a 3-stage Clos: leaf, spine, super-spine. Any two GPUs are at most four switch hops apart; same-leaf and same-pod peers are closer.

The hard part on a Clos is spreading flows evenly so no link becomes a hot spot. Approaches run from per-flow ECMP, through adaptive (dynamic) load balancing, to per-packet spray, the model Ultra Ethernet uses with reordering handled at the NIC. A separate family, cell-based scheduled fabrics such as Broadcom DDC, segments traffic into cells and schedules it inside the fabric. OcNOS keeps the GPU plane balanced with DLB today and adds fabric-wide GLB on the 7.1 train, and is UEC-ready as UEC NICs arrive.

The diagram is schematic: it draws a reduced tier count. The 4,096-GPU build is 128 leaves / 64 spines / 32 super-spines on TH5 800G.

Three-stage Clos AI fabric topology: a super-spine tier over a spine tier over leaf switches feeding GPU pods, sized to 4,096 GPUs on 800G Tomahawk 5 with DLB at every hop.
Scheduled 3-stage Clos: leaf, spine, and super-spine tiers scale a non-blocking GPU plane to thousands of GPUs, with DLB at every hop and fabric-wide GLB on the OcNOS 7.1 train.

OcNOS pieces: eBGP-unnumbered L3 underlay, RoCEv2 lossless (PFC + ECN), DLB at every tier, GLB end-to-end on the OcNOS 7.1 train, and gNMI streaming telemetry to your observability stack. Built on HCL-listed TH5 64x800G chassis throughout.

Subscription is a dial, not a fixed rule. These counts make each 1,024-GPU pod 1:1 non-blocking and use a cost-optimized ~2:1 super-spine for cross-pod traffic, the rail-optimized approach hyperscale Ethernet fabrics rely on (published large-scale designs oversubscribe the top tier far more, because collective traffic stays pod-local). Want maximal any-to-any headroom instead? A fully non-blocking 1:1 build is 128 / 128 / 64 at 4,096 GPUs and 512 / 512 / 256 at 16,384; only the spine and super-spine counts change. Model either in the AI Fabric Design Suite.

Reference design 3

Multi-DC AI fabric: coherent DCI

When a single training run spans more than one data hall, increasingly common for trillion-parameter models, the fabric extends across the WAN. OcNOS-DC supports 400G ZR / ZR+ coherent optics directly on the spine for transponder-free DCI across sites. The underlying 3-stage Clos in each site is unchanged.

Multi-data-center AI fabric topology: two leaf-spine sites joined by 400G ZR and ZR+ coherent optics on the spine, extending the fabric across the WAN, no external transponders.
Coherent multi-DC DCI: two AI data centers joined by 400G ZR and ZR+ optics on the spine, extending the fabric across sites without external transponders.

OcNOS pieces: 400G ZR/ZR+ pluggable coherent optics on a DWDM-capable spine or border-leaf port, with gNMI telemetry across sites. No external transponders required. Reach: 400ZR to roughly 120 km amplified; OpenZR+ reaches farther on oFEC.

Design guidance

Design rules of thumb

A handful of rules keep an AI fabric on the right side of the performance and cost curve, whichever reference design you build from.

  • Match the topology to the GPU count. Smallest pods (under one leaf's NIC radix): rail-only is enough. Single-pod scale: rail-optimized leaf-spine. Multi-pod: 3-stage Clos is the only design that scales without oversubscription compromises.
  • Always 1:1 subscription on the AI plane. Storage and CPU racks can run higher oversubscription ratios. The GPU plane should not.
  • Plan the rail count from xCCL, not from cabling convenience. 8 rails is the current de-facto standard for 8-NIC GPU servers. Do not combine rails into fewer leaves.
  • Pick the silicon by power and density, not the badge. TH4 (25.6T) and TH5 (51.2T) are the workhorses; the choice between them is rack power and breakout-cable cost.
  • Plan for GLB / UEC at design time. Build the telemetry plane in from day one, even on a 7.0 fabric, so the OcNOS 7.1 GLB upgrade is purely a software step. See GLB and Ultra Ethernet.
  • Validate against the HCL. Every reference here is built on hardware listed in the OcNOS Hardware Compatibility List; pick from there for first-class support.
RoCEv2 lossless fabric DLB: adaptive routing GLB: fabric-wide (7.1) Ultra Ethernet (UEC) EVPN-VXLAN BGP underlay OcNOS feature matrix Validated hardware (HCL) OcNOS-DC AI Fabric solution
FAQ

AI fabric topology FAQ

What is a rail-optimized topology, and how is it different from rail-only?
Rail-optimized wiring connects each of a GPU server’s 8 NICs to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf and never traverses the spine. Rail-only is the small-cluster case: a single rack-row of rail-aligned leaves with no spine tier, where cross-rail traffic relies on the GPU scale-up domain. Rail-optimized adds a non-blocking spine so cross-rail flows have a network path.
How many GPUs can a 3-stage Clos scale to?
It depends on how each 800G port is mapped to GPUs. On radix-64 Tomahawk 5 switches, a 2-tier leaf-spine reaches about 2,048 GPUs at one 800G fabric port per GPU (1:1 non-blocking), and scales toward 8,000+ GPUs when 800G ports break out to multiple GPU NICs. A 3-stage Clos with a super-spine tier extends that to 16,000+ GPUs in the reference designs above, and up to about 65,000 GPUs at the theoretical fat-tree limit. Because most collective traffic stays rail-local, the super-spine plane is sized to the cross-pod ratio you actually need.
Should I use Tomahawk 4 or Tomahawk 5?
Both run OcNOS-DC. Tomahawk 4 (25.6 Tbps, 64×400G) is the cost-optimized choice for entry pods and 400G GPU NICs. Tomahawk 5 (51.2 Tbps, 64×800G) is the workhorse for 800G GPU servers and larger fabrics. Tomahawk 4 has no native 800G, so match the switch to your NIC speed.
Do I need InfiniBand, or is Ethernet enough?
Ethernet is now a first-class AI-fabric transport. RoCEv2 with PFC and ECN delivers lossless RDMA today. Ultra Ethernet (UEC) removes the network-wide PFC dependency using endpoint packet-spray, selective retransmission, and link-level retry as UEC NICs ship. OcNOS-DC runs the RoCEv2 fabric today and is UEC-ready.
Where does scale-up end and scale-out begin?
Inside a GPU server and its NVLink domain (for example GB200 NVL72), GPUs communicate over the scale-up fabric at terabit speeds. The rail, leaf-spine, and Clos network is the scale-out fabric between servers and pods. Most same-rail collective traffic is absorbed by scale-up first, so the network carries the cross-rail and cross-pod remainder, which is why 1:1 non-blocking matters most on the GPU plane.
Datasheet and Solution Briefs

Go deeper. Take it with you.

The product datasheet plus short, technical downloads that go further than this page.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Non-blocking RoCEv2 fabric on Broadcom Tomahawk 4/5 spines: SKU tiers, validated platforms, and deployment architecture.

Get the brief
Solution Brief

EVPN-VXLAN Data Center Fabric

Carrier-grade leaf-spine data center fabric: symmetric IRB, Type-2/Type-5 routes, and distributed anycast gateway.

Get the brief

Designing your AI fabric? We'll do the port-count maths with you.

Tell us the workload and the GPU scale, and an IP Infusion engineer will size the leaf, spine, and super-spine tiers with you, or start with a first-pass layout in the AI Fabric Design Suite.

Book an architecture review Open the AI fabric design suite
AI Fabric

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract.
You are here AI Fabric Topologies Rail-optimized, scheduled 3-stage Clos, and coherent DCI, sized in real port counts.
Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA?
The technology inside RoCEv2 lossless Rail-optimized network DLB adaptive routing GLB (7.1) Ultra Ethernet DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI