AI fabric design tool
Design a non-blocking RoCEv2 GPU fabric end to end, vendor-neutral. Size the leaf and spine switches, plan the transceiver and cable counts, compare InfiniBand against Ethernet for your cluster, and get a lossless congestion profile. This designs the network only. It is not a cost estimate or a bill of materials.
Size the GPU fabric
Pick a starting size, or enter your own GPU count, port speed, and switch silicon. The tool sizes a folded two-tier leaf-spine pod and flags when a cluster crosses into three-tier scale.
…
Counts assume a folded leaf-spine (Clos) fabric with one fabric NIC per GPU, where each leaf splits its ports between GPUs and spine uplinks. Rail-optimized wiring (multiple NICs per GPU) changes traffic locality, not these switch counts. See the AI fabric topology reference designs for the rail-optimized and three-tier layouts.
Component summary
Fabric link, transceiver, and cable quantities for the sized design. These are counts for planning only, not a bill of materials, quote, or pricing.
| Component | Basis | Quantity |
|---|---|---|
| GPU-facing links | GPUs × 1 NIC | … |
| Leaf to spine fabric links | leaves × uplinks | … |
| Fabric transceivers | links × 2 ends | … |
| Fabric cables | 1 per link | … |
Media by reach, chosen per your cabling: up to 3 m DAC or AEC, up to 50 to 100 m AOC or SR, up to 500 m DR (the workhorse), up to 2 km FR. Reach classes only. No pricing, SKUs, or vendor part numbers. On a three-tier design, add the spine to super-spine links on top of the counts above.
InfiniBand vs Ethernet
A vendor-neutral scorecard across six factors. It shows where each factor leans for your inputs and why. There is no single winner here by design. Use it as an input to a design review, not a final recommendation.
| Factor | InfiniBand | Ethernet / RoCEv2 | Ultra Ethernet (trajectory) |
|---|---|---|---|
| Peak performanceleans InfiniBand | A mid-teens percent raw throughput edge at low tuning effort, plus SHARP in-network reduction that offloads collective operations from the GPUs. | Matches InfiniBand with proper tuning. Meta has described training its largest models on a 24,576-GPU RoCEv2 Ethernet cluster in production. The gap is tuning effort, not the ceiling. | Adds packet spray and NIC-side reordering to cut tail latency and reduce PFC dependence, narrowing the out-of-box gap. |
| Costleans Ethernet | Carries a meaningful premium at fabric scale, commonly cited in the 30 to 60 percent range versus comparable Ethernet. | Merchant-silicon switches and a broad optics supply chain generally make it the lower-cost fabric per port. | Same open merchant-silicon economics as Ethernet. |
| Opennessleans Ethernet | Single-vendor stack end to end (switches, NICs, manager), so multi-vendor sourcing is limited. | Open and multi-vendor: standards-based switches, NICs, and NOS from many suppliers, preserving supply-chain optionality. | Industry consortium standard (UEC 1.0, 2025) reinforcing the open direction. |
| Operational simplicityneutral | UFM gives a single, purpose-built fabric manager, which is turnkey for teams without deep RDMA staff. | Rides the large Ethernet talent pool and standard tooling, but lossless RoCEv2 needs deliberate end-to-end tuning. | Aims to reduce the tuning burden, moving Ethernet toward IB-like out-of-box behavior. |
| Multi-tenancy and convergenceleans Ethernet | Typically a dedicated back-end island, without native multi-tenant isolation or storage convergence. | One converged Ethernet fabric can carry back-end, storage, and front-end traffic with standard tenant isolation. | Extends the converged-Ethernet direction on one open fabric. |
| Trajectoryleans Ethernet | Mature and stable, performance-leading at low tuning, but on a single-vendor roadmap. | Ethernet crossed over to lead AI back-end deployments by mid-2025, with broad ecosystem momentum. | UEC 1.0 (2025) is a multi-vendor spec explicitly targeting AI and HPC back-end fabrics. |
Assumptions: performance is compared with tuning (untuned Ethernet trails, well-tuned Ethernet reaches parity); cost is a range, never a dollar figure; openness means multi-vendor sourcing, not a quality judgment; Ultra Ethernet reflects a 2025-onward direction, not shipping product. Collective tuning refers to xCCL libraries generically. This informs the decision; validate a fabric choice with an IP Infusion engineer.
RoCEv2 lossless profile
RoCEv2 is not a single switch feature. It is a profile composed from PFC, ECN, and ETS, with DCQCN doing the steady-state work and PFC as a backstop. This recommends the mechanisms and the order to apply them for your design. It does not emit threshold values and it does not touch any device.
| Mechanism | Role in the lossless profile | OcNOS availability |
|---|---|---|
| PFC | No-drop guarantee for the RoCE class. Last-resort backstop, not the primary control. | All 4 AI platforms |
| ECN | Marks congestion early so senders throttle before queues overflow. Tune its marking point below the PFC trigger. | All 4 AI platforms |
| DCQCN (ECN + PFC) | Primary steady-state control. ECN marking drives the DCQCN reaction at the NICs; PFC backs it up. Composed, not a standalone feature. | All 4 AI platforms |
| ETS | Bandwidth guarantees and class isolation so the lossless class is not starved by best-effort traffic. | All 4 AI platforms |
| PFC deadlock detection and recovery | Detects a PFC pause deadlock and recovers the stalled queue, the safety net for the PFC backstop. | All 4, OcNOS 7.0 |
| DLB (Dynamic Load Balancing) | Spreads elephant RoCE flows that collide under static ECMP, targeting utilization above 90 percent on the same switches. | All 4 AI platforms |
| Dynamic ECN | Adapts ECN marking to live queue conditions, reducing manual re-tuning as load shifts. | TH5 only, OcNOS 7.0 |
| DLB reactive / random | Advanced DLB placement modes for tighter flow spreading. On TH4, use standard DLB. | TH5 only, OcNOS 7.0 |
| GLB (Global Load Balancing) | Fabric-wide load balancing beyond local DLB decisions. | OcNOS 7.1 |
| Ultra Ethernet | Packet spray with NIC-side reordering and reduced PFC dependence, the trajectory for lossless AI Ethernet. | Runs on UEC-capable NICs |
Advisory only. These recommendations are not pushed to any device and no threshold values are generated here. Availability is per the OcNOS-DC Feature Matrix and Hardware Compatibility List. Validate against your selected platforms and OcNOS release before deployment.
Fabric power
A typical power range for the fabric switches in the sized design. Ranges only, optics-loaded. Network switches are a small part of total cluster power; the GPUs dominate.
Cooling: rear-door or direct-liquid cooling is typically considered above roughly 30 to 40 kW per rack, driven by GPU-server density, not the network. This is guidance, not a computed thermal load. Switch power ranges are typical values for 51.2T (Tomahawk 5) and 25.6T (Tomahawk 4) class boxes with optics populated; confirm exact figures on each platform datasheet.
Matching OcNOS-DC platforms
The fabric runs a single OcNOS-DC image on open Broadcom Tomahawk hardware. These are the supported 800G and 400G platforms it designs onto.
This tool provides a structural network-design estimate for planning only. It is not a performance guarantee, a bill of materials, or a cost estimate. Actual designs depend on rail optimization, NIC count per GPU, cabling, and failure-domain choices. GPU counts are reference-design ceilings from Clos radix math, not measured figures. Broadcom and Tomahawk are trademarks of Broadcom Inc.; other names are the trademarks of their respective owners.
AI Fabric Reference Design
Get the reference design PDF: topology, switch counts, component summary, and a RoCEv2 lossless starting profile.
AI Fabric Reference Design
Quick form: your PDF will download immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
Frequently asked questions
How do I size a leaf-spine fabric for a GPU cluster?
What is rail-optimized topology?
How many transceivers does an AI fabric need?
Is RoCEv2 or InfiniBand better for AI?
How do you make an AI fabric lossless for RoCEv2?
Does the same OcNOS image run on every switch in the fabric?
Design the whole AI fabric with OcNOS
From the business case to the port-count maths, pick up wherever you are in the build.