Lossless 800G Ethernet fabrics for GPU clusters

OcNOS AI Fabric is the complete AI fabric IP Infusion delivers: OcNOS-DC preloaded on qualified 800G and 400G Broadcom Tomahawk switches from Edgecore or UfiSpace, with lossless RoCEv2, dynamic load balancing and PFC deadlock detection and recovery, under one support contract. Your NICs and GPUs stay your choice.

900+operators run OcNOS
60+countries
18qualified OcNOS-DC switches
Rail-optimized AI fabric, live viewTwo Tomahawk 5 spines connect to four rail leaves. Each GPU server sends one NIC to each rail leaf. Same-rail traffic stays on its leaf; cross-rail traffic crosses a spine.SpinesRail leavesGPU serversTomahawk 5 · 800G
Each GPU NIC rides its own rail. Same-rail traffic stays on its leaf; cross-rail traffic crosses a spine.
Best for
Teams building GPU clusters on lossless 800G or 400G Ethernet, with their choice of NICs and GPUs.
Qualified on
Edgecore AIS800-64DUfiSpace S9321-64EEdgecore AS9736-64D

What a GPU fabric has to get right

An AI fabric is the Ethernet network that connects GPU servers. It has to carry RoCEv2 traffic without loss and keep every link busy, because one dropped packet stalls a collective while the NIC resends it, and one congested link holds back the training job.

Three traffic patterns stall training jobs. OcNOS-DC has a fix for each.

Same traffic, same four uplinks. What changes is how flows are placed.

Uplink 1
Uplink 2
Uplink 3
Uplink 4

Static ECMP pins each large flow to one uplink by hash, so two uplinks jam while two sit almost idle.

Illustrative.

Every GPU talks at once
Static ECMP pins big flows to one uplink, so some links jam while others sit idle.
→ DLB moves flowlets from congested uplinks to less-loaded paths.
How DLB works →
Bursts hit one port
Many senders reach one port within microseconds.
→ ECN marking tells DCQCN-capable NICs to slow down before buffers fill.
→ PFC deadlock detection and recovery clears a stuck queue on its own.
How DCQCN works →
One drop stalls the job
The NIC has to resend lost data, so one drop stalls the collective.
→ PFC and ECN keep RoCEv2 traffic lossless.
How RoCEv2 works →

GLB, coming in OcNOS 7.1, will balance load across the whole fabric. About GLB →

Every GPU NIC on its own rail

Each GPU server connects one NIC to each of eight rail leaves. Same-rail AllReduce stays on its leaf; only cross-rail traffic reaches the 800G Tomahawk 5 spines. Pods scale on the same switches.

Rail-optimized AI fabric topology: four 800G Tomahawk 5 spines above eight rail leaves, with four GPU servers each mapping one of its eight NICs to a different rail leaf, so intra-rail AllReduce stays on one leaf.
Rail-optimized single pod: each GPU server's 8 NICs map one per rail to 8 dedicated leaves, so same-rail AllReduce stays on the leaf and only cross-rail traffic reaches the spine.

A 1,024-GPU rail-optimized fabric, worked switch by switch: port math, component counts, scaling to 4,096 GPUs and beyond, platforms and OcNOS-DC editions, and a RoCEv2 lossless starting profile.

Get the AI Fabric Reference Design See the reference topologies →

Technical summary

What OcNOS-DC runs in the fabric. Follow a link for the deep dive.

As of OcNOS-DC 7.0. Sources, checked October 2026: the OcNOS Feature Matrix (data center platforms), the Hardware Compatibility List and the AI Fabric Reference Design.

OcNOS-DC AI fabric technical summary
Area OcNOS-DC Deep dive
Silicon OcNOS AI Fabric supports Broadcom Tomahawk 5 and Tomahawk 4 for AI leaves and spines today. Tomahawk 6 support is planned for OcNOS 7.2.Sources: Hardware Compatibility List, Tomahawk 6 page Silicon →
Lossless transport RoCEv2 with PFC per priority, ECN marking and ETS. PFC over L3 across routed boundaries.Source: Feature Matrix (PFC with QoS, PFC over L3, ECN, ETS) RoCEv2 →
Congestion control ECN marking for DCQCN-capable NICs, with Dynamic ECN on Tomahawk 5. Switch packet buffer tuning.Source: Feature Matrix (ECN, Dynamic ECN, switch packet buffer tuning) DCQCN →
Load balancing Dynamic load balancing (DLB) on Tomahawk 4 and 5, with RoCEv2 destination-QP hashing, and Reactive Path Rebalance on Tomahawk 5. GLB, coming in OcNOS 7.1, extends it across the fabric.Sources: Feature Matrix (DLB, DLB Reactive Path Rebalance), OcNOS-DC System Management Guide, GLB page DLB →
Deadlock protection PFC deadlock detection and recovery.Source: Feature Matrix (PFC deadlock detection and recovery) PFC deadlock →
Server settings DCBX exchanges PFC and ETS settings with the link peer over LLDP.Source: Feature Matrix (DCBX, DCBX for PFC over L3)
Telemetry gNMI streaming over OpenConfig models: on-change, dial-in and dial-out.Source: Feature Matrix (gNMI dial-in, dial-out and on-change, OpenConfig models) Telemetry →
Automation ZTP over IPv4 and IPv6. Ansible playbooks from Day 0 to Day 2. IP Maestro over NETCONF.Source: Feature Matrix (ZTP over IPv4 and IPv6, Ansible, NETCONF) Automation →
AI agents gNMI telemetry, NETCONF with YANG models and commit confirmed with automatic revert, so AI agents and AIOps tools can read, change and roll back the fabric.Source: Feature Matrix (NETCONF confirmed commit, CLI commit confirmed) AI agents →
Topologies Rail-only, rail-optimized two-tier and rail-optimized three-tier.Source: AI Fabric Reference Design, October 2026 Topologies →
Scale Two tiers to 4,096 GPUs at 800G per GPU on Tomahawk 5. Three tiers to 16,384 GPUs. Reference-design ceilings from Clos radix math, not measured figures.Source: AI Fabric Reference Design, October 2026 Sizing →
Ultra Ethernet IP Infusion is a member of the Ultra Ethernet Consortium.Source: Ultra Ethernet page UEC →

Scale-across: one GPU cluster in several buildings

Large GPU clusters now need more power than one building can supply, so operators join the fabrics in several buildings and sites into one cluster. OcNOS-DC runs the spines at each site, and OcNOS-SP runs the deep-buffer routers between regions.

Coherent optics in the spine port

On the qualified Tomahawk 4 and Trident 4 switches, OcNOS-DC supports 400G ZR/ZR+. On the Tomahawk 5 Edgecore AIS800-64D and UfiSpace S9321-64E, it supports 800G ZR/ZR+. The spine connects directly to the OLS / ROADM, without transponder or muxponder shelves.

How coherent DCI works

Congestion signals on routed links

ECN marking runs on routed interfaces on all 18 qualified OcNOS-DC switches, so DCQCN-capable NICs can detect congestion on routed links between buildings. BFD on BGP sessions detects path failures.

How DCQCN works

Deep-buffer routers between regions

Between regions, OcNOS-SP runs on deep-buffer Jericho 2C+ and Qumran 2C routers with SR-MPLS, EVPN and BGP. Seven OcNOS-SP routers support 400G ZR/ZR+.

See ZR support by platform

What OcNOS AI Fabric does not do

These limits help you plan the rest of the cluster.

InfiniBand
It is an Ethernet fabric. It does not switch InfiniBand.
Collectives
It runs the switch side. The collective library (xCCL: NCCL, RCCL or oneCCL) and the RDMA endpoint run on the NIC and host. The cluster scheduler places jobs.
Scale-up
It connects servers. GPU-to-GPU links inside a server or rack (scale-up links) are outside the fabric.
Ultra Ethernet
It runs RoCEv2 today. The Ultra Ethernet transport runs on UEC-capable NICs. IP Infusion is a member of the Ultra Ethernet Consortium.
Tomahawk 6 and GLB
Tomahawk 6 is not yet supported. OcNOS-DC support is planned for OcNOS 7.2. GLB is coming in OcNOS 7.1.
Tomahawk 5 only
Dynamic ECN and Reactive Path Rebalance run on Tomahawk 5 switches only.
GPU counts
GPU counts on this page are reference-design ceilings from Clos radix math, not measured figures. GPU-side optics and cables are not counted.

Size it to your GPU count

Pick a cluster size and see the fabric it takes. Then size your own in the Design Suite, including link, transceiver and cable counts.

SpinesRail leavesGPU servers123456781234567812345678123456781-3233-6465-9697-128
A 1,024-GPU rail-optimized pod on Broadcom Tomahawk 5 uses 32 leaf switches and 16 spine switches, 48 switches in all, in a 1:1 non-blocking design. Each leaf takes one NIC from each of 32 servers, so same-rail traffic stays on the leaf. These are reference-design figures from Clos radix math, not measured figures.
  • 16 spines, each with 64 × 800G.
  • 32 leaves, each with 32 × 800G down, split 2 × 400G per GPU, and 32 × 800G up.
  • 128 servers, each with 8 GPUs and 8 NICs.

1,024 GPUs

32
leaves
16
spines
48
switches

Two tiers

Rail-optimized reference designs on Tomahawk 5. GPU counts are reference-design ceilings from Clos radix math, not measured figures.

Open the AI Fabric Design Suite Book an architecture review

Qualified 800G and 400G switches

Tomahawk 5 leaves and spines, and a Tomahawk 4 leaf, from Edgecore and UfiSpace.

Edgecore AIS800-64D 800G spine switch

Edgecore AIS800-64D

64 x 800G · 51.2 Tbps

QSFP-DD800 · breakout to 2x400G / 4x200G / 8x100G

Leaf or spine · Tomahawk 5
Datasheet →
UfiSpace S9321-64E 800G spine switch

UfiSpace S9321-64E

64 x 800G · 51.2 Tbps

QSFP-DD800 · breakout to 2x400G / 4x200G / 8x100G

Leaf or spine · Tomahawk 5
Datasheet →
Edgecore AS9736-64D 400G leaf switch

Edgecore AS9736-64D

64 x 400G · 25.6 Tbps

QSFP56-DD 400G · breakout to 4x100G

Leaf · Tomahawk 4
Datasheet →

See all 18 qualified OcNOS-DC switches →

One system, one contract

One support path

OcNOS-DC preloaded and qualified per switch model and ASIC stepping. One PO, one support contract, one RMA path. P1 response from 8 hours (Standard) to 30 minutes (Enterprise).

See support details

One console for the fleet

IP Maestro manages topology, faults, configuration and software images over NETCONF, beneath your OSS.

Explore IP Maestro

Optics with proven multi-vendor interoperability

Optics from multiple vendors have proven interoperability. Ask us for the transceiver list for your build.

Ask for the transceiver list

Take the design to your team

Short PDFs. A quick form, then the download starts.

800G lossless AI fabric

OcNOS-DC editions, qualified platforms and deployment architecture.

Get the brief

OcNOS-DC datasheet

Feature set, software SKUs and supported hardware.

Get the datasheet

Frequently asked questions

Can AI agents and AIOps tools operate an OcNOS AI fabric?
Yes. Every qualified OcNOS-DC switch exposes NETCONF with YANG models and commit confirmed with automatic revert. gNMI streaming telemetry is available in the PLUS edition. An agent can read fabric state, apply a change on trial and let OcNOS revert it if the change is not confirmed. The Feature Matrix lists each edition and switch.
Which AI fabric mechanisms run in the OcNOS-DC switch?
OcNOS-DC implements the switch-side mechanisms AI workloads depend on: lossless RoCEv2 with PFC and ECN, Dynamic ECN on Tomahawk 5, dynamic load balancing (DLB), switch packet buffer tuning, and PFC deadlock detection and recovery. Collective logic (xCCL), the NIC side of congestion control such as DCQCN, and job scheduling run in the NIC and the cluster scheduler, outside the switch.
Where does OcNOS-DC stop, and where do the NIC and cluster scheduler take over?
OcNOS-DC runs the switch side: PFC and ECN for lossless RoCEv2, dynamic load balancing, PFC deadlock detection and recovery, and gNMI streaming telemetry. The NIC runs the collective library (xCCL) and the RDMA endpoint. The cluster scheduler places jobs and can use the switch telemetry. That separation keeps your NIC, GPU, and orchestration swappable.
What does IP Infusion deliver for an AI fabric?
A complete system: qualified open switches with OcNOS-DC preloaded, delivered as OcNOS Systems under one support contract, from the same OcNOS family that runs service-provider networks. OcNOS-DC runs on Broadcom silicon with lossless RoCEv2 (PFC and ECN), dynamic load balancing with Reactive Path Rebalance on Tomahawk 5, and PFC deadlock detection and recovery. Each switch runs the OcNOS-DC edition that fits its role: DC-PLUS for the full AI fabric set, DC-IPBASE for the lossless basics, and DC-MGMT for out-of-band management. Your NIC, GPU, and switch vendor stay your choice.
What does Ultra Ethernet (UEC) 1.0 mean for OcNOS AI Fabric?
OcNOS-DC runs lossless RoCEv2 with PFC and ECN, and dynamic load balancing, available today, and IP Infusion is a member of the Ultra Ethernet Consortium (UEC). UEC 1.0 defines a multipath Ethernet transport that spreads one flow across many paths instead of pinning it to a single ECMP hash, with that endpoint behavior running on UEC-capable NICs. To plan how UEC fits a specific build, talk to an IP Infusion engineer. Read the Ultra Ethernet deep-dive.
How does OcNOS-DC keep the fabric lossless, and what protects against PFC deadlock?
Three mechanisms: PFC pauses per-priority traffic before buffers overflow, ECN marks packets early to slow senders, and ETS assigns link bandwidth to the RDMA class. On top, PFC deadlock detection and recovery detects a queue stuck in a pause cycle and recovers it, so a stalled port recovers without a manual reset. PFC over L3 is supported across routed boundaries.
What is DLB, and what changes with GLB in OcNOS 7.1?
During AllReduce your biggest flows collide when standard ECMP pins each one to a single uplink for its lifetime, and the sync waits on the congested link. DLB moves flowlets off congested uplinks to less-loaded paths at the local hop today, and Tomahawk 5 adds Reactive Path Rebalance. GLB, coming in OcNOS 7.1, extends load balancing across the whole fabric.
What switch hardware and 800G optics does the OcNOS AI fabric support?
The fabric runs on qualified Broadcom Tomahawk 5 and Tomahawk 4 switches from Edgecore and UfiSpace. The 800G switches (Edgecore AIS800-64D and UfiSpace S9321-64E, Tomahawk 5, 64 x 800G QSFP-DD800, 51.2 Tbps) serve as leaves and spines and break out to 400G, 200G, and 100G. The 400G leaf (Edgecore AS9736-64D, Tomahawk 4, 64 x 400G QSFP56-DD, 25.6 Tbps) breaks out to 4 x 100G and fits fabrics with one 400G NIC per GPU. Optics from multiple vendors have proven interoperability; contact us for the transceiver list for your build. The qualified switches are listed in the HCL.
Can an OcNOS AI fabric span several buildings or sites?
Yes. ZR/ZR+ coherent optics plug into the spines, ECN marking runs on the routed links between buildings, and OcNOS-SP runs on deep-buffer routers between regions. Read the coherent DCI deep dive.
Who provides support for the AI fabric, and is the hardware covered too?
When the fabric is bought as OcNOS Systems, one IP Infusion contract covers the OcNOS-DC software and the qualified switch hardware together, so there is one TAC and one SLA. Support is tiered: Standard includes email TAC, business-hours phone, and hardware RMA coordination with an 8-hour P1 response; Premium adds a 24/7 customer support portal and a 2-hour P1 response; Enterprise adds a 30-minute P1 response and a Customer Success Manager. The 24/7 portal is available on Premium and Enterprise.

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

You are here Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract.
Reference designs AI Fabric Topologies Rail-optimized two-tier and three-tier fabrics, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA? AI networking switches
The technology inside RoCEv2 lossless Rail-optimized network DLB GLB (7.1) Ultra Ethernet Tomahawk 6 at 1.6T DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI

Plan your GPU fabric with our engineers

Bring your design questions to an IP Infusion engineer, or take the reference design to your team.

Book an architecture review Get the AI Fabric Reference Design