How to choose an AI networking fabric
Your GPUs run at their rated throughput only if the fabric under them keeps up. This is a vendor-neutral framework for scoring an AI fabric on the six criteria that decide it, before you sign.
Read the fabric as one decision, made of six
An AI networking fabric is bought once and lived with for years, so score it on how six properties fit the way you will run the cluster, not on a single headline number. Three kinds of fabric sit on the table:
- Closed single-vendor stacks bundle the NIC, switch, and scheduler into one SKU and set the trade-offs for you.
- Community and open-source NOS options give you code but leave integration and support on your desk.
- Open commercial NOS options keep hardware choice while carrying a support contract.
The criteria below score any of them on the same terms.
Performance and losslessness
AI collectives use RDMA, and RDMA needs a lossless substrate. RoCEv2 delivers that on Ethernet, but only with the full congestion toolkit, not a checkbox. A fabric that pauses badly or drops under load is the most common cause of stalled training throughput, so settle this criterion first.
What to ask
- Do you deliver the lossless Ethernet fabric RoCEv2 requires, including PFC over L3 with DCBX and LLDP?
- Is ECN marking present, and is there a dynamic variant that adapts thresholds under load?
- Is there PFC deadlock detection and recovery, plus WRED and buffer tuning?
- How is tail latency held down when a link gets hot during a collective?
How to evaluate
Ask for the exact congestion features and confirm they run on the silicon you will buy. A credible answer names PFC (including over L3, with DCBX and LLDP), ECN and Dynamic ECN, ETS, WRED, PFC deadlock detection and recovery, and dynamic load balancing that rebalances flows off hot links. A static hash can leave large flows colliding on a subset of links, while dynamic load balancing on Tomahawk 4 and Tomahawk 5 targets fabric utilisation in the 90% range, so treat it as a throughput feature, not a nicety. See the RoCE vs InfiniBand comparison for how the lossless model differs from InfiniBand credit flow.
Scale and topology fit
The right topology is the smallest non-blocking design that keeps every GPU's link saturated during a collective, with a clean path to the next size up. A topology that cannot grow forces a forklift later, so size for where the cluster is going, not where it starts.
What to ask
- Is the GPU plane 1:1 non-blocking, and where does oversubscription begin?
- What is the largest cluster this design reaches before a tier has to be added?
- Does it support rail-optimized wiring and a super-spine tier for multi-pod scale?
How to evaluate
Map candidates against concrete GPU counts. On Tomahawk 5 switches, a 2-tier leaf-spine reaches 4,096 GPUs at 1:1 non-blocking with 800G per GPU, each switch run as 128×400G and each GPU on two 400G leaf ports. Above that, a super-spine tier over the same rail-optimized pods extends to 16,000+ GPUs. Use rail-optimized wiring for single-pod locality and a Clos for multi-pod scale; the reference topologies show the port maths, and the AI Fabric Design Suite gives a first cut.
Openness and lock-in
A closed AI fabric ties the NIC, switch, and scheduler to one supplier, which sets your pricing and roadmap for you. An open approach runs a commercial NOS on standard Ethernet silicon from more than one hardware vendor, so you keep hardware choice and can second-source. This is where multi-year cost and negotiating leverage are decided.
What to ask
- Can the switch, the NOS, and the optics each be sourced independently?
- Is the fabric built on published, standards-based Ethernet or a proprietary transport?
- Is there more than one qualified hardware platform for the same software?
How to evaluate
Look for a qualified hardware list with more than one vendor behind it. Open commercial NOS options run the same software across Broadcom Tomahawk 5 platforms from different makers, for example Edgecore AIS800-64D and UfiSpace S9321-64E, keeping a second source real rather than theoretical. Contrast that with closed single-vendor NIC, switch, and scheduler SKUs where the parts only work together. The useful question is not one product versus another but how many suppliers you can hold accountable for the same fabric.
Operations, automation, and telemetry
A fabric that is hard to bring up or blind in production costs more than its price tag. The operational model, how you provision, observe, and tune, is a first-class buying criterion, not an afterthought once the hardware lands.
What to ask
- Is there streaming telemetry over gNMI with OpenConfig models?
- Is zero-touch provisioning (ZTP) supported for fabric bring-up at scale?
- Does it automate with standard tools such as Ansible, and signal QoS with DCBX?
How to evaluate
Ask to see the telemetry and automation in the candidate's own docs, not a slide. A strong answer covers gNMI and OpenConfig streaming telemetry into your observability stack, ZTP for provisioning, DCBX and LLDP for lossless signalling, and Ansible-based configuration. The point is closed-loop tuning during bring-up: per-queue counters and buffer state you can watch while you shake out PFC and ECN thresholds, not a black box you set once and hope.
Support and accountability
When a training run stalls at 3 a.m., the question is who owns the fix. A multi-vendor build with no single accountable contract turns into finger-pointing between the NIC, switch, optics, and software vendors. The remedy is one contract, not one closed stack.
What to ask
- Is there a single support contract covering software and the qualified hardware?
- Is there one TAC and one SLA, or several to coordinate?
- Who validated the specific hardware and software combination you are buying?
How to evaluate
Separate accountability from lock-in: you want single-vendor accountability without single-vendor hardware. The strong pattern is open hardware choice paired with one support agreement, one contract covering the software plus the qualified hardware, one TAC, one SLA, so a fault is owned end to end even though you sourced the parts openly.
Standards and roadmap
AI fabric transport is moving fast. The Ultra Ethernet Consortium (UEC) publishes an open multipath Ethernet transport for AI and HPC, with endpoint behavior that runs on UEC-capable NICs, so separate what each vendor runs in production today from what it plans.
What to ask
- Is the vendor a member of the Ultra Ethernet Consortium?
- What does the fabric run in production today, and on which silicon?
- What is on the near-term roadmap for load balancing and congestion control?
How to evaluate
Separate what ships from what is planned. IP Infusion is a member of the Ultra Ethernet Consortium, and OcNOS-DC runs lossless RoCEv2 with PFC, ECN marking, and DLB in production today on Broadcom Tomahawk 4 and Tomahawk 5 switches. DCQCN runs on the NICs and reacts to those ECN marks. Global Load Balancing (GLB) is coming in OcNOS 7.1. On roadmap, ask for named items: latency-based ECN (7.1.0), link-level retry, credit-based flow control, and packet trimming. See InfiniBand vs Ethernet for how the open path compares.
The buyer checklist
Run every candidate fabric, closed single-vendor stack, community NOS, or open commercial NOS, through the same list. If a box cannot be ticked, ask why before you decide it does not matter.
- Lossless substrate. RoCEv2 with PFC (including over L3, DCBX, LLDP), ECN and Dynamic ECN, ETS, WRED, and PFC deadlock detection and recovery.
- Dynamic load balancing. Flows rebalance off hot links, not just static ECMP hashing.
- 1:1 non-blocking GPU plane. No oversubscription on the GPU fabric, sized for your target GPU count with headroom to the next tier.
- Topology fit. Rail-optimized single-pod and three-tier multi-pod designs on the silicon you are buying.
- Hardware choice. More than one qualified hardware vendor for the same NOS, so a second source is real.
- Open transport. Standards-based Ethernet, not a proprietary NIC, switch, and scheduler bundle.
- Telemetry and automation. gNMI and OpenConfig streaming telemetry, ZTP, DCBX, and Ansible.
- One accountable contract. Single support agreement across software and hardware bought as one system, one TAC, one SLA.
- Standards roadmap. Vendor is a member of the Ultra Ethernet Consortium and names its load-balancing and congestion-control roadmap items.
Where open Ethernet with OcNOS-DC fits
Read the criteria above and pick whatever scores best for your cluster. For buyers who want open hardware choice with a single accountable contract, here is how IP Infusion lines up against the same checklist, stated plainly so you can compare it with the alternatives.
Production RoCEv2 fabric
OcNOS-DC delivers the lossless Ethernet fabric RoCEv2 requires on Broadcom Tomahawk 5: PFC over L3 with DCBX and LLDP, ECN and Dynamic ECN, ETS, WRED, PFC deadlock detection and recovery, and DLB with reactive path rebalance.
4,096 to 16,000+ GPUs
A 2-tier TH5 pod reaches 4,096 GPUs at 1:1 non-blocking with 800G per GPU; above that, a super-spine tier extends to 16,000+ GPUs.
Multi-vendor hardware
One NOS across Tomahawk 5 platforms from more than one maker, including Edgecore AIS800-64D and UfiSpace S9321-64E and S9321-64EO with 400G ZR+, so a second source stays real.
Open telemetry and automation
gNMI and OpenConfig streaming telemetry, ZTP, DCBX, and Ansible, with RTAG7 hashing and buffer tuning for closed-loop bring-up.
OcNOS Systems: one contract, one TAC, one SLA
With OcNOS Systems, a single IP Infusion agreement covers the OcNOS software plus the qualified hardware, so you get single-vendor accountability without single-vendor lock-in.
UEC member, named roadmap
IP Infusion is a member of the Ultra Ethernet Consortium. GLB is coming in OcNOS 7.1, and latency-based ECN is on the OcNOS 7.1 train for OcNOS-DC.
AI fabric buyer guide FAQ
What should I evaluate first when choosing an AI fabric?
Is Ethernet or InfiniBand better for AI?
How many GPUs do I need to plan for?
How do I avoid vendor lock-in?
What does UEC mean for my purchase?
Do I need one vendor for hardware and software?
Design the whole AI fabric with OcNOS
From the business case to the port-count maths, pick up wherever you are in the build.
Go deeper. Take it with you.
Two short, technical downloads that go further than this page: the full OcNOS-DC datasheet and the lossless 800G AI fabric architecture.

OcNOS-DC Datasheet
Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.
Get the datasheet
OcNOS 800G Lossless AI Fabric
Lossless RoCEv2 fabric on Tomahawk 5 spines: SKU tiers, qualified platforms, and deployment architecture.
Get the briefOcNOS-DC Datasheet
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
OcNOS 800G Lossless AI Fabric
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.