首页 · 文档 · 技术 · AI Fabric 拓扑

AI Fabric 拓扑：Rail-Optimized 与调度式设计

The shape of your fabric decides the shape of your training job. This page lays out the reference topologies OcNOS-DC ships against, from a rail-optimized single pod, through a scheduled 3-stage Clos, to coherent multi-DC DCI, sized in concrete port-counts on Broadcom Tomahawk 4 and Tomahawk 5 hardware.

Which AI fabric topology should you use? Pick the smallest non-blocking design that keeps every GPU's link saturated during collectives. Up to about 1,000 GPUs, use a rail-optimized leaf-spine pod (8 rails per GPU server, one rail per leaf). From roughly 1,000 to 16,000+ GPUs, move to a 3-stage Clos (leaf, spine, super-spine). To span data centers, extend with 400G ZR/ZR+ coherent DCI. All three run on OcNOS-DC over a RoCEv2 lossless (PFC + ECN) L3 fabric.

按 GPU 数量选择，而非按流行术语

AI fabric 拓扑只有一个使命：保持 every GPU 的出向链路在集合操作期间达到饱和，同时不产生尾延迟离群值。合适的拓扑是能够针对您的 GPU 数量实现这一目标的最小规模拓扑，并具备扩展至下一更大规模的回退路径。以下为 OcNOS-DC 当前支持的三种参考设计，并附有具体的端口计算。

正在为自己的集群做容量规划？ The AI Fabric Design Suite 可提供快速的初步测算：它可测算出一个 无阻塞两级架构 假设条件下的 leaf-spine pod 每个 GPU 一块 fabric NIC，并在进入三层规模时予以提示。下面的参考设计采用 same 非阻塞的leaf/spine计算，并将其扩展为大规模的3级Clos，从而使交换机数量与工具相吻合。 Rail-optimized here is the wiring discipline of an 8-NIC GPU server (one rail per leaf, so intra-rail AllReduce stays on the leaf) layered on that non-blocking fabric: it changes traffic locality, not the switch count. Use the tool for a ballpark; use these designs for the build.

256GPUs

入门级无阻塞 pod

在小型 spine 层之上部署一行轨道对齐 leaf。两层折叠式 Clos，1:1 无阻塞。

8 个 leaf · 4 个 spine · TH4 · 400G
1,024GPUs

轨道优化的两层 pod

采用轨道对齐 leaf 加 1:1 无阻塞 spine。轨道内 AllReduce 保持在 leaf 上完成；跨轨道流量经由 spine。标准的单 pod 可扩展单元。

32 个 leaf · 16 个 spine · TH5 · 800G
4,096GPUs

3 级 Clos

Leaf, spine, super-spine. Each 1,024-GPU pod is 1:1 non-blocking; a super-spine plane scales across pods. DLB at every tier; GLB end-to-end on the OcNOS 7.1 train.

128 leaves · 64 spines · 32 super-spines · TH5 · 800G
16,384GPUs

规模化 3 级 Clos

带有超级骨干平面的多 Pod 三级 Clos。专为万亿参数训练级别设计。

512 leaves · 256 spines · 128 super-spines · TH5 · 800G
参考设计 1

轨道优化单 Pod

每台 GPU 服务器配备 8 个 NIC,每个对应一条 "rail",即专用 xCCL (NCCL / RCCL / oneCCL) 集体通信通道。每条 rail 拥有独立的专用叶交换机、因此每台服务器的 8 个 NIC 都落到不同的叶交换机上。跨 rail-N 的 AllReduce 保持在 leaf-N 内部。主导集体通信模式不会对 spine 产生东西向压力。

轨道优化的 AI fabric：8 条轨道、轨道对齐 leaf、1:1 无阻塞 spine（示意图） 轨道优化的 AI 网络结构。底部的八台 GPU 服务器各配备八块网卡，分别对齐八台 rail-leaf。每台服务器的 Rail-N 连接至 leaf-N。leaf 之上的 spine 层承载跨轨道流量。占主导地位的 AllReduce 流量保持在单一轨道内，从不经过 spine。 Spine-1TH5 · 800G Spine-2TH5 · 800G Spine-3TH5 · 800G Spine-4TH5 · 800G Rail-1leaf Rail-2leaf Rail-3leaf Rail-4leaf Rail-5leaf Rail-6leaf Rail-7leaf Rail-8leaf GPU Server 1 8 × NIC · 8 条 rail GPU Server 2 8 × NIC · 8 条 rail GPU Server 3 8 × NIC · 8 条 rail GPU Server 4 8 × NIC · 8 条 rail RAIL-OPTIMIZED · 8 RAILS · INTRA-RAIL ALLREDUCE STAYS LOCAL

OcNOS 组件： BGP-unnumbered L3 underlay, RoCEv2 lossless (PFC + ECN) on every leaf, DLB at the spine tier. Built on HCL-listed hardware: the 800G scalable unit uses TH5 64×800G leaves and spines (Edgecore AIS800-64D or UfiSpace S9321-64E); the entry 256-GPU pod uses Edgecore AS9736-64D (TH4, 64×400G).

调度式对比 Rail-Aligned：规模化时的变化

Rail-optimized stops scaling somewhere between 1k and 2k GPUs: you run out of leaf radix, or the spine tier becomes too oversubscribed. Above that, most modern AI fabrics move to a 3-stage Clos: leaf, spine, super-spine. The hard part on a Clos is spreading flows evenly so no link becomes a hot spot. Approaches run from per-flow ECMP, through adaptive (dynamic) load balancing, to per-packet spray, the model Ultra Ethernet uses with reordering handled at the NIC. A separate family, cell-based scheduled fabrics such as Broadcom DDC, segments traffic into cells and schedules it inside the fabric. OcNOS keeps the GPU plane balanced with DLB today and adds fabric-wide GLB on the 7.1 train, and is UEC-ready as UEC NICs arrive.

参考设计 2

3-Stage Clos Scheduled Fabric: 4,096-16,384 GPUs

Three tiers: leaf, spine, super-spine. Any two GPUs are at most four switch hops apart; same-leaf and same-pod peers are closer. Non-blocking within a pod, with the super-spine plane setting the cross-pod ratio. DLB at every hop, GLB across the full path on the OcNOS 7.1 train, UEC packet-spray on UEC-capable NICs. The diagram is schematic: it draws a reduced tier count; the 4,096-GPU build is 128 leaves / 64 spines / 32 super-spines on TH5 800G.

三级 Clos AI fabric 调度型拓扑 三级 Clos 拓扑。顶层为四台 super-spine 交换机。中层为八台 spine 交换机。底层为 12 台 leaf 交换机，向 GPU pod 馈送流量。leaf 到 spine、spine 到 super-spine 之间为全互联链路。底部条带标注：4096 GPU 调度网络、各层级 DLB、端到端 GLB（基于 OcNOS 7.1）。 Super-Spine-1 Super-Spine-2 Super-Spine-3 Super-Spine-4 Spine-1 Spine-2 Spine-3 Spine-4 Spine-5 Spine-6 Spine-7 Spine-8 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 SUPER-SPINE SPINE LEAF GPU PODS 128 个 leaf · 每 leaf 32 个 GPU · 共 4,096 个 GPU · TH5 · 800G 3-STAGE CLOS · 4,096 GPU · DLB EVERY HOP · GLB E2E（OcNOS 7.1）· UEC-READY

OcNOS 组件： eBGP-unnumbered L3 underlay、RoCEv2 无损传输 (PFC + ECN)、每一层的 DLB、OcNOS 7.1 系列上的端到端 GLB，以及向你的可观测性栈传输的 gNMI 流式遥测；在需要多租户隔离的场景中提供 EVPN-VXLAN 多租户 overlay。全程基于 HCL 列出的 TH5 64×800G 机箱构建。

Subscription is a dial, not a fixed rule. These counts make each 1,024-GPU pod 1:1 non-blocking and use a cost-optimized ~2:1 super-spine for cross-pod traffic, the rail-optimized approach hyperscale Ethernet fabrics rely on (published large-scale designs oversubscribe the top tier far more, because collective traffic stays pod-local). Want maximal any-to-any headroom instead? A fully non-blocking 1:1 build is 128 / 128 / 64 at 4,096 GPUs and 512 / 512 / 256 at 16,384; only the spine and super-spine counts change. Model either in the AI Fabric Design Suite.

面向分布式训练的多 DC 与 DCI

当单次训练任务跨越多个数据机房时（这在万亿参数模型中日益普遍），网络结构需要延伸至 WAN。OcNOS-DC 在 spine 上直接支持 400G ZR / ZR+ 相干光模块，实现免转发器的 DCI，并通过 EVPN 隧道扩展跨站点承载 VXLAN 租户。

参考设计 3

多数据中心 AI 网络：相干 DCI

两个 AI 数据中心通过 spine 上的 400G ZR/ZR+ 互联。EVPN inter-DC 承载 L2/L3 租户扩展；各站点底层的三级 Clos 保持不变。

采用 400G ZR/ZR+ DCI 的多 DC AI fabric 两个 AI 数据中心，各自采用 leaf-spine fabric。两个 spine 通过跨 WAN 的 400G ZR/ZR+ 相干光模块互联。EVPN 数据中心间隧道将租户从一个站点延伸至另一个站点。底部条带：无需转发器的相干 DCI。 DATA CENTER A DATA CENTER B Spine-A1400G ZR+ Spine-A2400G ZR+ Spine-B1400G ZR+ Spine-B2400G ZR+ EVPN inter-DC · 400G ZR/ZR+ Leaf-A1 Leaf-A2 Leaf-A3 Leaf-B1 Leaf-B2 Leaf-B3 GPU pod · 站点 A GPU pod · 站点 B 相干 DCI · 免转发器 · EVPN 跨数据中心 · 400G ZR/ZR+

OcNOS 组件： 400G ZR/ZR+ pluggable coherent optics on a DWDM-capable spine or border-leaf port, EVPN inter-DC for tenant L2/L3 extension, gNMI telemetry across sites. No external transponders required. Reach: 400ZR to roughly 120 km amplified; OpenZR+ reaches farther on oFEC.

设计经验法则

  • 使拓扑与 GPU 数量相匹配。 最小规模 pod（不超过单台 leaf 的 NIC 基数）：仅用 rail-only 即可。单 pod 规模：采用 rail 优化的 leaf-spine。多 pod 规模：三级 Clos 是唯一能在不牺牲超额订阅的前提下实现扩展的设计。
  • AI 平面始终保持 1:1 订阅比。 存储与 CPU 机架可承受更高的超额订阅比。GPU 平面则不应如此。
  • 轨道数应依据 xCCL 规划,而非布线便利。 对于 8-NIC GPU 服务器，8 轨是当前的事实标准。请勿将多条轨合并到更少的 leaf 上。
  • 按功耗和密度选择芯片，而非按品牌标识。 TH4（25.6T）和 TH5（51.2T）是主力芯片；二者之间的取舍在于机架功耗和 breakout 线缆成本。
  • 在设计阶段就为 GLB / UEC 做好规划。 从第一天起就将遥测平面构建到位，即便是在 7.0 fabric 上，这样 OcNOS 7.1 GLB 升级便纯粹是一次软件操作。详见 GLB and Ultra Ethernet.
  • 对照 HCL 进行验证。 此处的每个参考方案均构建于所列硬件之上，详见 OcNOS 硬件兼容性列表；从那里开始即可获得一流支持。
RoCEv2 无损网络 DLB：自适应路由 GLB: fabric-wide (7.1) Ultra Ethernet（UEC） EVPN-VXLAN BGP 底层 OcNOS 功能矩阵 已验证硬件（HCL） OcNOS-DC AI 架构解决方案
常见问题

AI fabric topology FAQ

What is a rail-optimized topology, and how is it different from rail-only?
Rail-optimized wiring connects each of a GPU server's 8 NICs to its own dedicated rail leaf, so the dominant same-rail AllReduce traffic stays on one leaf and never traverses the spine. Rail-only is the small-cluster case: a single rack-row of rail-aligned leaves with no spine tier, where cross-rail traffic relies on the GPU scale-up domain. Rail-optimized adds a non-blocking spine so cross-rail flows have a network path.
How many GPUs can a 3-stage Clos scale to?
On radix-64 switches (Tomahawk 5 at 800G) a 2-tier leaf-spine tops out at 2,048 GPUs. A 3-stage Clos with a super-spine tier extends that to 16,000+ GPUs in the reference designs above, and up to about 65,000 GPUs at the theoretical fat-tree limit. Because most collective traffic stays rail-local, the super-spine plane is sized to the cross-pod ratio you actually need.
Should I use Tomahawk 4 or Tomahawk 5?
Both run OcNOS-DC. Tomahawk 4 (25.6 Tbps, 64×400G) is the cost-optimized choice for entry pods and 400G GPU NICs. Tomahawk 5 (51.2 Tbps, 64×800G) is the workhorse for 800G GPU servers and larger fabrics. Tomahawk 4 has no native 800G, so match the switch to your NIC speed.
Do I need InfiniBand, or is Ethernet enough?
Ethernet is now a first-class AI-fabric transport. RoCEv2 with PFC and ECN delivers lossless RDMA today. Ultra Ethernet（UEC） removes the network-wide PFC dependency using endpoint packet-spray, selective retransmission, and link-level retry as UEC NICs ship. OcNOS-DC runs the RoCEv2 fabric today and is UEC-ready.
Where does scale-up end and scale-out begin?
Inside a GPU server and its NVLink domain (for example GB200 NVL72), GPUs communicate over the scale-up fabric at terabit speeds. The rail, leaf-spine, and Clos network is the scale-out fabric between servers and pods. Most same-rail collective traffic is absorbed by scale-up first, so the network carries the cross-rail and cross-pod remainder, which is why 1:1 non-blocking matters most on the GPU plane.

正在设计您的 AI fabric？我们与您一起完成端口数量的测算。

预约架构评审 →
AI 网络

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

解决方案 Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract.
您当前所在位置 AI Fabric 拓扑 Rail-optimized, scheduled 3-stage Clos, and coherent DCI, sized in real port counts.
Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
The technology inside RoCEv2 lossless DLB adaptive routing GLB (7.1) Ultra Ethernet DCQCN PFC 死锁 InfiniBand 与以太网对比 Coherent DCI
解决方案简介

深入了解，随身带走。

两份简短的技术下载资料，其内容比本页更为深入：lossless 800G AI fabric 架构与 EVPN-VXLAN data center 参考设计。

解决方案简报

OcNOS 800G 无损 AI Fabric

基于 Broadcom Tomahawk 4/5 spine 的无阻塞 RoCEv2 fabric：SKU 级别、经验证的平台以及部署架构。

获取简报
解决方案简报

EVPN-VXLAN 数据中心网络

carrier-grade 的 leaf-spine data center fabric：对称 IRB、Type-2/Type-5 路由，以及分布式 anycast 网关。

获取简报