For the last few years, the limit on AI training was how many GPUs you could buy. Now it is how much power a single site can get. When a campus runs out of megawatts, the next cluster goes somewhere else, often tens or hundreds of kilometers away, and the training job has to follow.
The industry now calls this scale-across. NVIDIA built Spectrum-XGS around it, Broadcom aimed Jericho4 at it, and Microsoft links its Fairwater sites into a single AI WAN. For network teams, it means more than a fatter pipe between buildings. Scale-across turns data center interconnect (DCI) into a routing problem.
From Pipe to Fabric
Traditional DCI had a simple job: move replication, backup, and spillover traffic between sites, reliably and cheaply. The hard questions were optical: wavelengths, coherent optics, fiber. Routing was an afterthought, often a few static paths or a stretched Layer 2 domain.
Distributed training rewrites that contract. Even when teams keep tensor and pipeline parallelism inside each site and send only gradient synchronization across, the DCI link becomes part of the training fabric. Every step waits for the slowest transfer, so the interconnect sets the pace of the cluster itself.
Why It's a Routing Problem
Five traits of scale-across traffic push DCI into the routing domain:
- Hashing breaks down. AI traffic is a few very large, long-lived flows. ECMP hashing can pile several onto one link while parallel paths sit idle.
- Path latency matters as much as bandwidth. Collectives run at the speed of the slowest member. Light in fiber covers about 200 km per millisecond, so a route 200 km longer adds a millisecond each way to every synchronization.
- A failure stalls the whole job. One link fault can idle thousands of GPUs, so sub-50 ms protection becomes a direct cost control.
- Long-haul capacity is the expensive part. WAN bandwidth costs far more per bit than campus links, so it has to run hot without congesting.
- Neoclouds are multi-tenant. Scale-across sold as a service needs hard tenant isolation across sites.
Service provider networks solved each of these problems years ago. What's new is that these problems now sit on the critical path of a GPU cluster.
Isn't This a Transport Problem?
A fair objection is that this is really about congestion control and buffering: distance-aware transport like Spectrum-XGS, RoCE tuning, Ultra Ethernet. Those matter. But no endpoint can fix a network that sends two elephant flows down one link, or one that takes seconds to route around a fiber cut. Transport and routing have to work together, and routing is where most data center teams have the least experience.
What Scale-Across Networks Need, and Where OcNOS Fits
Scale-across needs a service provider's routing toolkit with a data center team's operating model. That is what OcNOS was built for, with an answer to each problem above:
- Hashing and latency: SR-TE policies steer traffic onto engineered low-latency or high-capacity paths, rather than trusting a hash.
- Failures: TI-LFA precomputes loop-free backup paths for sub-50 ms recovery.
- Long-haul cost: 100G and 400G ZR/ZR+ coherent optics managed from OcNOS remove standalone transponders.
- Multi-tenancy: EVPN multi-site DCI with route-target filtering keeps tenants isolated across sites.
- Visibility: On-change gNMI streaming telemetry shows congestion as it happens, so long-haul links can run hot safely.
The business case matters as much as the features. WAN links are among the costliest bits in an AI build, and operators can't afford to tie them to one vendor's roadmap. OcNOS on open hardware lets them pick the silicon, optics, and platforms that fit each site, from Tomahawk 6 at 102.4 Tbps for the fabric inside a site to Qumran3D, a 25.6 Tbps router with on-package HBM buffers, at its edge. OcNOS support for both is on the way, taking one operating system from the GPU fabric to the WAN.
Open-source options such as SONiC have earned their place in hyperscale fabrics, but they grew up as leaf-spine operating systems for teams with deep engineering benches. The features scale-across leans on hardest, such as SR-TE, TI-LFA, EVPN over MPLS, and coherent optics management, vary in maturity across distributions, leaving the operator to own integration and support. OcNOS delivers them as one tested, supported stack, already proven in service provider networks.
The Takeaway
Scale-across moves the bottleneck of AI training onto the wide-area network. Teams that treat DCI as a pipe will pay in idle GPUs and stalled jobs. Teams that treat it as a routing problem, with engineered paths, fast protection, and real-time visibility, will get more work out of every GPU they own.
The tools already exist, proven in the world's largest service provider networks. The task now is to bring them to the AI fabric, on open infrastructure that keeps the economics in the operator's hands.
Ready to design your scale-across interconnect? Explore OcNOS for AI data centers or contact us here.
