Deadlock recovery · RoCEv2 · Lossless

PFC Deadlock Detection & Recovery

A PFC deadlock is the one failure that can hang a lossless fabric indefinitely: a cyclic buffer dependency where paused priority queues wait on each other. OcNOS-DC deadlock detection and recovery runs per port and per priority. It detects the paused-queue cycle and auto-drains the queue before jobs hang, on Broadcom Tomahawk 3, 4 and 5, Trident 3 (X4, X5, X7) and Trident 4.

Book an architecture review Open the AI fabric design suite
Per-portand per-priority deadlock detection
Configurabledetection and recovery timers
TH3 to TH5plus Trident 3 (X4, X5, X7) and Trident 4
OcNOS-DC PLUSlicense tier

A 3-switch pause cycle

Three switches in a circular dependency. Each is paused on its lossless priority queue, waiting for the next switch to drain. Without intervention, the cycle is stable forever. OcNOS deadlock detection triggers after the configured timeout, drains queue 3 on Switch-A, and the cycle collapses.

PFC deadlock cycle and its recovery Three switches arranged in a triangle. Pause arrows point clockwise from each switch to the next, indicating each switch is paused waiting for the downstream switch. A timer icon at Switch-A shows the OcNOS deadlock detection timer expiring to drain the queue and break the cycle. PFC pause (CoS 3) PFC pause PFC pause WD timer expires drain queue 3 Switch-Apaused on Q3waiting on B Switch-Bpaused on Q3waiting on C Switch-Cpaused on Q3waiting on A PFC DEADLOCK · DETECTION TIMEOUT · QUEUE DRAIN · AUTO-RECOVERY

How PFC creates a deadlock

PFC is what makes RoCEv2 lossless, and on a well-behaved topology it never causes trouble. The failure mode appears only when a paused-queue cycle closes on a multi-path fabric.

PFC is a hop-by-hop pause: a switch asserts pause to its upstream when its lossless ingress queue fills past the threshold, and the upstream stops sending. On a tree topology with a single direction of flow, that is safe. On a leaf-spine fabric with multiple paths, ECMP rerouting around a link failure can, under specific conditions, create a circular path where every switch is paused waiting for the next. That cyclic buffer dependency, paused priority queues waiting on each other, is a PFC deadlock.

Once the cycle forms it is stable: there is enough buffer to hold the paused frames, the routing protocol believes everything is converged, and PFC keeps reasserting on every switch. Without intervention, the affected lossless priority hangs indefinitely. RoCEv2 traffic stops, xCCL (RCCL, oneCCL) collectives time out, and the training job stalls.

OcNOS-DC deadlock detection and recovery

A per-port, per-priority timer that detects the paused-queue cycle and drains the affected queue before jobs hang, then restores normal PFC on its own. Detection, recovery, and restoration are the core triple; tuning and licensing round it out.

Detection

Per-port, per-priority timer

A timer runs per ingress port and per lossless priority. If the priority is paused continuously for the configured detection period, deadlock detection triggers and recovery starts.

Recovery

Automatic queue drain

When detection triggers, recovery runs for the configured recovery time. By default the paused traffic is forwarded so the cycle collapses. If the recovery action is set to drop, those frames are dropped and the RDMA NIC retransmits them.

Restoration

Auto-restore after recovery

After the configured restore interval, normal PFC operation resumes on the affected priority. No operator intervention required.

Tunable

Operator-configurable timers

Detection multiplier, time granularity, and recovery time are CLI-configurable per interface. The operator can shorten timers on high-stakes clusters.

Scope

DC-PLUS license tier

Part of the OcNOS-DC PLUS SKUs. Supported on Broadcom Tomahawk 3, Tomahawk 4, Tomahawk 5, Trident 3 (X4, X5, X7) and Trident 4 platforms.

Deadlock recovery is a safety net under the rest of the lossless stack

The best deadlock is the one that never forms. Congestion control, the lossless transport it protects, and dynamic load balancing all keep queues off the pause threshold in the first place; deadlock recovery is the last line when they do not.

Congestion control

DCQCN marks and paces flows with ECN before ingress queues fill to the PFC threshold, so pause is asserted far less often and cycles rarely get the chance to close.

DCQCN congestion control

Lossless transport

RoCEv2 is the RDMA transport PFC exists to protect. Deadlock recovery is what makes running RoCEv2 losslessly at scale safe rather than theoretically risky.

RoCEv2 lossless fabric

Dynamic load balancing

DLB rebins flowlets away from congested uplinks, spreading load so ingress queues are less likely to reach the threshold that triggers pause.

Dynamic load balancing (DLB)

Rare does not mean never, and that is the whole point

In a well-designed fabric with proper topology and routing, PFC deadlocks are rare. Deadlock recovery matters because the conditions can still arrive.

What triggers it

A link failure during a routing convergence window, a misconfiguration of PFC priorities on a single port, or a transient congestion event on an unusual traffic pattern can all create the conditions.

Without deadlock recovery

When a deadlock does form, the AI cluster stops and the operations team is left chasing a hang that the routing protocol reports as healthy.

With deadlock recovery

You get a brief retransmission burst and a logged event instead of a stalled job. Lay out the topology and switch tiers that keep deadlocks rare with the AI Fabric Sizing Tool.

AI Fabric Sizing Tool

Why deadlock recovery makes PFC safe to deploy at scale

It turns the one theoretical objection to lossless Ethernet into a logged, self-clearing event.

  • Lossless safety net. Deadlock recovery is the difference between "PFC is theoretically risky on production AI fabrics" and "PFC is safe to deploy at scale."
  • Automatic recovery. Recovery runs for the configured recovery time, then normal PFC resumes. By default the paused traffic is forwarded rather than dropped.
  • Per-interface configuration. Deadlock detection and recovery is enabled on each interface that carries lossless traffic, together with its detection and recovery timers.
  • Tunable for stakes. Lower the timers on critical training clusters. See PFC in the feature matrix and the validated hardware list.

PFC deadlock, answered

What causes a PFC deadlock?
PFC is a hop-by-hop pause. On a multi-path fabric, ECMP rerouting around a link failure can, under specific conditions, create a circular path where every switch is paused waiting for the next. The cyclic buffer dependency is stable and the lossless priority hangs until something breaks it.
How does OcNOS PFC deadlock recovery work?
A per-port, per-priority timer fires if a lossless priority stays paused for the configured detection period. Recovery then runs for the configured recovery time and ends on its own, with no operator intervention. By default the paused traffic is forwarded during recovery; OcNOS can be configured to drop it instead.
Does deadlock recovery drop traffic?
Not by default. During recovery OcNOS forwards the paused traffic. If the recovery action is set to drop, frames in the recovery window are dropped and the RDMA NIC transport retransmits them.
Is PFC deadlock detection enabled by default?
Deadlock detection and recovery is enabled per interface, where the detection and recovery timers are set from the CLI. It is part of the OcNOS-DC PLUS SKUs on Broadcom Tomahawk 3, Tomahawk 4, Tomahawk 5, Trident 3 (X4, X5, X7) and Trident 4 platforms.

Go deeper. Take it with you.

The product datasheet plus short, technical downloads that go further than this page.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Non-blocking RoCEv2 fabric on Broadcom Tomahawk 4/5 spines: SKU tiers, validated platforms, and deployment architecture.

Get the brief
Solution Brief

EVPN-VXLAN Data Center Fabric

Carrier-grade leaf-spine data center fabric: symmetric IRB, Type-2/Type-5 routes, and distributed anycast gateway.

Get the brief

Validating lossless behaviour on a new fabric? Start with deadlock recovery

Tell us the workload and the GPU scale, and an IP Infusion engineer will review your lossless design with you, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.

Book an architecture review Open the AI fabric design suite

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs AI Fabric Topologies Rail-optimized two-tier and three-tier fabrics, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA? AI networking switches
The technology inside RoCEv2 lossless Rail-optimized network DLB GLB (7.1) Ultra Ethernet Tomahawk 6 at 1.6T DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI