PFC Deadlock Detection & Recovery
A PFC deadlock is the one failure that can hang a lossless fabric indefinitely: a cyclic buffer dependency where paused priority queues wait on each other. The OcNOS-DC watchdog is per-port and per-priority. It detects the paused-queue cycle and auto-drains the queue before jobs hang, on Broadcom Tomahawk 3, 4 and 5, Trident 3 (X4, X5, X7) and Trident 4.
A 3-switch pause cycle
Three switches in a circular dependency. Each is paused on its lossless priority queue, waiting for the next switch to drain. Without intervention, the cycle is stable forever. The OcNOS watchdog fires after the configured timeout, drains queue 3 on Switch-A, and the cycle collapses.
How PFC creates a deadlock
PFC is what makes RoCEv2 lossless, and on a well-behaved topology it never causes trouble. The failure mode appears only when a paused-queue cycle closes on a multi-path fabric.
PFC is a hop-by-hop pause: a switch asserts pause to its upstream when its lossless ingress queue fills past the threshold, and the upstream stops sending. On a tree topology with a single direction of flow, that is safe. On a leaf-spine fabric with multiple paths, ECMP rerouting around a link failure can, under specific conditions, create a circular path where every switch is paused waiting for the next. That cyclic buffer dependency, paused priority queues waiting on each other, is a PFC deadlock.
Once the cycle forms it is stable: there is enough buffer to hold the paused frames, the routing protocol believes everything is converged, and PFC keeps reasserting on every switch. Without intervention, the affected lossless priority hangs indefinitely. RoCEv2 traffic stops, xCCL (RCCL, oneCCL) collectives time out, and the training job stalls.
The OcNOS-DC watchdog
A per-port, per-priority timer that detects the paused-queue cycle and drains the affected queue before jobs hang, then restores normal PFC on its own. Detection, recovery, and restoration are the core triple; telemetry, tuning, and licensing round it out.
Per-port, per-priority timer
A timer runs per ingress port and per lossless priority. If the priority is paused continuously for the configured detection period, the watchdog fires.
Automatic queue drain
On fire, recovery runs for the configured recovery time. By default the paused traffic is forwarded so the cycle collapses. If the recovery action is set to drop, those frames are dropped and the RDMA NIC retransmits them.
Auto-restore after recovery
After the configured restore interval, normal PFC operation resumes on the affected priority. No operator intervention required; the fabric is back to lossless within seconds.
gNMI counters
Watchdog fire events, drain durations, and per-priority pause counters stream over gNMI for closed-loop fabric monitoring. SREs see deadlock events as alerts, not as silent training stalls.
Operator-configurable timers
Detection multiplier, time granularity, and recovery time are CLI-configurable per interface. The operator can shorten timers on high-stakes clusters.
DC-PLUS license tier
Part of the OcNOS-DC PLUS SKUs. Supported on Broadcom Tomahawk 3, Tomahawk 4, Tomahawk 5, Trident 3 (X4, X5, X7) and Trident 4 platforms.
The watchdog is a safety net under the rest of the lossless stack
The best deadlock is the one that never forms. Congestion control, the lossless transport it protects, and adaptive routing all keep queues off the pause threshold in the first place; the watchdog is the last line when they do not.
Congestion control
DCQCN marks and paces flows with ECN before ingress queues fill to the PFC threshold, so pause is asserted far less often and cycles rarely get the chance to close.
Lossless transport
RoCEv2 is the RDMA transport PFC exists to protect. The watchdog is what makes running RoCEv2 losslessly at scale safe rather than theoretically risky.
Adaptive routing
DLB rebins flowlets away from congested uplinks in sub-millisecond windows, spreading load so ingress queues are less likely to reach the threshold that triggers pause.
Rare does not mean never, and that is the whole point
In a well-designed fabric with proper topology and routing, PFC deadlocks are rare, and most operators never see one in years of running. The watchdog matters because the conditions can still arrive, and when they do the difference is measured in hours.
What triggers it
A link failure during a routing convergence window, a misconfiguration of PFC priorities on a single port, or a transient congestion event on an unusual traffic pattern can all create the conditions.
Without the watchdog
When a deadlock does form, the AI cluster stops and the operations team has hours of debugging ahead, chasing a hang that the routing protocol reports as healthy.
With the watchdog
You get a brief retransmission burst and a logged event instead of a stalled job. Lay out the topology and switch tiers that keep deadlocks rare with the AI Fabric Sizing Tool.
Why the watchdog makes PFC safe to deploy at scale
It turns the one theoretical objection to lossless Ethernet into a logged, self-clearing event that your existing observability stack already sees.
- Lossless safety net. The watchdog is the difference between "PFC is theoretically risky on production AI fabrics" and "PFC is safe to deploy at scale."
- Automatic recovery. Recovery runs for the configured recovery time, then normal PFC resumes. By default the paused traffic is forwarded rather than dropped.
- Per-interface configuration. The watchdog is enabled on each interface that carries lossless traffic, together with its detection and recovery timers.
- Observable. Every fire is logged, counted, and streamed via gNMI for closed-loop monitoring with your existing observability stack.
- Tunable for stakes. Lower the timers on critical training clusters. See PFC in the feature matrix and the validated hardware list.
PFC deadlock, answered
What causes a PFC deadlock?
How does the OcNOS PFC watchdog recover the fabric?
Does the watchdog drop traffic?
Is the PFC deadlock watchdog enabled by default?
Go deeper. Take it with you.
The product datasheet plus short, technical downloads that go further than this page.
OcNOS-DC Datasheet
Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.
Get the datasheetOcNOS 800G Lossless AI Fabric
Non-blocking RoCEv2 fabric on Broadcom Tomahawk 4/5 spines: SKU tiers, validated platforms, and deployment architecture.
Get the briefEVPN-VXLAN Data Center Fabric
Carrier-grade leaf-spine data center fabric: symmetric IRB, Type-2/Type-5 routes, and distributed anycast gateway.
Get the briefOcNOS-DC Datasheet
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
OcNOS 800G Lossless AI Fabric
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
EVPN-VXLAN Data Center Fabric
Quick form. Your PDF opens in a new tab immediately after submit.
✓ Opening your PDF in a new tab…
If it didn't open, use the link below.
Validating lossless behaviour on a new fabric? Start with the watchdog
Tell us the workload and the GPU scale, and an IP Infusion engineer will review your lossless design with you, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.
Design the whole AI fabric with OcNOS
From the business case to the port-count maths, pick up wherever you are in the build.