ECN · CNP · RoCEv2

DCQCN: Quantized Congestion Control for RDMA

DCQCN (Data Center Quantized Congestion Notification) is the end-to-end congestion control that lets a lossless RoCEv2 fabric run without living in PFC pause. The switch marks ECN early, the receiver echoes a CNP, and the sender quantizes its rate, so senders slow down before queues overflow.

Bring us your traces Open the AI fabric design suite
Closed loopECN mark, CNP, rate cut
Before PFCECN marks ahead of pause

The DCQCN closed loop

Sender NIC, congested switch, receiver NIC. The switch's WRED ECN marker fires before the queue hits the PFC pause threshold. The receiver generates a Congestion Notification Packet (CNP); the sender's reaction point reduces rate, then ramps back. Lossless, no PFC pressure, fast convergence.

DCQCN closed-loop congestion control Three boxes left to right: Sender NIC (reaction point), Congested switch with WRED ECN marker, Receiver NIC (notification point). Forward arrow shows a packet flowing left to right and being marked by the switch. Reverse arrow shows the CNP traveling right to left back to the sender. Labels indicate ECN K-min, K-max thresholds and CNP feedback path. data + ECN-capable ECN = CE (marked) CNP: Congestion Notification Packet Sender NICReaction Pointα-update · cut · ramp OcNOS-DC SwitchWRED ECN markerK-min · K-max · P-max Receiver NICNotification PointCNP every 50 µs DCQCN · WRED MARKING · CNP FEEDBACK · QUANTIZED RATE LIMIT · LOSSLESS RoCEv2

The job DCQCN does in an AI fabric

RoCEv2 has two ways to handle congestion: PFC pause, which is back-pressure that propagates hop by hop, and DCQCN, an end-to-end rate-control loop. PFC alone works, but it pushes congestion upstream and risks pause storms and head-of-line blocking. DCQCN works ahead of PFC, marking packets with ECN before the queue reaches the pause threshold, so the sender slows down before the switch ever has to assert pause.

Done well, you spend most of your fabric life on DCQCN feedback alone, with PFC as a safety backstop. Done badly, ECN thresholds are misaligned with PFC headroom and you get pause storms even with DCQCN configured.

Reaction Point

Sender NIC quantizes its rate

Receives CNPs and runs the DCQCN α-update, multiplicative-decrease, additive-increase loop to quantize its sending rate.

RoCEv2 lossless fabric
Congestion Point

Switch marks ECN early

Marks ECN-capable packets to CE using a WRED curve when queue depth crosses K-min, with marking probability rising linearly to P-max at K-max.

PFC deadlock detection
Notification Point

Receiver echoes a CNP

Generates a CNP back to the sender on each marked flow, rate-limited to roughly one per 50 µs per flow.

PFC pause and DCQCN, side by side

Both keep RoCEv2 lossless, but they act at different points. The tuning goal is to spend the fabric's life on DCQCN feedback and hold PFC in reserve as a backstop.

Axis PFC pausebackstop DCQCNprimary control
MechanismHop-by-hop back-pressure that propagates upstream.End-to-end rate-control loop from sender to receiver.
Trigger pointFires when the queue reaches the pause threshold.Marks ECN before the queue reaches the pause threshold.
Failure modeRisks pause storms and head-of-line blocking.Sender slows before the switch ever has to assert pause.
Role in the fabricSafety backstop only, for when buffers still fill.Primary control for most of the fabric's life.
ScopeLink-level, one hop at a time.End-to-end, sender all the way to receiver.

ECN marking in OcNOS-DC

WRED Curve

K-min, K-max, P-max

Per-priority-queue WRED ECN marking with configurable K-min and K-max thresholds and a P-max marking probability.

ECN in feature matrix
Per-Priority

Independent of PFC

ECN marking is configured independently from PFC pause thresholds.

PFC deadlock detection
VXLAN Aware

ECN over VXLAN

On Trident 3 (X4, X5, X7) and Trident 4 platforms, ECN bits are preserved through VXLAN encap and decap so DCQCN works end-to-end across an EVPN-VXLAN overlay, not just on the underlay.

AI Fabric solution
Scope

License tiers

PFC and ECN are in the OcNOS-DC IPBASE and PLUS SKUs. Dynamic ECN, PFC deadlock detection and recovery, and PFC and ECN over VXLAN are in the PLUS SKUs.

OcNOS-DC

Why this matters more than it sounds

ECN threshold misalignment has two ends that look opposite, but both leave the fabric mis-tuned.

  • ECN configured but never marks (K-min too high): PFC carries the whole congestion-control burden.
  • ECN marks too aggressively (K-min too low): senders cut rate before there is any real congestion.

DCQCN, answered

What is DCQCN?
DCQCN (Data Center Quantized Congestion Notification) is the congestion-control algorithm for RoCEv2 RDMA traffic on Ethernet AI and storage fabrics. It throttles senders before queues overflow, so GPU and storage flows spend less time in PFC pause and avoid stalls.
How does DCQCN work?
DCQCN works in a closed loop: a switch marks packets with ECN when a queue crosses its threshold, the receiver echoes a Congestion Notification Packet (CNP) back to the sender, and the sender cuts its send rate, then recovers gradually as congestion clears.
How do DCQCN and PFC work together?
DCQCN is the primary control and PFC is the backstop. ECN marking and CNP feedback slow senders early so flows rarely hit pause. PFC only triggers if buffers still fill, preventing drops. Tuning aims to keep the fabric in ECN control and out of sustained PFC pause.
What are the ECN K-min and K-max thresholds in DCQCN?
K-min and K-max are the queue-depth thresholds that drive ECN marking. Below K-min no packets are marked; above K-max all are marked. Between them the marking probability rises toward a set maximum, so congestion signaling scales with queue buildup instead of switching on abruptly.
Does DCQCN require RoCEv2?
Yes, DCQCN is designed for RoCEv2 RDMA traffic and relies on RoCEv2 endpoints generating and acting on CNP feedback. It pairs ECN in the network with rate control at RoCEv2 senders, so it is the standard congestion-control choice for RoCEv2 fabrics.

Go deeper. Take it with you.

The product datasheet plus short, technical downloads that go further than this page.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Lossless RoCEv2 fabric on Tomahawk 5 spines: SKU tiers, qualified platforms, and deployment architecture.

Get the brief
Solution Brief

EVPN-VXLAN Data Center Fabric

Leaf-spine data center fabric on EVPN-VXLAN: a VRF and VNIs per tenant, EVPN multihoming, BGP unnumbered and route-target filtering.

Get the brief

Tuning DCQCN for a real workload? Bring us your traces

Send us your ECN counters and queue traces, and an IP Infusion engineer will align your K-min, K-max, and PFC headroom, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.

Request a technical demo Open the AI fabric design suite

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs AI Fabric Topologies Rail-optimized two-tier and three-tier fabrics, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA? AI networking switches
The technology inside RoCEv2 lossless Rail-optimized network DLB GLB (7.1) Ultra Ethernet Tomahawk 6 at 1.6T DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI