ECN · CNP · RoCEv2

DCQCN: Quantized Congestion Control for RDMA

DCQCN is how a RoCEv2 fabric stays lossless without living in PFC pause. The switch marks ECN early, the receiver echoes a CNP, and the sender quantizes its rate, so senders slow down before queues overflow. OcNOS-DC ships defaults pre-tuned for xCCL collective workloads (NCCL, RCCL, oneCCL) and exposes every threshold for fabrics that need to deviate.

Bring us your traces Open the AI fabric design suite
Closed loopECN mark, CNP, rate cut
Before PFCECN marks ahead of pause
xCCLpre-tuned OcNOS-DC defaults
DC-PLUSOcNOS-DC license tier
The mechanism

The DCQCN closed loop

Sender NIC, congested switch, receiver NIC. The switch's WRED ECN marker fires before the queue hits the PFC pause threshold. The receiver generates a Congestion Notification Packet (CNP); the sender's reaction point reduces rate, then ramps back. Lossless, no PFC pressure, fast convergence.

DCQCN closed-loop congestion control Three boxes left to right: Sender NIC (reaction point), Congested switch with WRED ECN marker, Receiver NIC (notification point). Forward arrow shows a packet flowing left to right and being marked by the switch. Reverse arrow shows the CNP traveling right to left back to the sender. Labels indicate ECN K-min, K-max thresholds and CNP feedback path. data + ECN-capable ECN = CE (marked) CNP: Congestion Notification Packet Sender NICReaction Pointα-update · cut · ramp OcNOS-DC SwitchWRED ECN markerK-min · K-max · P-max Receiver NICNotification PointCNP every 50 µs DCQCN · WRED MARKING · CNP FEEDBACK · QUANTIZED RATE LIMIT · LOSSLESS RoCEv2
What it does

The job DCQCN does in an AI fabric

RoCEv2 has two ways to handle congestion: PFC pause, which is back-pressure that propagates hop by hop, and DCQCN, an end-to-end rate-control loop. PFC alone works, but it pushes congestion upstream and risks pause storms and head-of-line blocking. DCQCN works ahead of PFC, marking packets with ECN before the queue reaches the pause threshold, so the sender slows down before the switch ever has to assert pause.

Done well, you spend most of your fabric life on DCQCN feedback alone, with PFC as a safety backstop. Done badly, ECN thresholds are misaligned with PFC headroom and you get pause storms even with DCQCN configured. Threshold tuning is the whole game, and OcNOS-DC publishes defaults tuned for xCCL collective traffic, while exposing every knob for fabrics that have specific traffic patterns.

Reaction Point

Sender NIC quantizes its rate

Receives CNPs and runs the DCQCN α-update, multiplicative-decrease, additive-increase loop to quantize its sending rate.

RoCEv2 lossless fabric
Congestion Point

Switch marks ECN early

Marks ECN-capable packets to CE using a WRED curve when queue depth crosses K-min, with marking probability rising linearly to P-max at K-max.

PFC deadlock detection
Notification Point

Receiver echoes a CNP

Generates a CNP back to the sender on each marked flow, rate-limited to roughly one per 50 µs per flow.

Two ways to handle congestion

PFC pause and DCQCN, side by side

Both keep RoCEv2 lossless, but they act at different points. The tuning goal is to spend the fabric's life on DCQCN feedback and hold PFC in reserve as a backstop.

Axis PFC pausebackstop DCQCNprimary control
MechanismHop-by-hop back-pressure that propagates upstream.End-to-end rate-control loop from sender to receiver.
Trigger pointFires when the queue reaches the pause threshold.Marks ECN before the queue reaches the pause threshold.
Failure modeRisks pause storms and head-of-line blocking.Sender slows before the switch ever has to assert pause.
Role in the fabricSafety backstop only, for when buffers still fill.Primary control for most of the fabric's life.
ScopeLink-level, one hop at a time.End-to-end, sender all the way to receiver.
In OcNOS-DC

The OcNOS DCQCN implementation

xCCL-class defaults out of the box, with every threshold exposed as a YANG-modeled path, validated against PFC headroom before it is applied, and streamed back as telemetry during bring-up.

WRED Curve

K-min, K-max, P-max

Per-priority-queue WRED ECN marking with configurable K-min and K-max thresholds and a P-max marking probability. xCCL-class defaults out of the box; exposed as YANG paths for tuning.

ECN in feature matrix
Per-Priority

Independent of PFC

ECN marking is configured independently from PFC pause thresholds. Misalignment is the most common DCQCN configuration error. OcNOS validates the relationship between K-max, headroom, and the pause asserts before applying.

PFC deadlock detection
VXLAN Aware

ECN over VXLAN

ECN bits are preserved through VXLAN encap and decap so DCQCN works end-to-end across an EVPN-VXLAN overlay, not just on the underlay.

AI Fabric solution
Telemetry

Per-queue ECN counters

gNMI-streamed counters for ECN-marked packets per egress queue, queue depth distribution, and CNP-trigger rates. Closed-loop tuning during cluster bring-up.

Diagnostics

Verify before you commit

CLI sanity-check that K-min, K-max, and PFC headroom are mathematically consistent with the buffer space allocated to the lossless priority. Fail fast on a misconfig.

Validated hardware
Scope

DC-PLUS license tier

Part of the OcNOS-DC PLUS SKU. Same image, same support; no per-feature add-on required to activate the lossless RDMA stack.

OcNOS-DC
The failure mode

Why this matters more than it sounds

Most "RoCEv2 isn't behaving" support cases land on DCQCN threshold misalignment. The two ends of the mistake look opposite, but both leave the fabric mis-tuned.

  • ECN configured but never marks (K-min too high): PFC carries the whole congestion-control burden.
  • ECN marks too aggressively (K-min too low): senders cut rate before there is any real congestion.
  • OcNOS-DC ships defaults that work on most TH4 and TH5 fabrics; where a fabric needs to deviate, every parameter is YANG-modeled and verifiable.

To size the leaf-spine tiers those defaults run on, try the AI Fabric Sizing Tool, or review the AI fabric topologies the defaults are tuned for.

The IP Infusion view

DCQCN carries the load, PFC stays the backstop

A healthy RoCEv2 fabric runs on ECN feedback and barely touches PFC. OcNOS-DC ships that balance as the default and keeps every threshold in reach when a workload needs a different one.

DCQCN carries the load

ECN marking and CNP feedback slow senders early, so a healthy fabric spends most of its life on DCQCN and rarely reaches PFC pause.

DLB adaptive routing

Tuned for xCCL out of the box

OcNOS-DC ships defaults pre-tuned for xCCL collective traffic, so most fabrics run well on TH4 and TH5 without touching a threshold.

GLB: fabric-wide (7.1)

Every knob is modeled

K-min, K-max, and P-max are YANG-modeled and verifiable, so fabrics with specific traffic patterns can deviate safely.

Ultra Ethernet (UEC)
FAQ

DCQCN, answered

What is DCQCN?
DCQCN (Data Center Quantized Congestion Notification) is the congestion-control algorithm that keeps RoCEv2 RDMA traffic lossless on Ethernet AI and storage fabrics. It throttles senders before queues overflow, so GPU and storage flows avoid packet drops and stalls.
How does DCQCN work?
DCQCN works in a closed loop: a switch marks packets with ECN when a queue crosses its threshold, the receiver echoes a Congestion Notification Packet (CNP) back to the sender, and the sender cuts its send rate, then recovers gradually as congestion clears.
How do DCQCN and PFC work together?
DCQCN is the primary control and PFC is the backstop. ECN marking and CNP feedback slow senders early so flows rarely hit pause. PFC only triggers if buffers still fill, preventing drops. Tuning aims to keep the fabric in ECN control and out of sustained PFC pause.
What are the ECN K-min and K-max thresholds in DCQCN?
K-min and K-max are the queue-depth thresholds that drive ECN marking. Below K-min no packets are marked; above K-max all are marked. Between them the marking probability rises toward a set maximum, so congestion signaling scales with queue buildup instead of switching on abruptly.
Does DCQCN require RoCEv2?
Yes, DCQCN is designed for RoCEv2 RDMA traffic and relies on RoCEv2 endpoints generating and acting on CNP feedback. It pairs ECN in the network with rate control at RoCEv2 senders, so it is the standard congestion-control choice for RoCEv2 fabrics.
Datasheet and Solution Briefs

Go deeper. Take it with you.

The product datasheet plus short, technical downloads that go further than this page.

Datasheet

OcNOS-DC Datasheet

Full OcNOS-DC specification: the EVPN-VXLAN and Ethernet for AI feature set, software SKUs, supported hardware platforms, and the solution ordering guide.

Get the datasheet
Solution Brief

OcNOS 800G Lossless AI Fabric

Non-blocking RoCEv2 fabric on Broadcom Tomahawk 4/5 spines: SKU tiers, validated platforms, and deployment architecture.

Get the brief
Solution Brief

EVPN-VXLAN Data Center Fabric

Carrier-grade leaf-spine data center fabric: symmetric IRB, Type-2/Type-5 routes, and distributed anycast gateway.

Get the brief

Tuning DCQCN for a real workload? Bring us your traces

Send us your ECN counters and queue traces, and an IP Infusion engineer will align your K-min, K-max, and PFC headroom, or start with a first-pass leaf-spine layout in the AI Fabric Design Suite.

Request a technical demo Open the AI fabric design suite
AI Fabric

Design the whole AI fabric with OcNOS

From the business case to the port-count maths, pick up wherever you are in the build.

Solution Open AI Fabric The complete 800G AI fabric: open switches, OcNOS-DC, and support under one contract. Reference designs AI Fabric Topologies Rail-optimized, scheduled 3-stage Clos, and coherent DCI, sized in real port counts. Size & build AI Fabric Design Suite Size a GPU fabric: leaf, spine, and super-spine counts with a component and power summary.
New to AI fabrics? Start here What is an AI fabric? What is a GPU fabric? What is lossless Ethernet? What is RDMA?
The technology inside RoCEv2 lossless Rail-optimized network DLB adaptive routing GLB (7.1) Ultra Ethernet DCQCN PFC deadlock InfiniBand vs Ethernet RoCE vs InfiniBand AI fabric architecture Coherent DCI