Overlay ECMP with EVPN-VXLAN in OcNOS, Part 2: Multi-VM Subnet Scaling

Part 2 scales overlay ECMP in an EVPN-VXLAN fabric across multiple VMs that share one IP subnet on a hypervisor. OcNOS advertises each host with EVPN type-5 routes, so equal-cost paths to every VTEP install as ECMP and flows spread evenly across the leaf-spine fabric. See EVPN on OcNOS.

In Part 1, we distributed traffic across multiple hypervisors each hosting a single VM with the same service IP. Part 2 addresses the more complex scenario: multiple VMs sharing the same IP subnet hosted on the same hypervisor, and how OcNOS handles ECMP routing in this case.

Topology: Multiple VMs per Hypervisor

Key Difference: Subnet Routes vs. Host Routes

When multiple VMs on the same hypervisor share the same IP subnet, we cannot use EVPN type-2 (MAC+IP) routes for load balancing, because those are per-host. Instead, each VM establishes an EBGP session to the leaf switch’s VRF and advertises its loopback IP (the service IP-X) as a host route. The leaf then advertises these into the fabric as EVPN type-5 (IP-prefix) routes that every VTEP learns.

In this design the VM connected interfaces and the leaf gateway all sit in the same tenant subnet (10.10.10.0/24 in the blog diagrams). The leaf VRF uses an anycast gateway IP (10.10.10.1) as the next-hop for the default route it returns to each VM, plus a per-leaf secondary IP on the IRB (10.10.10.201 on Leaf-1, 10.10.10.202 on Leaf-2) that anchors the logical BGP sessions. The VRF is mapped to L3 VNI 100.

EBGP: VM to Leaf VRF Configuration

! OcNOS: Leaf VRF and EBGP toward the hypervisor VMs.
! IP addresses, VRF name, and L3 VNI follow the Part 2 topology
! diagram (Figure 5); ASN values are illustrative.
!
! Step 1: Create the tenant VRF and map it to the L3 VNI.
ip vrf VRF1
 l3vni 100
!
! Step 2: Enable the VXLAN network virtualization overlay.
nvo vxlan enable
!
! Step 3: EBGP from the leaf VRF toward the VM subnet.
!         Configure Method A or Method B, not both.
router bgp 65003
 !
 ! Method A: explicit neighbors, one per VM.
 address-family ipv4 vrf VRF1
  ! VM Eth1 addresses on 10.10.10.0/24 (VM3=.3, VM4=.4, ... VMN=.N)
  neighbor 10.10.10.3 remote-as 65100
  neighbor 10.10.10.3 activate
  neighbor 10.10.10.4 remote-as 65100
  neighbor 10.10.10.4 activate
 exit-address-family
 !
 ! Method B: dynamic listen-range peering (Figure 5), so the leaf
 !           config stays constant as VMs come and go.
 address-family ipv4 vrf VRF1
  neighbor VM-group peer-group range 10.10.10.0/24
  neighbor VM-group update-source 10.10.10.201   ! Leaf-2 uses 10.10.10.202
 exit-address-family
 !
 ! Step 4: Advertise the VRF IPv4 routes into EVPN as type-5 prefixes.
 address-family l2vpn evpn
  advertise ipv4 unicast
 exit-address-family

The dynamic peer-group lines are taken from the Part 2 topology diagram (Figure 5); the surrounding commands are the standard OcNOS EVPN-VXLAN scaffolding that maps the tenant VRF to L3 VNI 100 and advertises its prefixes into EVPN as type-5 routes.

Scaling the Sessions: RR VMs and ADD-PATH

Adding a new service VM under Method A means editing the leaf VRF BGP configuration, which is awkward for orchestration. The blog offers two alternatives. First, introduce Route-Reflector VMs (RR1 and RR2, at 10.10.10.100 and 10.10.10.101) inside the hypervisor: the service VMs peer to the RR VMs over IBGP, and only the RR VMs peer to the leaf VRF over EBGP, so the leaf configuration no longer changes as service VMs come and go. Because several VMs advertise the same IP-X, BGP Additional Paths (ADD-PATH) is required on the EBGP session so the leaf VRF can install one ECMP path per VM with its own next-hop. Second, use dynamic (listen-range) peering on the leaf, as shown in Method B above, which keeps the leaf configuration constant at the cost of the leaf VRF managing more sessions directly.

VM-Side EBGP Configuration

! Linux VM (FRR/BIRD): EBGP to the leaf VRF. VM side, NOT OcNOS.
! Each VM peers from its Eth1 address on 10.10.10.0/24 and advertises
! its service loopback IP-X; the leaf returns a default route whose
! next-hop is the anycast gateway 10.10.10.1. ASN values are illustrative.
!
router bgp 65100
 bgp router-id 10.10.10.3               ! VM3 Eth1 address (Figure 5)
 neighbor 10.10.10.201 remote-as 65003  ! leaf IRB secondary (Figure 5)
 !
 address-family ipv4 unicast
  network IP-X/32                        ! service loopback (symbolic in the diagrams)
  neighbor 10.10.10.201 activate
 exit-address-family

Verifying Multi-VM ECMP

! OcNOS: Verify ECMP across the fabric for the IP-X subnet.
!
! Check the EVPN type-5 (IP-prefix) route for the IP-X subnet.
! Expect one next-hop per advertising VTEP, e.g. via Leaf-1 (10.10.10.201)
! and Leaf-2 (10.10.10.202).
show bgp l2vpn evpn
!
! Inspect the tenant VRF route table for the ECMP paths to IP-X.
show ip route vrf VRF1
!
! Check the BGP peer and route state in the tenant VRF.
show bgp vrf VRF1

All of the features discussed in this article are supported in OcNOS 6.3.

• Overlay ECMP Part 1: Fundamentals
• EVPN-ELAN Configuration Guide
• EVPN-L3VPN over ISIS-SR
• OcNOS-DC Product Page
• Contact IP Infusion

IP Infusion Engineering Team