EBGP-based Data Center with OcNOS

CHAPTER 1

Data Center Interconnect Overview

Large-Scale Data Center Requirements

Large-Scale Data Center Topologies

Large-Scale Data Center Routing

EBGP-Routed Clos Topology-Based Data Center

EBGP Data Center Design using OcNOS

CHAPTER 2

Configuration 8

ToR (leaf node)

Tier-2 (spine node)

Tier-3 (core node)

Tier-2 (border router)

Tier-3 (WAN router)

Other Configurations

Validation

Conclusion

References

Appendix A: Configuring the Data Center through NetConf

Appendix B: NetConf User Guide

Glossary

BGP Border Gateway Routing Protocol
EBGP External BGP
STP Spanning Tree Protocol
TRILL Transparent Interconnection of Lots of Links
SPB Shortest Path Bridging
ECMP Equal Cost Multipath

Chapter 1

Data Center Overview

Network Automation provides IT administrators and network operators significant benefits. This solution guide

describes an approach to build data centers using Layer3 BGP routing protocol.

It also summarizes on some design philosophies for data center and why E-BGP is better suited.

  • Large-scaledata center requirements
  • Large-scaledata center topologies
  • Large-scaledata center routing
  • EBGP-routedlarge-scale Clos topology-based data center

Large-Scale Data Center Requirements

The design of large-scale data centers is driven by operational simplicity and network stability. Operational simplicity and network stability ensures easier manageability and therefore reduced operational expenses. From the network design aspect, the requirements are:

  • Abilityto accommodate the variable-application bandwidth and strict latency
  • Abilityto handle the increased east-west (server-to-server) traffic within the data center due to massive

data replication between clusters and virtual machine migrations.

  • Traffic-Engineeringwith application load The network infrastructure should itself perform controlled per-hop traffic engineering.
  • MinimizeCAPEX and incorporate vendor diversity by using a simple, interoperable routing protocol with a minimal set of
  • Adesign to minimize OPEX by keeping the failure domain at the lowest level in the network

Large-Scale Data Center Topologies

A traditional tree-based (upside down) topology with a three-layer hierarchy of core, aggregation and access

layer can be used in a data center design. This approach is suited if the majority of the traffic is entering

and leaving (north-south) the data center. An increase in bandwidth requirements then can be addressed by upgrading the device line cards or port density. However with the current trend of increasing server-to-server (east-west) traffic, scaling these networks horizontally is expensive or impossible at times.

A Clos network (leaf and spine) is a horizontally scalable topology where every leaf node is connected to every other spine. The topology can be extended to different stages for scaling. Clos networks are fully non- blocking and load balancing is inherent in the topology itself as all available paths are ECMP. Clos networks are ideal for the current requirements of a large-scale data center.

Large-Scale Data Center Routing

Layer 2-only routing was used in a traditional tree-based data center topology. Traditional layer-2 protocols such as STP do not give bi-sectional bandwidth, whereas recent developments such as TRILL, SPB have selected vendor support.

However, a hybrid of layer 2 /layer 3 can be used to limit the size of failure domain and scale up the data center. Layer 3 routing can be used in tier 1 (core) and layer 2 in tier 3 (access). Tier 2 can be based on either layer 2 or layer 3. A hybrid model has the advantage of seamless Virtual Machine mobility and requires less IP subnets for the data center. Although this design can scale-up, it is difficult and complex to manage the different protocols.

A layer 3 only design simplifies the network and improves network stability and scalability, as well as

localizing the failure domain (confined to the L2 broadcast domain). Seamless virtual mobility can be achieved in a L3 only based data center by using L2 overlay networks. From experiment and analysis, External BGP (EBGP) is considered ideal compared to IGPs due to the following [See Reference]:

  • Lesscomplex protocol, simple state machine
  • Informationflooding overhead is less, no frequent updates unlike IGPs
  • Networkfailure recovery is very Although BGP convergence is slower than IGP, in a Clos topology with ECMP links, the failure is masked as soon as an alternate path is found.
  • Failuredomain is minimized in a Clos topology with Some of the failures are local/hidden/not propagated if the failed link was not selected/advertised as the best path among the ECMP paths by the BGP speaker. The failures, where all devices have to withdraw some prefixes or update the ECMP groups in the FIB, are very limited and in those failures the failed link/node does not impact the re-convergence process.
  • Administratorcan define the application traffic BGP provides services like prefix distribution, prefix filtering, traffic engineering, traffic tagging, and multi-vendor stability better than other IGPs.
  • Easierto

EBGP-Routed CLOS Topology-Based Data Center

EBGP-routed CLOS topology is considered the best choice for laying the IP fabric in a data center because of the horizontal scalability feature of Clos topology and the ease of use and services provided by EBGP especially prefix-filtering, prefix distribution, and traffic engineering which are required extensively in a data center.

Configuration Guidelines

Configuration guidelines for laying IP fabric using EBGP efficiently are as follows:

  • Runall EBGP sessions over single-hop point-to-point
  • Useprivate Autonomous System Numbers (ASNs) (64512-65534) to avoid ASN
  • Giveall tier 1 (core) devices a single
  • Giveall tier 2 devices in the same cluster the same unique A cluster or pod is a group of tier 2 (spine switches) + tier 3 switches (ToR/leaf) + servers.
  • Giveevery tier 3 (ToR) device in a cluster a unique
  • Reusetier-3 ASNs across Configure tier-3 devices with the BGP “allowas-in” feature to allow route learning of prefixes from the same ASNs in other clusters.
  • Announceserver subnets on tier-3 devices via BGP without using route summarization on tier-2 and tier-1
  • Useedge clusters (pods) for external connectivity. Each edge cluster consists of border routers (tier-2) and WAN routers (tier-3). Give each WAN router a unique public ASN to connect the data center to the external world.
  • Forborder routers, remove private ASNs before sending the information to WAN routers by configuring border routers with the “remove-private-AS” BGP
  • Torelax the BGP ECMP criteria for AS paths, configure BGP “as-path multipath-relax” on all routers/ This way, an equal cost path with a different AS PATH, but the same AS PATH length is also considered an equal cost path (ECMP).
  • Forfaster failure detection, configure the BGP session with
  • Toavoid recurring BPG update/selection for a single failure through all peers or BGP update message dispersion on a particular speaker, use BGP update The BGP update group feature processes an update once and sends it to a group of neighbors that share a common outbound policy. The BGP RIB is scanned every time for each peer to apply the outbound filter.
  • Toavoid micro routing loops, configure tier-2 and tier-1 with static discard or null routes rather than a default Routing loops can happen when a tier-2 device has lost all its learned prefixes, but has a default

route to a tier-1 device and that tier-1 device still has a route back to the tier-2 device.

EBGP Data Center Design using OcNOS

Figure-1 shows a minimal representation that encompasses all the elements in a layer 3 data center. The number of ECMPs in the data center is equal to the number of cores (tier-1 switches).

Figure 1 . IP fabric using EBGP

Core Tier 1

Figure 2 shows the Autonomous System Number (ASN) allocation scheme used in the data center. The WAN routers are assigned a public ASN, which connects the data center to external world. The tier-3 ASNs per ToR are reused across the clusters.

Figure 2: ASN allocation in an EBGP-based data center

ASN 65534

Chapter 2

Configuration

To R (Leaf node)

Command Purpose
Step 1 (config)#interface xe1 Enter interface mode.
Step 2 (config-if)#ip address 32.1.0.3/24 Configure ip address on the Interface
Step 3 (config-if)#exit Exit interface mode.
Step 4 (config)#interface xe2 Enter interface mode.
Step 5 (config-if)#exit Exit interface mode.
Step 6 (config)#router bgp 65500 Configure the EBGP routing process with private ASN
Step 7 (config-router)#max-paths ebgp 8 Exit interface mode.
Step 8 (config-router)#neighbor 32.1.0.2 remote-as 64601 Configure maximum EBGP ECMP that can be installed in BGP.
Step 9 (config-router)#neighbor 32.1.0.2 fall-over bfd Configure the EBGP neighbor over the connected interface using the neighbor IP and remote private ASN
Step 10 (config-router)#neighbor 32.1.0.2 allowas-in Configure BFD for the BGP session for faster failure detection.
Step 11 (config-router)#neighbor 32.2.0.2 remote-as 64601 Configure “allowas-in” for the neighbor to accept routes with same ASN learned over this neighbor
Step 12 (config-router)#neighbor 32.2.0.2 fall-over bfd Configure the EBGP neighbor over the connected interface using the neighbor IP and remote private ASN
Step 13 (config-router)#neighbor 32.2.0.2 allowas-in Configure BFD for the BGP session for faster failure detection.
Step 14 (config-router)#exit Configure “allowas-in” for the neighbor to accept routes with same ASN learned over this neighbor
Step 15 (config-router)#exit Exit Router mode.

Tier-2 (Spine node)

Command Purpose
Configure the interfaces
Step 1 (config)#interface xe1 Enter interface mode.
Step 2 (config-if)#ip address 32.1.0.2/24 Configure ip address on the interface
Step 3 (config-if)#exit Exit interface mode.
Step 4 (config)#interface xe46 Enter interface mode.
Step 5 (config-if)#ip address 32.3.0.2/24 Configure an IP address on the interface
Step 6 (config)#interface xe47 Enter interface mode.
Step 7 (config-if)#ip address 32.4.0.2/24 Configure an IP address on the interface
Step 8 (config)#interface xe48 Enter interface mode.
Step 9 (config-if)#ip address 21.1.0.2/24 Configure an IP address on the interface
Step 10 (config)#interface xe46 Enter interface mode.
Step 11 (config-if)#ip address 21.2.0.2/24 Configure an IP address on the interface
Step 12 (config)#router bgp 64601 Configure the eBGP routing process with private ASN

Tier-2 (Spine node) cont.

Command Purpose
Configure BGP on the router
Step 13 (config-router)#bgp bestpath as-path multipath-relax Configure “as-path multipath-relax” to relax the AS-PATH exact match (if AS-PATH length are same) criteria for BGP ECMP
Step 14 (config-router)#max-paths ebgp 8 Configure maximum EBGP ECMP that can be installed in BGP.
Step 15 (config-router)#neighbor 32.1.0.3 remote-as 65000 Configure the EBGP neighbor over the connected interface using the neighbor IP and remote ASN
Step 16 (config-router)#neighbor 32.1.0.3 fall-over bfd Configure BFD for the BGP session for faster failure detection.
Step 17 (config-router)#neighbor 32.3.0.3 remote-as 65001 Configure the EBGP neighbor over the connected interface using the neighbor IP and remote ASN
Step 18 (config-router)#neighbor 32.3.0.3 fall-over bfd Configure BFD for the BGP session for faster failure detection.
Step 19 (config-router)#neighbor 32.4.0.3 remote-as 65002 Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 20 (config-router)#neighbor 32.1.0.3 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 21 (config-router)#neighbor 21.1.0.1 remote-as 65534 Configure the EBGP neighbor over the connected interface using the neighbor IP and remote ASN
Step 22 (config-router)#neighbor 21.1.0.1 fall-over bfd Configure BFD for the BGP session for faster failure detection.
Step 23 (config-router)#neighbor 21.2.0.1

remote-as 65534

Configure the EBGP neighbor over the connected interface using the neighbor IP and remote ASN
Step 24 (config-router)#neighbor 21.2.0.1 fall-over bfd Configure BFD for the BGP session for faster failure detection.
Step 25 (config-router)#exit Exit Router mode.

Tier-3 (Core node)

Command Purpose
Configure the interfaces
Step 1 (config)#interface xe1 Enter interface mode.
Step 2 (config-if)#ip address 21.1.0.1/24 Configure ip address on the interface
Step 3 (config-if)#exit Exit interface mode.
Step 4 (config)#interface xe46 Enter interface mode.
Step 5 (config-if)#ip address 21.5.0.1/24 Configure an IP address on the interface
Step 6 (config)#interface xe49 Enter interface mode.
Step 7 (config-if)#ip address 41.1.0.1/24 Configure an IP address on the interface
Step 8 (config)#interface xe50 Enter interface mode.
Step 9 (config-if)#ip address 41.5.0.1/24 Configure an IP address on the interface
Configure BGP on the router
Step 10 (config)#router bgp 65534 Configure the eBGP routing process with private ASN
Step 11 (config-router)#bgp bestpath as-path multipath-relax Configure “as-path multipath-relax” to relax the AS-PATH exact match (if AS-PATH length are same) criteria for BGP ECMP
Step 12 (config-router)#max-paths ebgp 8 Configure maximum EBGP ECMP that can be installed

in BGP.

Step 13 (config-router)#neighbor 21.1.0.2

remote-as 64601

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 14 (config-router)#neighbor 21.1.0.2 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 15 (config-router)#neighbor 21.5.0.3

remote-as 64602

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 16 (config-router)#neighbor 21.5.0.3 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 17 (config-router)#neighbor 41.1.0.3

remote-as 64603

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 18 (config-router)#neighbor 41.1.0.3 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 19 (config-router)#neighbor 41.5.0.3

remote-as 64603

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 20 (config-router)#neighbor 41.5.0.3 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 21 (config-router)#exit Exit BGP mode

Tier 2 (Border router)

Command Purpose
Configure the interfaces
Step 1 (config)#interface xe1 Enter interface mode.
Step 2 (config-if)#ip address 41.1.0.2/24 Configure an IP address on the interface
Step 3 (config-if)#exit Exit interface mode.
Step 4 (config)#interface xe46 Enter interface mode.
Step 5 (config-if)#ip address 41.2.0.2/24 Configure an IP address on the interface
Step 6 (config)#interface xe47 Enter interface mode.
Step 7 (config-if)#ip address 41.3.0.2/24 Configure an IP address on the interface
Step 8 (config)#interface xe48 Enter interface mode.
Step 9 (config-if)#ip address 41.4.0.2/24 Configure an IP address on the interface
Step 10 (config)#interface xe49 Enter interface mode.
Step 11 (config-if)#ip address 51.1.0.2/24 Configure an IP address on the interface
Step 12 (config)#interface xe50 Enter interface mode.
Step 13 (config-if)#ip address 51.3.0.2/24 Configure an IP address on the interface
Step 14 (config)#router bgp 64603 Configure the eBGP routing process with private ASN
Step 15 (config-router)#bgp bestpath as-path multipath-relax Configure “as-path multipath-relax” to relax the AS-PATH exact match (if AS-PATH length are same) criteria for BGP ECMP
Step 16 (config-router)#max-paths ebgp 8 Configure maximum EBGP ECMP that can be installed in BGP.
Step 17 (config-router)#neighbor 41.1.0.1

remote-as 65534

Configure the EBGP neighbor over the connected interface

using the neighbor IP and remote ASN

Step 18 (config-router)#neighbor 41.1.0.1 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 19 (config-router)#neighbor 41.2.0.1

remote-as 65534

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 20 (config-router)#neighbor 41.2.0.1 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 21 (config-router)#neighbor 41.3.0.1

remote-as 65534

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 22 (config-router)#neighbor 41.3.0.1 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 23 (config-router)#neighbor 41.4.0.1

remote-as 65534

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 24 (config-router)#neighbor 41.4.0.1 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 25 (config-router)#neighbor 51.1.0.3

remote-as 100

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Configure BGP on the router
Step 26 (config-router)#neighbor 51.1.0.3 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 27 (config-router)#neighbor 51.1.0.3 remove-private-AS Configure “remove –private-AS” to remove the private ASNs

for the routes advertised to this neighbor.

Step 28 (config-router)#neighbor 51.3.0.3

remote-as 101

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Tier 2 (Border router) Cont.

Command Purpose
Configure BGP on the router
Step 29 (config-router)#neighbor 51.3.0.3 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 30 (config-router)#neighbor 51.3.0.3 remove-private-AS Configure “remove –private-AS” to remove the private ASNs

for the routes advertised to this neighbor.

Step 31 (config-router)#exit Exit BGP mode

Tier-3 (WAN router)

This is a partial list and does not contain the Internet configuration.

Command Purpose
Configure the interfaces
Step 1 (config)#interface xe1 Enter interface mode.
Step 2 (config-if)#ip address 51.1.0.3/24 Configure an IP address on the interface
Step 3 (config-if)#exit Exit interface mode.
Step 4 (config)#interface xe46 Enter interface mode.
Step 5 (config-if)#ip address 51.2.0.3/24 Configure an IP address on the interface
Configure BGP on the router
Step 6 (config)#router bgp 100 Configure the eBGP routing process with public ASN
Step 7 (config-router)#max-paths ebgp 8 Configure maximum EBGP ECMP that can be installed in

BGP.

Step 8 (config-router)#neighbor 51.1.0.2

remote-as 64603

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 9 (config-router)#neighbor 51.1.0.2 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 10 (config-router)#neighbor 51.3.0.2

remote-as 64603

Configure the EBGP neighbor over the connected

interface using the neighbor IP and remote ASN

Step 11 (config-router)#neighbor 51.3.0.2 fall-over bfd Configure BFD for the BGP session for faster failure

detection.

Step 12 (config-router)#exit Exit BGP mode

Other Configurations

You must repeat similar configurations for all ToR, spine, core, border, and WAN devices as well.

Validation

Use the show ip bgp command to validate the output at each node.

Consider the following case: for application load balancing and high availability/reliability, two similar application servers can be placed at two clusters. For users accessing the application server through the Internet, the access to the server is load balanced and failure of one of the application servers does not impact the accessibility. The following is the output at various nodes for a subnet, such as:

  • 70.70.70.1(application server) at ToR1 in cluster 1 and cluster 2
  • 80.80.80.1at ToR 1 cluster 1
  • 90.90.1at ToR 1 cluster 2

Conclusion

OcNOS with EBGP routing is a highly scalable, simple and flexible way of laying IP fabric in a data center.

The data center can be easily scaled for:

  • Highercomputing needs by adding more
  • Higherperformance and redundancy by adding more cores
  • Higheruplink speeds by adding more external/edge

References

Use of BGP for routing in large scale data centers: https://tools.ietf.org/html/draft-ietf-rtgwg-bgp-routing-large-dc-09

Appendix A: Configuring the Data

Center through NetConf

This appendix shows how to configure a data center through NetConf. This section contains XML payloads which can be used by any NetConf client to send configurations to OcNOS devices.

The configurations below in XML are communicated via RPC messages from the NetConf client to the

NetConf server.

This document uses the yang-cli which is part of open source OpenYumaMaster which and is preinstalled on a ZebM-enabled OcNOS ONIE package as a NetConf client.

Pre-requisite

Refer to Yangcli Operations section of the NetConf User Guide and perform these operations first:

  1. EstablishConnection
  2. LoadModules

Once the above operations are complete, edit-config operation can be performed with XML payloads using

the command:

yangcli [email protected]> edit-config [email protected]/<path-to-xml-file>/<xml-file>.xml

Save the below xml payloads as an .xml file in any desired location. Make sure that each XML payload starts

with <vr xmlns=”<namespace>”> and ends with </vr>.

TOR (leaf node) TOR.xml

This table shows the XML to configure an IP address:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ ZebOS”> namespace of the ZebOS module
<vrId>0</vrId> Vrid
<interface>

<ifName>xe49</ifName>

<ipAddr>50.52.1.1/24</ipAddr>

</interface>

<interface>

<ifName>ge2</ifName>

<ipAddr>50.54.1.1/24</ipAddr>

</interface>

Configure the interfaces which are connected to the

SPINE routers.

This payload configures the IP address of the

interfaces xe49 and ge2.

</vr> Close namespace for the ZebOS module

This is an example of the XML payload to be sourced in the yang CLI to configure an interface IP addresses:

<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”>

<vrId>0</vrId>

<interface>

<ifName>xe49</ifName>

<ipAddr>50.52.1.1/24</ipAddr>

</interface>

<interface>

<ifName>ge2</ifName>

<ipAddr>50.54.1.1/24</ipAddr>

</interface>

</vr>

This table shows the XML to configure BGP attributes:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ ZebOS”> Namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<bgp>

<bgpAs>64602</bgpAs>

Open BGP tag.

Configure the BGP AS number.

<multipathRelax>1</multipathRelax>

<vrfName>default</vrfName>

Configure “as-path multipath-relax” to

relax the AS-PATH criteria for BGP ECMP

<multipath>

<bgpType>ebgp</bgpType>

<multipathType>

<multipathsNum>8</ multipathsNum>

</multipathType>

</multipath>

Configure maximum EBGP ECMP that can be

installed in BGP.

<bgpPeer>

<peerAddr>50.52.1.2</peerAddr>

<allowAsNum>3</allowAsNum>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>64902</peerAs>

</bgpPeer>

Configure BGP peer with AS 64902 and peer

address 50.52.1.2

Enable BFD for link failure detection

Configure “allowas” in for the neighbor to accept routes with same ASN learned over this neighbor

<bgpPeer>

<peerAddr>50.54.1.2</peerAddr>

<allowAsNum>3</allowAsNum>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>64902</peerAs>

</bgpPeer>

Configure BGP peer with AS 64902 and peer

address 50.54.1.2

Configure “allowas” in for the neighbor to accept routes with same ASN learned over this neighbor

Enable BFD for link failure detection.

<bgpRedistList>

<redistType>connected</redistType>

</bgpRedistList>

<bgpRedistList>

<redistType>ospf</redistType>

</bgpRedistList>

Redistribute the connected and OSPF routes
</bgp> Close BGP tag.
</vr> Close namespace for the ZebOS module

Tier-2 (spine node) SPINE.xml

This table shows the XML to configure IP addresses:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<interface>

<ifName>xe49/1</ifName>

<ipAddr>50.52.1.2/24</ipAddr>

</interface>

<interface>

<ifName>xe49/2</ifName>

<ipAddr>52.31.1.1/24</ipAddr>

</interface>

Configure the interfaces which are connected

to the SPINE routers.

This payload configures the IP address of the

interfaces xe49/1 and xe49/2.

</vr> Close namespace for the ZebOS module

This table shows the XML to configure BGP attributes:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<bgp>

<bgpAs>64902</bgpAs>

Open BGP tag.

Configure the BGP AS number.

<multipathRelax>1</multipathRelax>

<vrfName>default</vrfName>

Configure “as-path multipath-relax” to relax the

AS-PATH criteria for BGP ECMP

<multipath>

<bgpType>ebgp</bgpType>

<multipathType>

<multipathsNum>8</multipathsNum>

</multipathType>

</multipath>

Configure maximum EBGP ECMP that can be

installed in BGP.

<bgpRedistList>

<redistType>connected</redistType>

</bgpRedistList>

Redistribute the connected routes
<bgpPeer>

<peerAddr>50.52.1.1</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>64601</peerAs>

</bgpPeer>

Configure BGP peer with AS 64602 and peer

address 50.52.1.1

Enable BFD for link failure detection.

<bgpPeer>

<peerAddr>52.31.1.2</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>65501</peerAs>

</bgpPeer>

Configure BGP peer with AS 65501 and peer

address 52.31.1.2

Enable BFD for link failure detection.

</bgp> Close BGP tag.
</vr> Close namespace for the ZebOS module

Tier-3 (core node) CORE.xml

This table shows the XML to configure IP addresses:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<interface>

<ifName>xe1</ifName>

<linkStatus>1</linkStatus>

<ipLabel>NULL</ipLabel>

<ipAddr>52.31.1.2/24</ipAddr>

</interface>

<interface>

<ifName>ge8</ifName>

<linkStatus>0</linkStatus>

<ipLabel>NULL</ipLabel>

<ipAddr>31.22.1.1/24</ipAddr>

</interface>

Configure the interfaces that are connected to

the spine routers.

This payload configures the IP address of the

interfaces xe1 and ge8.

</vr> Close namespace for the ZebOS module

This table shows the XML to configure BGP attributes:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<bgp>

<bgpAs>65501</bgpAs>

Open BGP tag.

Configure the BGP AS number.

<multipathRelax>1</multipathRelax>

<vrfName>default</vrfName>

Configure “as-path multipath-relax” to relax

the AS-PATH criteria for BGP ECMP

<multipath>

<bgpType>ebgp</bgpType>

<multipathType>

<multipathsNum>8</multipathsNum>

</multipathType>

</multipath>

Configure maximum EBGP ECMP that can

be installed in BGP.

<bgpRedistList>

<redistType>connected</redistType>

</bgpRedistList>

Redistribute the connected routes
<bgpPeer>

<peerAddr>52.31.1.1</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>64902</peerAs>

</bgpPeer>

Configure BGP peer with AS 64902 and peer

address 52.31.1.1

Enable BFD for link failure detection.

<bgpPeer>

<peerAddr>31.22.1.2</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>65503</peerAs>

</bgpPeer>

Configure BGP peer with AS 65503 and peer

address 31.22.1.2

Enable BFD for link failure detection.

</bgp> Close BGP tag.
</vr> Close namespace for the ZebOS module

Tier-2 (Border Router)

This table shows the XML to configure IP addresses:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<interface>

<ifName>ge7</ifName>

<ipAddr>53.22.1.2/24</ipAddr>

</interface>

<interface>

<ifName>ge6</ifName>

<ipAddr>31.22.1.2/24</ipAddr>

</interface>

<interface>

<ifName>ge2</ifName>

<ipAddr>23.22.1.1/24</ipAddr>

</interface>

Configure the interfaces which are connected

to the SPINE routers.

This payload configures the IP address of the

interface ge7, ge6 and ge2.

</vr> Close namespace for the ZebOS module

This table shows the XML to configure BGP attributes:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<bgp>

<bgpAs>65503</bgpAs>

Open BGP tag

Configure the BGP AS number.

<multipathRelax>1</multipathRelax>

<vrfName>default</vrfName>

Configure “as-path multipath-relax” to relax

the AS-PATH criteria for BGP ECMP

<multipath>

<bgpType>ebgp</bgpType>

<multipathType>

<multipathsNum>8</multipathsNum>

</multipathType>

</multipath>

Configure maximum EBGP ECMP that can be

installed in BGP.

<bgpRedistList>

<redistType>connected</redistType>

</bgpRedistList>

Redistribute the connected routes
<bgpPeer>

<peerAddr>53.22.1.1</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>64902</peerAs>

</bgpPeer>

Configure BGP peer with AS 64902 and peer

address 52.31.1.1

Enable BFD for link failure detection.

<bgpPeer>

<peerAddr>31.22.1.2</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>65503</peerAs>

</bgpPeer>

Configure BGP peer with AS 65503 and peer

address 31.22.1.2

Enable BFD for link failure detection.

<bgpPeer>

<peerAddr>23.22.1.2</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>65502</peerAs>

<peerRemovePvtAs>1</peerRemovePvtAs>

</bgpPeer>

Configure BGP peer with AS 65502 and peer

address 23.22.1.2

Enable BFD for link failure detection.

Configure remove-private-as for 23.22.1.2 peer

</bgp> Close BGP tag.
</vr> Close namespace for the ZebOS module

Tier-3 (WAN Router) –Partial -Does Not Contain the Internet Configuration

This table shows the XML to configure an IP address:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<interface>

<ifName>ge1/1/1</ifName>

<ipAddr>22.23.1.2/24</ipAddr>

</interface>

Configure the interfaces that are connected to

the SPINE routers.

This payload configures the IP address of the

interface ge1/1/1.

</vr> Close namespace for the ZebOS module

This table shows the XML to configure BGP attributes:

XML Payload Description
<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”> Enter namespace of the ZebOS module
<vrId>0</vrId> Enter Vrid
<bgp>

<bgpAs>65502</bgpAs>

Open BGP tag

Configure the BGP AS number.

<bgpRedistList>

<redistType>connected</redistType>

</bgpRedistList>

Redistribute the connected routes
<bgpPeer>

<peerAddr>22.23.1.1</peerAddr>

<bgpPeerBfd>1</bgpPeerBfd>

<peerAs>65503</peerAs>

</bgpPeer>

Configure BGP peer with AS 64902 and peer

address 52.31.1.1

Enable BFD for link failure detection.

</bgp> Close BGP tag.
</vr> Close namespace for the ZebOS module

Similar configurations must be repeated for all TORs, spines, cores, and border routers.

Appendix B: NetConf User Guide

This document describes managing OcNOS devices using NetConf.

This document is intended for network administrators and other engineering professionals who configure and

manage devices running OcNOS.

There are three different northbound applications in OcNOS (CLI , NetConf, and SNMP). All the northbound

applications are text-based, with each command usually associated to a specific task.

OcNOS NetConf supports transactions as described in Transactions.

NetConf Quick Start

The NetConf protocol defines a simple mechanism through which a network device can be managed, configuration information can be retrieved, and new configuration data can be uploaded.

NetConf uses a simple RPC-based mechanism to communicate between a client and a server. The client can be a script or application typically running as part of a network manager. The server is usually a network device.

A NetConf session is the logical connection between a network administrator or network configuration application

and a network device. A device must support at least one NetConf session and can support multiple sessions. Global configuration attributes can be changed during any session and the effects are visible in all sessions. The candidate and running and startup configuration are shared across all the sessions.

Note: A limited number of protocol modules are supported through NetConf. A detailed list is in the NetConf command reference.

NetConf Clients

You can use any NetConf client application to manage the device using Yang modules. These client applications require a Yang module which you downlaod from:

https://github.com/IPInfusion/OcNOS/tree/1.2

This application establishes a secure connection with the daemon running on the device to perform commands and send system responses.

There are different NetConf client applications. In this document, OpenYuma’s yangcli application is used to show NetConf operations.

Refer to the standard Yangcli operations at: https://github.com/OpenClovis/OpenYuma/blob/master/netconf/doc/yuma_docs/openyuma-yangcli-manual.odt

The NetConf operations get-config and get return large amounts of data. To improve the readability of the output, the subtree filter based sget-config and sget operations are used in this document.

Install Yangcli

Check out the git repository, compile and install OpenYuma components. Refer to the README for more details.

Download Yang files

Download the Yang files from the website:

https://github.com/IPInfusion/OcNOS/tree/1.2

Copy the files to /usr/share/yuma/modules/netconfcentral on the host machine where the client application runs. This system path only works with OpenYuma’s Yangcli client application. If you are running a different client application, follow the respective reference document to copy the Yang files to the appropriate location.

Creating User Accounts

User level access control applies to all NetConf operations. The table below describes access levels for different user types and commands to create different type of user account

Account type Access Command
User Read username <user name> role network-user password

<password>

Operator All, except full configuration store

level change.

username <user name> role network-operator password <password>
Admin All username <user name> role network-admin password

<password>

You must login into the device using network administrator account to create new user accounts. These are the steps:

[[email protected] ~]# ssh [email protected] [email protected]’s password:

Last login: Wed Jun 15 21:27:39 2016

OcNOS version 1.2.0.179-OCNOS-DC-IPBASE-ZEBM IPIRouter 06/14/16 21:30:36

OcNOS>en

OcNOS#configure terminal

Enter configuration commands, one per line. End with CNTL/Z. OcNOS(config)#username netuser role network-user password [email protected] OcNOS(config)#username netoperator role network-operator password [email protected] OcNOS(config)#username netadmin role network-admin password [email protected]

After creating the account, use the write command to write the configured data into persistent storage as

described in Copy Running Configuration to Startup

OcNOS(config)#write Building configuration… [OK]

The NetConf operations for different user account types are shown in Supported Operations.

Yangcli Operations

Establish Connection

These are the steps to establish a connection between the NetConf client and the server that is running on the device.

# yangcli server=<ip _ address> user=<user _ name> password=<password> ip _ address: Address of the device to be managed

user _ name & password: User account detail for authentication

The interactive version of this command is shown below:

# yangcli yangcli> connect

Enter string value for leaf <user> yangcli:connect> <user _ name>

Enter string value for leaf <server> yangcli:connect> <ip _ address>

Enter string value for leaf <password> yangcli:connect> <password>

Load Modules

The load command loads a YANG module into the server. This command is not part of the RFC and is only supported by the Yang client.

The ZebOS module is the parent for all protocol modules supported in OcNOS. This module includes all the sub modules. Therefore, loading this module is mandatory to start managing the device. Here is the portion of ZebOS.yang file that includes other sub modules.

module ZebOS {

namespace “http://www.ipinfusion.com/CMLSchema/ZebOS”;

prefix “ZebOS”; include nsmLACP; include oamBfd;

}

include bridge; include ospf; include vlan; include mstp; include layer2LACP; include bgp; include vr;

include vrf; include interface; include lldpv2; include ospf6; include rib;

include vlaninterface;

> load ZebOS

RPC Data Reply 1 for session 1:

rpc-reply {

mod-revision 2015-10-08

}

After using the load command, you can also use the mgrload command to keep the current session synchronized with the server.

> mgrload ZebOS

Load module ‘ZebOS’ OK

Configure the Device

Configuration details are placed in an XML file and sent to the netconfd server. You must refer to the Yang file to prepare the XML based configuration file with the correct hierarchy. If the hierarchy is not correct, yangcli throws an error.

One portion of the BGP protocol module Yang model is presented below. This module is a sub-module for the parent ZebOS yang module.

submodule bgp {

belongs-to ZebOS { prefix ZebOS; } include interface;

include vrf;

import cml _ data _ types { prefix cml _ data _ types;

}

revision “2015-04-25” {

description “Revised on 2015-04-25.”; }

grouping bgp-grouping { list bgp {

description “bgp”; config true;

key “bgpAs”; leaf vrId {

mandatory false; type leafref {

path “/vr/vrId”;

}

} // END of vrId definition.

leafkeepAlive { mandatory false;

type cml _ data _ types:CML _ UINT16 _ T { range “0..65535”;

}

default “30”; config true;

} // END of keepAlive definition.

leafholdTime { mandatory false;

type cml _ data _ types:CML _ UINT16 _ T { range “0..65535”;

}

default “90”; config true;

} // END of holdTime definition.

} // END of bgpDebug-grouping definition.

Based on the hierarchy in the Yang module. the following XML file named bgp.xml is created with the configuration data. The bgp.xml file is referenced in the edit-config operation specified below.

<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”>

<vrId>0</vrId>

<bgp>

<bgpAs>100</bgpAs>

<keepAlive>60</keepAlive>

<holdTime>180</holdTime>

</bgp>

</vr>

Use this command to globally set or reset the keepalive and holdtime values for all the neighbors.

yangcli [email protected]> edit-config [email protected]/root/bgp.xml Filling container /edit-config/input/target:

RPC OK Reply 15 for session 1:

Retrieve Candidate Configuration

Candidate configuration datastore is used to hold configuration data that can be manipulated without impacting the device’s current configuration. The candidate configuration is a full configuration data set that serves as a work place for creating and manipulating configuration data. Additions, deletions, and changes can be made to this data to construct the desired configuration data.

>sget-config /vr/bgp source=candidate Filling list /vr/bgp:

RPC Data Reply 1 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default

}

}

}

}

Commit Candidate Configuration

A <commit> operation MAY be performed at any time that causes the device’s running configuration to be set to the value of the candidate configuration.

yangcli [email protected]> commit RPC OK Reply 16 for session 1:

Retrieve Running Configuration

Configuration data is the set of writable data that is required to transform a system from its initial default state into its current state. You can use the get-config operation to fetch the running configuration data.

>sget-config /vr/bgp source=running Filling list /vr/bgp:

RPC Data Reply 1 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default

}

}

}

}

Retrieve Running Configuration and State Data

State data is the additional data on a system that is not configuration data such as read-only status information and collected statistics. You can use the get operation to fetch a protocol module’s running configuration and state data.

>sget /vr/bgp rpc-reply {

data { vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default bgpTableVersion 0

ntwkPrefixCount 0

ibgpMetric 0

pathAttrBest 0 clusterList fill _ value

routerRunIpAddr fill _ value bgpShowTypeStr fill _ value bgpShowType 0

rfdMaxPenaltyCeil 0

rfdMinPenaltyFloor 0

dampeningStr 0

rfdCbStr 0

ibgpMetric 0

}

}

}

}

Copy Running Configuration to Startup

NetConf supports startup options, so if you have configured the device and want to retain the configuration after a device reboot, copy the running configuration into startup configuration. The yangcli command is:

> copy-config source=running target=startup The equivalent OcNOS command is

# write

Error Messages

NetConf operations return protocol, management layer, and protocol module errors. The example below depicts an error returned by a protocol module.

Copy the content below into bgp_err.xml.

<vr xmlns=”http://www.ipinfusion.com/CMLSchema/ZebOS”>

<vrId>0</vrId>

<bgp>

<bgpAs>1</bgpAs>

<keepAlive>300</keepAlive>

<holdTime>200</holdTime>

</bgp>

</vr>

Execute the following command and commit the changes

yangcli [email protected]> edit-config [email protected]/root/bgp _ err.xml Filling container /edit-config/input/target:

RPC OK Reply 12 for session 1: yangcli [email protected]> commit

mgr _ rpc: got invalid reply on session 1 (invalid XPath expression syntax) RPC Error Reply 13 for session 1:

rpc-reply { rpc-error {

error-type protocol error-tag

error-severity warning

error-app-tag general-warning error-path ‘RPC operation failed’

error-message ‘%% Hold time should be greater than the keepalive time’ error-info {

error-number 4294962411

}

}

}

Transactions

OcNOS supports transaction-oriented configuration management. Transactions are created implicitly by edit- config operations; commit and discard-changes operations close or terminate the transactions. Successive edit-config operations are placed in the same transaction.

Discard Changes

Discard the transaction or candidate configuration changes.

yangcli [email protected]> sget-config /vr/ospf source=candidate Filling list /vr/ospf:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 14 for session 2:

rpc-reply { data {

vr {

ospf 0 {

ospfProcessId 0 ospfShutDown true

}

ospf 200 {

ospfProcessId 200

}

}

}

}

yangcli [email protected]> discard-changes RPC OK Reply 15 for session 2:

yangcli [email protected]> sget-config /vr/ospf source=candidate Filling list /vr/ospf:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 16 for session 2:

rpc-reply { data {

vr {

ospf 0 {

ospfProcessId 0 ospfShutDown true

}

}

}

}

Commit

Commit the transaction or candidate configuration changes.

yangcli [email protected]> create /vr/ospf

Filling list /vr/ospf:

Filling key leaf /vr/ospf/ospfProcessId: Enter int32 value for leaf <ospfProcessId> yangcli [email protected]:create> 500

Filling key leaf /vr/vrId:

Enter uint32 value for leaf <vrId> yangcli [email protected]> 0

RPC OK Reply 17 for session 2: yangcli [email protected]> commit RPC OK Reply 18 for session 2:

yangcli [email protected]> sget-config /vr/ospf source=running Filling list /vr/ospf:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 19 for session 2:

rpc-reply { data {

vr {

ospf 0 {

ospfProcessId 0 ospfShutDown true

}

ospf 500 {

ospfProcessId 500

}

}

}

}

Save Point

OcNOS supports the savepoint feature. A savepoint is a snapshot of the device current state. You can switch the device state to any savepoint at any time.

This example creates two savepoints HT180 and HT300

  • HT180has holdTime set to 180
  • HT300has holdTime set to 300

yangcli [email protected]> create

create create-savepoint create-subscription yangcli [email protected]> create-savepoint

Enter string value for leaf <savepointName> yangcli [email protected]:create-savepoint> HT180

RPC OK Reply 4 for session 2:

yangcli [email protected]> edit-config [email protected]/root/bgp _ ht.xml Filling container /edit-config/input/target:

RPC OK Reply 5 for session 2: yangcli [email protected]> commit RPC OK Reply 6 for session 2:

yangcli [email protected]> sget-config /vr/bgp source=running

Filling list /vr/bgp:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 7 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 300

keepAlive 60

vrfName default

}

}

}

}

yangcli [email protected]> create-savepoint HT300 RPC OK Reply 8 for session 2:

Roll Back

This example shows rolling back the configuration. First, notice the current value of the holdTime attribute:

yangcli [email protected]> sget-config /vr/bgp source=running

Filling list /vr/bgp:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 10 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default

}

}

}

}

Now roll back to savepoint HT300 and note the value of holdTime attribute.

yangcli [email protected]> rollback-transaction HT300 RPC OK Reply 11 for session 2:

yangcli [email protected]> sget-config /vr/bgp source=running

Filling list /vr/bgp:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 12 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 300

keepAlive 60 vrfName default

}

}

}

}

Supported Operations

All the NetConf operations are captured based on the capability. Hence, any operation falling in multiple

capability are documented separately.

Note: Capability “base:1.0” supports candidate and running configuration store.

Capability Operation User Role Supported (Yes/No) Comments
:base:1.0 <get> User, Operator, Admin Yes
:base:1.0 <get> with subtree filter User, Operator, Admin Yes
:base:1.0 <get-config> User, Operator, Admin Yes
:base:1.0 get-config source=<target> User, Operator, Admin Yes
:base:1.0 <get-config> with subtreefilter User, Operator, Admin Yes
:base:1.0 <edit-config> <target> as parameter Operator, Admin Yes Running configuration as target is not supported because writtable- running capability is not supported
:base:1.0 <edit-config> <config> as parameter Operator, Admin Yes
:base:1.0 <edit-config>:

<merge>

Operator, Admin Yes
:base:1.0 <edit-config>:

<replace>

Operator, Admin Yes
:base:1.0 <edit-config>:

<create>

Operator, Admin Yes
:base:1.0 <edit-config>:

<delete>

Operator, Admin Yes
:base:1.0 <edit-config>:

<remove>

Operator, Admin Yes
Capability Operation User Role Supported (Yes/No) Comments
:base:1.0 <edit-config>: <none> Operator, Admin Yes
:base:1.0 <edit-config>: <error- option > as stop-on- error Operator, Admin No
:base:1.0 <edit-config>: <error- option > as continue- on-error Operator, Admin No
:rollback-on- error:1.0 <edit-config>: <error- option > as rollback- on-error Operator, Admin Partial By default, this is the behavior. So there is no need to pass this error- option.
:validate: 1.1 <edit-config>: <test- option > Operator, Admin No By default configuration entries are validated and stored, hence

this operation and its parameters are not handled. External configuration store validation is not supported (i.e URL).

:base:1.0 <copy-config> : < target> <source> Operator, Admin Yes
:base:1.0 <lock> : < target> Operator, Admin Yes Candidate configuration

lock is not supported.

:base:1.0 <unlock> : < target> Operator, Admin Yes Candidate configuration

unlock is not supported

:base:1.0 <close-session> close current session User, Operator, Admin Yes
:base:1.0 <kill-session> : Close other session User, Operator, Admin Yes
:base:1.0 subtreefiltering User, Operator, Admin Yes
:Startup:1.0 get-config

<source=startup>

User, Operator, Admin Yes
:Startup:1.0 copy-config

<source=startup>

Admin No
:Startup:1.0 copy-config

<target=startup>

Admin Partial Running to startup copy is supported.
:Startup:1.0 lock <startup> Admin Yes
:Startup:1.0 unlock <startup> Admin Yes
Capability Operation User Role Supported (Yes/No) Comments
:Startup:1.0 validate

<source=startup>

Admin No Always configuration entries are validated and stored, but external configuration store validation is not supported.
:Startup:1.0 delete-config Admin Yes
:url:1.0 URL capability User, Operator, Admin No Not supported
:with-defaults <get> User, Operator, Admin Partial By default, “report-all” functionality is supported when with-default

option is passed. Other options (explicit, tagged, explicit-tagged) are not supported

:with-defaults <get-config> User, Operator, Admin Partial By default, “report-all” functionality is supported when with-default

option is passed. Other options (explicit, tagged, explicit-tagged) are not supported

:confirmed- commit <commit> <confirmed> Operator, Admin Yes
:confirmed- commit <commit> <confirmed>

<persist>

Operator, Admin Yes