CHAPTER 1

Data Center Interconnect Overview

Large-Scale Data Center Requirements

Large-Scale Data Center Topologies

Large-Scale Data Center Routing

EBGP-Routed Clos Topology-Based Data Center

EBGP Data Center Design using OcNOS

CHAPTER 2

Configuration 8

ToR (leaf node)

Tier-2 (spine node)

Tier-3 (core node)

Tier-2 (border router)

Tier-3 (WAN router)

Other Configurations

Validation

Conclusion

References

Appendix A: Configuring the Data Center through NetConf

Appendix B: NetConf User Guide

Glossary

Chapter 1

Data Center Overview

Network Automation provides IT administrators and network operators significant benefits. This solution guide

describes an approach to build data centers using Layer3 BGP routing protocol.

It also summarizes on some design philosophies for data center and why E-BGP is better suited.

• Large-scaledata center requirements
• Large-scaledata center topologies
• Large-scaledata center routing
• EBGP-routedlarge-scale Clos topology-based data center

Large-Scale Data Center Requirements

The design of large-scale data centers is driven by operational simplicity and network stability. Operational simplicity and network stability ensures easier manageability and therefore reduced operational expenses. From the network design aspect, the requirements are:

• Abilityto accommodate the variable-application bandwidth and strict latency
• Abilityto handle the increased east-west (server-to-server) traffic within the data center due to massive

data replication between clusters and virtual machine migrations.

• Traffic-Engineeringwith application load The network infrastructure should itself perform controlled per-hop traffic engineering.
• MinimizeCAPEX and incorporate vendor diversity by using a simple, interoperable routing protocol with a minimal set of
• Adesign to minimize OPEX by keeping the failure domain at the lowest level in the network

Large-Scale Data Center Topologies

A traditional tree-based (upside down) topology with a three-layer hierarchy of core, aggregation and access

layer can be used in a data center design. This approach is suited if the majority of the traffic is entering

and leaving (north-south) the data center. An increase in bandwidth requirements then can be addressed by upgrading the device line cards or port density. However with the current trend of increasing server-to-server (east-west) traffic, scaling these networks horizontally is expensive or impossible at times.

A Clos network (leaf and spine) is a horizontally scalable topology where every leaf node is connected to every other spine. The topology can be extended to different stages for scaling. Clos networks are fully non- blocking and load balancing is inherent in the topology itself as all available paths are ECMP. Clos networks are ideal for the current requirements of a large-scale data center.

Large-Scale Data Center Routing

Layer 2-only routing was used in a traditional tree-based data center topology. Traditional layer-2 protocols such as STP do not give bi-sectional bandwidth, whereas recent developments such as TRILL, SPB have selected vendor support.

However, a hybrid of layer 2 /layer 3 can be used to limit the size of failure domain and scale up the data center. Layer 3 routing can be used in tier 1 (core) and layer 2 in tier 3 (access). Tier 2 can be based on either layer 2 or layer 3. A hybrid model has the advantage of seamless Virtual Machine mobility and requires less IP subnets for the data center. Although this design can scale-up, it is difficult and complex to manage the different protocols.

A layer 3 only design simplifies the network and improves network stability and scalability, as well as

localizing the failure domain (confined to the L2 broadcast domain). Seamless virtual mobility can be achieved in a L3 only based data center by using L2 overlay networks. From experiment and analysis, External BGP (EBGP) is considered ideal compared to IGPs due to the following [See Reference]:

• Lesscomplex protocol, simple state machine
• Informationflooding overhead is less, no frequent updates unlike IGPs
• Networkfailure recovery is very Although BGP convergence is slower than IGP, in a Clos topology with ECMP links, the failure is masked as soon as an alternate path is found.
• Failuredomain is minimized in a Clos topology with Some of the failures are local/hidden/not propagated if the failed link was not selected/advertised as the best path among the ECMP paths by the BGP speaker. The failures, where all devices have to withdraw some prefixes or update the ECMP groups in the FIB, are very limited and in those failures the failed link/node does not impact the re-convergence process.
• Administratorcan define the application traffic BGP provides services like prefix distribution, prefix filtering, traffic engineering, traffic tagging, and multi-vendor stability better than other IGPs.
• Easierto

EBGP-Routed CLOS Topology-Based Data Center

EBGP-routed CLOS topology is considered the best choice for laying the IP fabric in a data center because of the horizontal scalability feature of Clos topology and the ease of use and services provided by EBGP especially prefix-filtering, prefix distribution, and traffic engineering which are required extensively in a data center.

Configuration Guidelines

Configuration guidelines for laying IP fabric using EBGP efficiently are as follows:

• Runall EBGP sessions over single-hop point-to-point
• Useprivate Autonomous System Numbers (ASNs) (64512-65534) to avoid ASN
• Giveall tier 1 (core) devices a single
• Giveall tier 2 devices in the same cluster the same unique A cluster or pod is a group of tier 2 (spine switches) + tier 3 switches (ToR/leaf) + servers.
• Giveevery tier 3 (ToR) device in a cluster a unique
• Reusetier-3 ASNs across Configure tier-3 devices with the BGP “allowas-in” feature to allow route learning of prefixes from the same ASNs in other clusters.
• Announceserver subnets on tier-3 devices via BGP without using route summarization on tier-2 and tier-1
• Useedge clusters (pods) for external connectivity. Each edge cluster consists of border routers (tier-2) and WAN routers (tier-3). Give each WAN router a unique public ASN to connect the data center to the external world.
• Forborder routers, remove private ASNs before sending the information to WAN routers by configuring border routers with the “remove-private-AS” BGP
• Torelax the BGP ECMP criteria for AS paths, configure BGP “as-path multipath-relax” on all routers/ This way, an equal cost path with a different AS PATH, but the same AS PATH length is also considered an equal cost path (ECMP).

• Forfaster failure detection, configure the BGP session with
• Toavoid recurring BPG update/selection for a single failure through all peers or BGP update message dispersion on a particular speaker, use BGP update The BGP update group feature processes an update once and sends it to a group of neighbors that share a common outbound policy. The BGP RIB is scanned every time for each peer to apply the outbound filter.
• Toavoid micro routing loops, configure tier-2 and tier-1 with static discard or null routes rather than a default Routing loops can happen when a tier-2 device has lost all its learned prefixes, but has a default

route to a tier-1 device and that tier-1 device still has a route back to the tier-2 device.

EBGP Data Center Design using OcNOS

Figure-1 shows a minimal representation that encompasses all the elements in a layer 3 data center. The number of ECMPs in the data center is equal to the number of cores (tier-1 switches).

Figure 1 . IP fabric using EBGP

Core Tier 1

Figure 2 shows the Autonomous System Number (ASN) allocation scheme used in the data center. The WAN routers are assigned a public ASN, which connects the data center to external world. The tier-3 ASNs per ToR are reused across the clusters.

Figure 2: ASN allocation in an EBGP-based data center

ASN 65534

Chapter 2

Configuration

To R (Leaf node)

Tier-2 (Spine node)

Tier-2 (Spine node) cont.

Tier-3 (Core node)

Tier 2 (Border router)

Tier 2 (Border router) Cont.

Tier-3 (WAN router)

This is a partial list and does not contain the Internet configuration.

Other Configurations

You must repeat similar configurations for all ToR, spine, core, border, and WAN devices as well.

Validation

Use the show ip bgp command to validate the output at each node.

Consider the following case: for application load balancing and high availability/reliability, two similar application servers can be placed at two clusters. For users accessing the application server through the Internet, the access to the server is load balanced and failure of one of the application servers does not impact the accessibility. The following is the output at various nodes for a subnet, such as:

• 70.70.70.1(application server) at ToR1 in cluster 1 and cluster 2
• 80.80.80.1at ToR 1 cluster 1
• 90.90.1at ToR 1 cluster 2

Conclusion

OcNOS with EBGP routing is a highly scalable, simple and flexible way of laying IP fabric in a data center.

The data center can be easily scaled for:

• Highercomputing needs by adding more
• Higherperformance and redundancy by adding more cores
• Higheruplink speeds by adding more external/edge

References

Use of BGP for routing in large scale data centers: https://tools.ietf.org/html/draft-ietf-rtgwg-bgp-routing-large-dc-09

Appendix A: Configuring the Data

Center through NetConf

This appendix shows how to configure a data center through NetConf. This section contains XML payloads which can be used by any NetConf client to send configurations to OcNOS devices.

The configurations below in XML are communicated via RPC messages from the NetConf client to the

NetConf server.

This document uses the yang-cli which is part of open source OpenYumaMaster which and is preinstalled on a ZebM-enabled OcNOS ONIE package as a NetConf client.

Pre-requisite

Refer to Yangcli Operations section of the NetConf User Guide and perform these operations first:

1. EstablishConnection
2. LoadModules

Once the above operations are complete, edit-config operation can be performed with XML payloads using

the command:

yangcli ocnos@Hostname> edit-config config=@/<path-to-xml-file>/<xml-file>.xml

Save the below xml payloads as an .xml file in any desired location. Make sure that each XML payload starts

with <vr xmlns=”<namespace>”> and ends with </vr>.

TOR (leaf node) TOR.xml

This table shows the XML to configure an IP address:

This is an example of the XML payload to be sourced in the yang CLI to configure an interface IP addresses:

<vr xmlns=”https://www.ipinfusion.com/CMLSchema/ZebOS”>

<vrId>0</vrId>

<interface>

<ifName>xe49</ifName>

<ipAddr>50.52.1.1/24</ipAddr>

</interface>

<interface>

<ifName>ge2</ifName>

<ipAddr>50.54.1.1/24</ipAddr>

</interface>

</vr>

This table shows the XML to configure BGP attributes:

Tier-2 (spine node) SPINE.xml

This table shows the XML to configure IP addresses:

This table shows the XML to configure BGP attributes:

Tier-3 (core node) CORE.xml

This table shows the XML to configure IP addresses:

This table shows the XML to configure BGP attributes:

Tier-2 (Border Router)

This table shows the XML to configure IP addresses:

This table shows the XML to configure BGP attributes:

Tier-3 (WAN Router) –Partial -Does Not Contain the Internet Configuration

This table shows the XML to configure an IP address:

This table shows the XML to configure BGP attributes:

Similar configurations must be repeated for all TORs, spines, cores, and border routers.

Appendix B: NetConf User Guide

This document describes managing OcNOS devices using NetConf.

This document is intended for network administrators and other engineering professionals who configure and

manage devices running OcNOS.

There are three different northbound applications in OcNOS (CLI , NetConf, and SNMP). All the northbound

applications are text-based, with each command usually associated to a specific task.

OcNOS NetConf supports transactions as described in Transactions.

NetConf Quick Start

The NetConf protocol defines a simple mechanism through which a network device can be managed, configuration information can be retrieved, and new configuration data can be uploaded.

NetConf uses a simple RPC-based mechanism to communicate between a client and a server. The client can be a script or application typically running as part of a network manager. The server is usually a network device.

A NetConf session is the logical connection between a network administrator or network configuration application

and a network device. A device must support at least one NetConf session and can support multiple sessions. Global configuration attributes can be changed during any session and the effects are visible in all sessions. The candidate and running and startup configuration are shared across all the sessions.

Note: A limited number of protocol modules are supported through NetConf. A detailed list is in the NetConf command reference.

NetConf Clients

You can use any NetConf client application to manage the device using Yang modules. These client applications require a Yang module which you downlaod from:

https://github.com/IPInfusion/OcNOS/tree/1.2

This application establishes a secure connection with the daemon running on the device to perform commands and send system responses.

There are different NetConf client applications. In this document, OpenYuma’s yangcli application is used to show NetConf operations.

Refer to the standard Yangcli operations at: https://github.com/OpenClovis/OpenYuma/blob/master/netconf/doc/yuma_docs/openyuma-yangcli-manual.odt

The NetConf operations get-config and get return large amounts of data. To improve the readability of the output, the subtree filter based sget-config and sget operations are used in this document.

Install Yangcli

Check out the git repository, compile and install OpenYuma components. Refer to the README for more details.

Download Yang files

Download the Yang files from the website:

https://github.com/IPInfusion/OcNOS/tree/1.2

Copy the files to /usr/share/yuma/modules/netconfcentral on the host machine where the client application runs. This system path only works with OpenYuma’s Yangcli client application. If you are running a different client application, follow the respective reference document to copy the Yang files to the appropriate location.

Creating User Accounts

User level access control applies to all NetConf operations. The table below describes access levels for different user types and commands to create different type of user account

You must login into the device using network administrator account to create new user accounts. These are the steps:

[root@localhost ~]# ssh ocnos@10.12.45.253 ocnos@10.12.45.253’s password:

Last login: Wed Jun 15 21:27:39 2016

OcNOS version 1.2.0.179-OCNOS-DC-IPBASE-ZEBM IPIRouter 06/14/16 21:30:36

OcNOS>en

OcNOS#configure terminal

Enter configuration commands, one per line. End with CNTL/Z. OcNOS(config)#username netuser role network-user password Abc@6789 OcNOS(config)#username netoperator role network-operator password Abc@6789 OcNOS(config)#username netadmin role network-admin password Abc@6789

After creating the account, use the write command to write the configured data into persistent storage as

described in Copy Running Configuration to Startup

OcNOS(config)#write Building configuration… [OK]

The NetConf operations for different user account types are shown in Supported Operations.

Yangcli Operations

Establish Connection

These are the steps to establish a connection between the NetConf client and the server that is running on the device.

# yangcli server=<ip _ address> user=<user _ name> password=<password> ip _ address: Address of the device to be managed

user _ name & password: User account detail for authentication

The interactive version of this command is shown below:

# yangcli yangcli> connect

Enter string value for leaf <user> yangcli:connect> <user _ name>

Enter string value for leaf <server> yangcli:connect> <ip _ address>

Enter string value for leaf <password> yangcli:connect> <password>

Load Modules

The load command loads a YANG module into the server. This command is not part of the RFC and is only supported by the Yang client.

The ZebOS module is the parent for all protocol modules supported in OcNOS. This module includes all the sub modules. Therefore, loading this module is mandatory to start managing the device. Here is the portion of ZebOS.yang file that includes other sub modules.

module ZebOS {

namespace “https://www.ipinfusion.com/CMLSchema/ZebOS”;

prefix “ZebOS”; include nsmLACP; include oamBfd;

…

}

include bridge; include ospf; include vlan; include mstp; include layer2LACP; include bgp; include vr;

include vrf; include interface; include lldpv2; include ospf6; include rib;

include vlaninterface;

> load ZebOS

RPC Data Reply 1 for session 1:

rpc-reply {

mod-revision 2015-10-08

}

After using the load command, you can also use the mgrload command to keep the current session synchronized with the server.

> mgrload ZebOS

Load module ‘ZebOS’ OK

Configure the Device

Configuration details are placed in an XML file and sent to the netconfd server. You must refer to the Yang file to prepare the XML based configuration file with the correct hierarchy. If the hierarchy is not correct, yangcli throws an error.

One portion of the BGP protocol module Yang model is presented below. This module is a sub-module for the parent ZebOS yang module.

submodule bgp {

belongs-to ZebOS { prefix ZebOS; } include interface;

include vrf;

import cml _ data _ types { prefix cml _ data _ types;

}

revision “2015-04-25” {

description “Revised on 2015-04-25.”; }

grouping bgp-grouping { list bgp {

description “bgp”; config true;

key “bgpAs”; leaf vrId {

mandatory false; type leafref {

path “/vr/vrId”;

…

}

} // END of vrId definition.

leafkeepAlive { mandatory false;

type cml _ data _ types:CML _ UINT16 _ T { range “0..65535”;

}

default “30”; config true;

} // END of keepAlive definition.

leafholdTime { mandatory false;

type cml _ data _ types:CML _ UINT16 _ T { range “0..65535”;

…

}

default “90”; config true;

} // END of holdTime definition.

} // END of bgpDebug-grouping definition.

Based on the hierarchy in the Yang module. the following XML file named bgp.xml is created with the configuration data. The bgp.xml file is referenced in the edit-config operation specified below.

<vr xmlns=”https://www.ipinfusion.com/CMLSchema/ZebOS”>

<vrId>0</vrId>

<bgp>

<bgpAs>100</bgpAs>

<keepAlive>60</keepAlive>

<holdTime>180</holdTime>

</bgp>

</vr>

Use this command to globally set or reset the keepalive and holdtime values for all the neighbors.

yangcli ocnos@10.12.45.253> edit-config config=@/root/bgp.xml Filling container /edit-config/input/target:

RPC OK Reply 15 for session 1:

Retrieve Candidate Configuration

Candidate configuration datastore is used to hold configuration data that can be manipulated without impacting the device’s current configuration. The candidate configuration is a full configuration data set that serves as a work place for creating and manipulating configuration data. Additions, deletions, and changes can be made to this data to construct the desired configuration data.

>sget-config /vr/bgp source=candidate Filling list /vr/bgp:

RPC Data Reply 1 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default

}

}

}

}

Commit Candidate Configuration

A <commit> operation MAY be performed at any time that causes the device’s running configuration to be set to the value of the candidate configuration.

yangcli ocnos@10.12.45.253> commit RPC OK Reply 16 for session 1:

Retrieve Running Configuration

Configuration data is the set of writable data that is required to transform a system from its initial default state into its current state. You can use the get-config operation to fetch the running configuration data.

>sget-config /vr/bgp source=running Filling list /vr/bgp:

RPC Data Reply 1 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default

}

}

}

}

Retrieve Running Configuration and State Data

State data is the additional data on a system that is not configuration data such as read-only status information and collected statistics. You can use the get operation to fetch a protocol module’s running configuration and state data.

>sget /vr/bgp rpc-reply {

data { vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default bgpTableVersion 0

ntwkPrefixCount 0

ibgpMetric 0

pathAttrBest 0 clusterList fill _ value

routerRunIpAddr fill _ value bgpShowTypeStr fill _ value bgpShowType 0

rfdMaxPenaltyCeil 0

rfdMinPenaltyFloor 0

dampeningStr 0

rfdCbStr 0

ibgpMetric 0

}

}

}

}

Copy Running Configuration to Startup

NetConf supports startup options, so if you have configured the device and want to retain the configuration after a device reboot, copy the running configuration into startup configuration. The yangcli command is:

> copy-config source=running target=startup The equivalent OcNOS command is

# write

Error Messages

NetConf operations return protocol, management layer, and protocol module errors. The example below depicts an error returned by a protocol module.

Copy the content below into bgp_err.xml.

<vr xmlns=”https://www.ipinfusion.com/CMLSchema/ZebOS”>

<vrId>0</vrId>

<bgp>

<bgpAs>1</bgpAs>

<keepAlive>300</keepAlive>

<holdTime>200</holdTime>

</bgp>

</vr>

Execute the following command and commit the changes

yangcli tbyran@10.12.45.253> edit-config config=@/root/bgp _ err.xml Filling container /edit-config/input/target:

RPC OK Reply 12 for session 1: yangcli tbyran@10.12.45.253> commit

mgr _ rpc: got invalid reply on session 1 (invalid XPath expression syntax) RPC Error Reply 13 for session 1:

rpc-reply { rpc-error {

error-type protocol error-tag

error-severity warning

error-app-tag general-warning error-path ‘RPC operation failed’

error-message ‘%% Hold time should be greater than the keepalive time’ error-info {

error-number 4294962411

}

}

}

Transactions

OcNOS supports transaction-oriented configuration management. Transactions are created implicitly by edit- config operations; commit and discard-changes operations close or terminate the transactions. Successive edit-config operations are placed in the same transaction.

Discard Changes

Discard the transaction or candidate configuration changes.

yangcli ocnos@10.12.45.253> sget-config /vr/ospf source=candidate Filling list /vr/ospf:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 14 for session 2:

rpc-reply { data {

vr {

ospf 0 {

ospfProcessId 0 ospfShutDown true

}

ospf 200 {

ospfProcessId 200

}

}

}

}

yangcli ocnos@10.12.45.253> discard-changes RPC OK Reply 15 for session 2:

yangcli ocnos@10.12.45.253> sget-config /vr/ospf source=candidate Filling list /vr/ospf:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 16 for session 2:

rpc-reply { data {

vr {

ospf 0 {

ospfProcessId 0 ospfShutDown true

}

}

}

}

Commit

Commit the transaction or candidate configuration changes.

yangcli ocnos@10.12.45.253> create /vr/ospf

Filling list /vr/ospf:

Filling key leaf /vr/ospf/ospfProcessId: Enter int32 value for leaf <ospfProcessId> yangcli ocnos@10.12.45.253:create> 500

Filling key leaf /vr/vrId:

Enter uint32 value for leaf <vrId> yangcli ocnos@10.12.45.253> 0

RPC OK Reply 17 for session 2: yangcli ocnos@10.12.45.253> commit RPC OK Reply 18 for session 2:

yangcli ocnos@10.12.45.253> sget-config /vr/ospf source=running Filling list /vr/ospf:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 19 for session 2:

rpc-reply { data {

vr {

ospf 0 {

ospfProcessId 0 ospfShutDown true

}

ospf 500 {

ospfProcessId 500

}

}

}

}

Save Point

OcNOS supports the savepoint feature. A savepoint is a snapshot of the device current state. You can switch the device state to any savepoint at any time.

This example creates two savepoints HT180 and HT300

• HT180has holdTime set to 180
• HT300has holdTime set to 300

yangcli ocnos@10.12.45.253> create

create create-savepoint create-subscription yangcli ocnos@10.12.45.253> create-savepoint

Enter string value for leaf <savepointName> yangcli ocnos@10.12.45.253:create-savepoint> HT180

RPC OK Reply 4 for session 2:

yangcli ocnos@10.12.45.253> edit-config config=@/root/bgp _ ht.xml Filling container /edit-config/input/target:

RPC OK Reply 5 for session 2: yangcli ocnos@10.12.45.253> commit RPC OK Reply 6 for session 2:

yangcli ocnos@10.12.45.253> sget-config /vr/bgp source=running

Filling list /vr/bgp:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 7 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 300

keepAlive 60

vrfName default

}

}

}

}

yangcli ocnos@10.12.45.253> create-savepoint HT300 RPC OK Reply 8 for session 2:

Roll Back

This example shows rolling back the configuration. First, notice the current value of the holdTime attribute:

yangcli ocnos@10.12.45.253> sget-config /vr/bgp source=running

Filling list /vr/bgp:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 10 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 180

keepAlive 60 vrfName default

}

}

}

}

Now roll back to savepoint HT300 and note the value of holdTime attribute.

yangcli ocnos@10.12.45.253> rollback-transaction HT300 RPC OK Reply 11 for session 2:

yangcli ocnos@10.12.45.253> sget-config /vr/bgp source=running

Filling list /vr/bgp:

mgr _ rpc: got invalid reply on session 1 (missing index component) RPC Data Reply 12 for session 2:

rpc-reply { data {

vr {

bgp 100 {

bgpAs 100

holdTime 300

keepAlive 60 vrfName default

}

}

}

}

Supported Operations

All the NetConf operations are captured based on the capability. Hence, any operation falling in multiple

capability are documented separately.

Note: Capability “base:1.0” supports candidate and running configuration store.