Growing Your Data Center Efficiently

31st Jul 2017

Data centers of various scales are designed with growth in mind. Over the evolution of data center designs, few technologies have been proven to be the optimal choices for expansion and in-service expansion has become the new critical SLA for data centers.

As expected, these fall either in the category of a proprietary or a standards-based open technology. Choosing a reliable open technology helps operators to reduce capital and operational expenditures. It also allows them to incorporate innovations faster.

Irrespective of the choice, the technology must be designed to make the data center fabric reliable, expandable, and able to meet software service needs of applications running on the servers. These primarily come down to the fabric support for data transfers without any traffic loss, between application servers within a cluster or in and out of the cluster.

The requirement for a non-blocking server to server to connectivity is best met using the CLOS design architecture. It is a well proven design philosophy used in telecom to reduce line-to-line cross bar switch numbers, yet providing a non-interfering and non-blocking service.  Standards-based open technologies used for data center fabric design assume and rely on a CLOS topology framework as a basis. Over a base CLOS topology popular technologies, such as a L3 fabric, VXLAN or DC-Fabric, such as TRILL or SPB, are mapped out.

A large data center, when broken down to further logical groups at the minimal installation level consists of application servers, data links carrying traffic, switches/routers for interconnection. The data links, switches and the servers are connected in CLOS design paradigm as explained before. One common term for this minimum installation level unit of a data center is a ‘Pod’.  The ‘Pod’ therefore is used as logical unit when incrementing a data center capacity.

The physical size of a Pod can be estimated given the number of application servers that can fit in a rack. As an example 40 servers would require 40 * 1 RU and an additional 3 RU for switches; thus needing 43 RU per Pod.

Given a standard CLOS design with 1 spine and 2 leaf switches or 2 spine and 2 leaf switches:

(The 2 leaf and 2 spine design allows for redundancy at two stages)

 

With leaf switch capacity of 48 * 10G ports, and Spine capacity of 32*40G ports, total number of RUs required to accommodate the physical unit given a target number of servers would be:

As seen in the chart above, a single rack chassis composed of about 44 RUs gets utilized to maximum  capacity with about 40 servers, and therefore to accommodate a yet increased number the ‘pod’ would need to be more than one chassis rack.

Besides the size and space requirement, the total approximate power consumed by 40 servers, 2 leaf switches and 2 spine switches comes to 15.84 KW.

Server considered: Dell Poweredge-R330*

http://www.dell.com/en-us/work/shop/povw/poweredge-r330 – 350W * 40 = 14000 watt

Switch considered: Dell S4048-ON*

http://www.dell.com/support/home/us/en/04/Configuration/ServiceTag/J29YWC2 – 460W * 4 = 1840 watt

This challenge to add more number of compute and network servers in the same physical space in the rack and therefore increasing the overall server density of a data center has been solved by a few.

As seen in sample scaling graph below, there is a more of a linear increase in space (combined with switching unit space) as the number of servers increase.

 

“Aparna Systems” is a pioneer in this area and has designed a ‘Cloud in a box’ solution which integrates dense servers with a switched backplane, in a 4 RU solution. The Orca µcloud™ 4060 hosts up to 60 micro servers and the Orca µcloud™ 4015 hosts up to 15 micro servers in a 4 RU space.

http://aparnasystems.com/products/

Besides hosting multiple servers in a 4 RU format, it also integrates 2 leaf switches running the OcNOS network operating system. Given a 4015 Orca Server; it provides for a capacity for 15, 1 RU servers and 2 Leaf switches with integrated high speed internal data bus.  This gives Orca-4015 about 75% more efficient in space when compared with a similar discrete solution.

ORCA 4060 fully loaded (60 16 core servers) consumes 4.3KW, includes 1x TOR switches

 

When compared relative to standard solution:

Reduced space also implies common enhanced cooling mechanisms can be benefited and shared by the entire chassis rather than individual units. This drastically reduces the power consumption by the chassis. Additionally the design includes FRUs for servers, fans and PSUs & meets field operational requirements.

The TOR in a box solution offers SSD internal storage in the servers with 480TB and 120TB for the ORCA 4060 and 4105 respectively.

The power consumption of the ORCA solution including 2 leaf switches is significantly lower, considering the ORCA4060 product capable of 60 servers, consumes only 4300 watt. This when compared to an expanded CLOS solution, as referred above will consume (60 * 350Watt + 2 * 460 Watt = 21920 Watt), with above reference products; represents an 80% reduction in power.

OcNOS powering the switching silicon in the ORCA product sets up a flexible layer2 fabric with high redundancy. OcNOS which has been using industry proven networking software, provided operator class network instrumentation and operational features.

Besides offering a standard industry standard like CLI it offers NETCONF for remote NMS discovery and management. Secondary features such as HQOS allows for fine grained application traffic control. Extensive ACL’s and CoPP features can be used to block unwanted traffic.

Aparna Systems OMS (Orca uCloud Management System) provides single pane web interface to view the current status of the uServers present in the system, chassis configuration information in terms of the number of switches installed, network port status and all information of all FRUs installed in the Orca uCloud system.  OMS is implemented using modern web technologies and the API infrastructure provides a convenient mechanism to be integrated into management and orchestration systems such as VMWare vRealize Operations Manager via REST APIs and Python wrapper function APIs both at server layer and networking switch layer.  These REST APIs fall into several categories such as FRUs – servers, switches, chassis, ports (trunk and server), management access, user profiles and events.

A sample view of the chassis configuration view is given below:

Chassis Configuration:

 

 

OcNOS communicates using multiple transport technologies like L3, VxLAN overlay as well as MPLS.

On the fabric side OcNOS is the only OCP qualified NOS supporting L3 CLOS, VxLAN and TRILL based designs.

 

(*- as per data available from referred sources during time of publication)