Designing and Building Data Center Fabrics


Home » Courses » Data Center Infrastructure » Designing and Building Data Center Fabrics

You have to be registered for this online course, or have this course selected as part of an active Expert Subscription to access all course materials.

Designing and Building Data Center Fabrics

Introduction to Data Center Fabrics

54:02 What is a Data Center Fabric

Before starting the Designing and Building Data Center Fabrics online course, let’s try to answer a seemingly simple question “what is a data center fabric?”

What Is a Data Center Fabric 16:33 2016-05-19
Market Overview 18:16 2019-01-03
Choosing a Solution 12:39 2019-01-03
Keep It Simple 6:34 2018-11-22

Test your knowledge

Review Questions 152K 2017-10-09

1:22:50 Data Center Fabric Requirements

What services should your data center fabric provide? What requirements should you consider when designing it? This section lists a number of potential requirements and explains why they might be relevant to your design.

Optimal Bandwidth Utilization 15:59 2016-05-19
Redundant Edge Connectivity 4:14 2016-05-19
Layer-2 Transport and VM Mobility 9:03 2016-05-19
Enhanced Layer-2 Transport 11:22 2016-05-19
Optimal Layer-3 Forwarding 10:50 2016-05-19
Multiple Routing Domains 5:35 2016-05-19
Storage Integration 7:20 2016-05-19
Lossless Transport and Data Center Bridging 4:46 2016-05-19
Table Sizes and Packet Buffers 13:41 2016-05-19

Slide Deck

Data Center Fabrics - Requirements 9.6M 2016-05-22

Test your knowledge

Review Questions 186K 2017-10-09

3:54:41 Free items Typical Fabric Technologies Overview

Data center fabrics often rely on technologies you might not find anywhere else in your network. This section describes those technologies and their impact on fabric design.

3:03:22 Networking technologies

Some of the technologies used in data center fabrics are standardized. Others are not, but they’re still used by most vendors in one form or another.

This section describes some technologies you’ll find in most data center fabric solutions, including Multi-Chassis Link Aggregation (MLAG), Unified Forwarding Tables (UFT), chassis architectures, ASIC buffering, and 400 Gigabit Ethernet.

MLAG Technology Overview 17:28 2016-06-22
UFT Technology Overview 13:00 2017-08-02
Speeds and Breakout Options 6:42 2019-01-03

In March 2020, Mark Nowell did a deep dive into 400 GbE technology, optics, and form factors

In April 2020, Pete Lumbis described ASIC architectures and behavior from software engineer's perspective.

51:19 Storage technologies and lossless transport

Read this first 963 2022-12-12
Storage Protocols 101 11:12 2011-04-06
Fibre Channel over Ethernet 14:10 2011-04-06
iSCSI 6:26 2011-04-06
Long-Distance Storage 3:06 2011-04-06
Storage Myths 4:27 2011-04-06
Storage Q&A 11:58 2011-04-06

Test your knowledge

Review Questions 177K 2017-11-06

Leaf-and-Spine Fabrics

1:24:47 Introduction to leaf-and-spine fabrics

Traditional data center networks used a 3-tier design that was mostly mandated by hardware limitations, resulting in unequal bandwidth between endpoints based on their locations. In the last few years the networking industry rediscovered the work of Charles Clos (from 1953) and everyone started promoting leaf-and-spine fabrics.

Challenges of Traditional Data Center Networks 16:04 2017-11-05
Clos Networks and Leaf-and-Spine Fabrics 16:39 2017-11-05

Additional Resources

Slide deck 1.3M 2017-11-04

52:04 From the Design Clinic

Leaf-and-Spine Fabrics Outside of Data Centers 22:59 2021-12-27
Integrating Storage in Leaf-and-Spine Fabrics 22:32 2022-04-02
Migrating to a Leaf-and-Spine Fabric 6:33 2022-04-02

Further Reading

Demystifying DCN Topologies: Clos Networks
Demystifying DCN Topologies: Fat trees and leaf-and-spine fabrics

Test your knowledge

Review Questions 166K 2017-11-04

45:54 Physical Leaf-and-Spine Fabric Design

After mastering the basic principles of leaf-and-spine fabrics described in the Introduction section we're moving on to the physical design: how do you build a leaf-and-spine fabric given number of edge ports and oversubscription ratio? What if you need less than 100 ports? What if you need 50.000 ports? What do you do if you have to support low-speed edge interfaces?

The examples in the following videos use switches with 10GE/40GE ports. Don't be bothered by that; the same considerations and design calculations apply to fabrics built with higher-speed switches, for example switches with 100GE server ports and 400GE uplinks.

Physical Leaf-and-Spine Fabric Design 14:55 2017-11-05
Small Fabrics and Lower-Speed Interfaces 9:07 2017-11-05
Building Very Large Fabrics 21:52 2017-11-05
Physical Leaf-and-Spine Fabric Design 1.7M 2017-11-04

Test your knowledge

Review Questions 172K 2017-11-04

Design Assignment

Build a Small Data Center Fabric 124K 2017-12-30

2:39:15 Free items Layer-3 Fabrics with Non-Redundant Server Connectivity

We're starting the design part of the webinar with the simplest possible scenario – each leaf switch is a single IP subnet – and focus on routing protocol selection, route summarization, leaf-to-spine link aggregation, and core link addressing.

The second half of this section describes detailed guidelines on using BGP and OSPF as the underlay routing protocols in a leaf-and-spine fabric.

44:27 Overview and Design Principles

Introduction to Leaf-and-Spine Designs 4:39 2017-02-10
Layer-3 Fabric with Non-Redundant Server Connectivity 6:52 2017-02-10
Routing Protocol Selection 19:50 2017-02-10
Route Summarization and Link Aggregation 6:40 2017-02-10
Core Link Addressing 6:26 2017-02-10
Slide deck 1.4M 2016-03-25

1:04:14 Using BGP in Leaf-and-Spine Fabrics

Based on the work done by Petr Lapukhov at Microsoft, every vendor talks about using BGP as the routing protocol in leaf-and-spine fabrics. Does it make sense? You'll find some of the answers in this section presented by Dinesh Dutt (Cumulus Networks).

Using BGP in Leaf-and-Spine Fabrics 10:19 2017-02-10
Simplifying BGP Configurations 19:30 2017-02-10
Troubleshooting and Managing BGP 8:19 2017-02-10
BGP in Data Centers - Sample Deployments 3:15 2017-02-10
BGP in 3-tier Clos 8:45 2021-04-03
BGP QA 14:06 2021-04-03
Slide Deck: Operationalizing BGP in the Data Center 1.9M 2016-03-04
Slide Deck: BGP in 3-Tier Clos Topology 890K 2021-03-13

50:34 Using OSPF in Leaf-and-Spine Fabrics

While everyone talks about using BGP or yet-to-be-implemented routing protocols in leaf-and-spine fabrics, OSPFv2 works just fine in some of the biggest fabrics in the world. This section describes some of the design guidelines you should use when deploying OSPF as a fabric IGP.

Why Would You Use OSPF 16:01 2021-04-03
OSPF Design 20:02 2021-04-03
Configuration Snippets 10:04 2021-04-03
OSPF Footnotes 4:27 2021-04-03
OSPF in Clos Topology 1.2M 2021-03-13

Test your knowledge

Review Questions 202K 2017-11-06

1:06:44 Layer-3 Fabrics with Redundant Server Connectivity

After establishing the baseline in the Layer-3 fabrics with non-redundant server connectivity section we'll add complexity at the fabric edge: redundantly connected servers.

Layer-3 Fabrics with Redundant Server Connectivity 18:54 2016-12-13
Link Aggregation between Servers and Network 5:26 2016-12-13
Active-Standby Server Connectivity 8:18 2016-12-13
Slide deck 1.7M 2018-01-17

34:06 From the Design Clinic

Multi-Homed Servers 34:06 2022-01-18

Test your knowledge

Review Questions 165K 2017-12-31

1:50:15 Free items Layer-3-Only Data Centers

Is it possible to build a pure layer-3 data center fabric that supports redundant server connectivity and IP address mobility? You'll find out in this section.

6:53 Design Guidelines

Host Routing 6:53 2016-12-13
Slide deck 1.3M 2016-03-25

24:22 Building a Pure L3 Data Center with Cumulus Linux

Building a Pure L3 Data Center with Cumulus Linux 24:22 2016-12-13
Slide deck 949K 2016-03-29

Test your knowledge

Review Questions 157K 2018-01-17

1:19:00 More to explore

Layer-3-only Data Center Networks with Cumulus Linux on Software Gone Wild
Enterasys Robust Data Center Interconnect Solutions 1:19:00

36:40 Free items Routing on Servers

Another approach to building a pure layer-3 fabric is to extend the fabric routing protocol into the servers and announce servers' loopback IP addresses using BGP.

Runinng Routing Protocols on Servers 10:55 2016-12-13
Routing from Hosts - Deep Dive 10:24 2016-12-13
Examples from Real World 8:08 2016-12-13

7:13 From the Design Clinic

VXLAN and EVPN on Linux Hosts 7:13 2022-01-18

Test your knowledge

Review Questions 153K 2017-12-31

1:53:15 Free items Layer-2 Fabrics

We're leaving the stable world of L3-only fabrics and entering the realm of large VLANs that most enterprise data centers have to deal with. We'll cover numerous design scenarios, from traditional bridging to routing on layer 2 and MAC-over-IP encapsulation.

1:00:15 Design Guidelines

Layer-2 Fabrics 14:49 2017-03-22
Traditional Bridging 10:05 2017-03-22
Routing on Layer-2 13:12 2017-03-22
MAC-over-IP Encapsulation 13:25 2017-03-22
Redundant Server-to-Network Connectivity 8:44 2017-03-22
Slide deck 1.8M 2016-04-01

53:00 Shortest Path Bridging in Avaya Fabric

Avaya is one of the few data center switching vendors that still uses routing on layer 2 (SPB) technology instead of VXLAN encapsulation. In this guest presentation Roger Lapuh (Avaya) explains how SPB works and how you can use it to build layer-2 or layer-2+3 data center fabrics.

Introduction to SPB and Avaya Fabric Connect 18:25 2017-03-22
SPB Deep Dive 18:17 2017-03-22
Building Data Center Fabrics with SPB 16:18 2017-03-22
Slide deck 2.1M 2016-04-06

Test your knowledge

Review Questions 179K 2017-12-31

Design Assignment

Design a Robust Layer-2 Fabric 236K 2017-12-30

1:53:31 Free items Mixed Layer-2 + Layer-3 Fabrics

Most data center fabrics have to combine elements of large VLANs and routing. In this section we'll explore the various combinations, from traditional routing on spine switches to anycast routing on leaf switches.

31:05 Design Guidelines

Layer-2+3 Fabrics 6:45 2017-04-05
Routing on Spine Switches 9:04 2017-04-05
Routing on Leaf Switches 15:16 2017-04-05
Slide deck 1.4M 2016-04-20

1:18:52 VXLAN with BGP EVPN on Cisco Nexus OS

Major data center switching vendors use VXLAN to build large layer-2 domains across IP fabrics, and EVPN control plane to build flooding trees and exchange MAC address reachability information.

In this section Lukas Krattiger (guest speaker from Cisco Systems) explains how VXLAN transport and EVPN control plane work on Nexus switches.

Overlays in Data Center Fabrics 15:07 2017-04-05
Overview of VXLAN with BGP EVPN 15:59 2017-04-05
Introduction to BGP EVPN 15:29 2017-04-05
BGP EVPN Deep Dive 15:39 2017-04-05
EVPN Integrated Routing and Bridging 16:38 2017-04-05
Slide deck 12M 2016-04-21

3:34 From the Design Clinic

VXLAN and EVPN in Small Data Center 3:34 2022-04-02

Test your knowledge

Review Questions 169K 2017-12-31

2:16:42 Free items Stretching Leaf-and-Spine Fabrics

Should you stretch a single fabric across multiple sites? Does it make sense to split a large fabric into smaller fabrics (pods)? What could you do to improve the scalability of VXLAN-based EVPN fabrics?

This section contains the design guidelines and technology details you need to answer these questions.

44:42 Multi-Site and Multi-Pod Fabrics Design Guidelines

What Problem Are We Trying to Solve? 13:38 2019-01-28
Physical Multi-Site Topologies 11:54 2019-01-28
Data-, Control- and Management-Plane Failure Domains 17:05 2019-01-28
Conclusions 2:05 2019-01-28
Slide Deck 2.1M 2018-03-29
Open-Source Hybrid Cloud Reference Architecture
Data Center Interconnects 4:31:00

Focused exclusively on DCI issues, this webinar describes numerous designs and technologies (including local and global load balancers, VPLS, OTV and MPLS/VPN) that can be used to build active-active data centers. This webinar expands on the concepts described in the Data Center 3.0 for Networking Engineers webinar, which is thus a highly recommended prerequisite.

Designing Active-Active and Disaster Recovery Data Centers 3:37:00

This webinar covers typical design scenarios encountered when building a disaster recovery data center or deploying multiple data centers in an active-active configuration.

VMware NSX, Cisco ACI or Standard-Based EVPN 6:30:00

Are you building a new data center fabric? Trying to decide whether to use VMware NSX or Cisco ACI? How about EVPN? This fast-paced webinar will help you select the best option for your environment.

For more details, please watch these deep dive webinars:

1:32:00 Using VXLAN and EVPN in Multi-Pod and Multi-Site Fabrics

Introduction to Multi-Pod and Multi-Site Fabrics 6:58 2019-01-28
Multi-Pod Fabrics 27:04 2019-01-28
Multi-Site Fabrics 33:19 2019-01-28
Multi-Site Packet Forwarding 19:10 2019-01-28
Conclusions 5:29 2019-01-28
Slide Deck 29M 2018-03-27
EVPN Technical Deep Dive 11:49:00

This webinar will help you grasp the EVPN fundamentals, potential use cases, and its benefits and drawbacks. The deep dive part of the webinar discussed technical details that will enable you to deploy and troubleshoot EVPN in multi-vendor environments.

VXLAN Technical Deep Dive 3:42:00

VXLAN is a novel MAC-over-IP technology used to implement large-scale layer-2 multi-tenant virtual networking solutions within the VMware's vSphere ecosystem. This webinar will give you the architectural and technical details you need to design and deploy VXLAN-based virtual networks. It assumes familiarity with virtual networking concepts and VMware networking solutions.

Test your knowledge

Review Questions 195K 2018-12-17

Design Assignment

Extend a Leaf-and-Spine Fabric 105K 2017-12-30

19:31 QoS and Load Balancing in Leaf-and-Spine Fabrics

Do you need QoS in data center fabrics? How do you handle load balancing between elephant and mice flows? This section describes some of the answers.

The following video was recorded in the Autumn 2016 Q&A session of the Building Next-Generation Data Center online course

QoS and Load Balancing in Leaf-and-Spine Fabrics 19:31 2017-11-06

Advanced Topics

1:14:48 Whitebox Switching and Disaggregation

This section describes the differences between SDN, disaggregation and whitebox switching, the major components you'll find in any disaggregated solution (from hardware abstraction layer to control-plane components) and the benefits and drawbacks of several disaggregated network operating systems.

Whitebox and Disaggregation Drivers 13:38 2017-09-17
Components 18:19 2017-09-17
Network Operating Systems 19:28 2017-09-17
Challenges 13:12 2017-09-17
Questions 10:11 2017-09-17

Additional resources

Slide deck 1.2M 2017-05-09

Test your knowledge

Review Questions 167K 2017-12-30

1:06:50 Large-Scale Open Data Center Fabrics

In this section you'll learn how to build very large data center fabrics. The section covers:

  • planning and design
  • disaggregation of chassis switches into individual components
  • challenges and limitations of modern data center switches
  • pod architecture
  • sample single-SKU designs
Large Data Center Fabric Challenges 16:54 2017-11-20
Pod Architecture and Basic Access Topologies 6:39 2017-11-20
Single-SKU Data Center Networks 9:44 2017-11-20
Hardware Challenges 14:48 2017-11-20
Building Large Leaf-and-Spine Fabrics 18:45 2017-11-20

Additional resources

Slide deck 4.5M 2017-05-23

Test your knowledge

Review Questions 170K 2017-12-30

56:19 Data Center Fabric Architectures

As you analyze the vendor solutions, it quickly becomes obvious that every single one of them uses one of few typical control/management plane architectures, from completely decentralized model (sometimes heavily automated) to centralized control- or even data plane.

This section describes most of the architectures you might encounter in real life and documents their benefits and drawbacks.

Fabric Architectures - Introduction 15:08 2016-06-22
Independent Devices 4:42 2016-06-22
Shared Management Plane 7:49 2016-06-22
Centralized Control Plane 13:13 2016-06-22
Centralized Control Plane with Local Offload 6:37 2016-06-22
Centralized Data Plane 6:17 2016-06-22
Conclusions 2:33 2016-06-22

Slide Deck

Data Center Fabrics - Architectures 2.3M 2016-05-30

Test your knowledge

Review Questions 198K 2017-12-30

1:11:40 Free items Fabrics, Buffers and Drops

Buffering is a confusing and sometimes controversial topic when it comes to data center fabrics. In this short section JR Rivers discusses some of the related theory and practicalities as well as gives insight into tools that are available to networking specialists.

52:20 Switch Buffering Deep Dive

Switch Buffer Architectures 28:54 2017-09-16
Small Buffer or Big Buffer Switches 5:36 2017-09-16
Tools and Knobs 17:50 2017-09-16

19:20 Questions and Answers

Questions and Answers 19:20 2017-09-16

Reference material

Slide deck 2.5M 2016-11-09

Test your knowledge

Review Questions 192K 2017-12-30

Reference material

5:36:01 Reference: Data Center Fabric solutions

In this section you’ll find an overview of data center fabric solutions from major data center switching vendors. The Data Center Fabrics webinar (included with this course) contains even more information.

2:12:55 Arista EOS

This section contains information on Arista’s data center switches and software features available in Arista EOS.

Product Overview - Arista 7:26 2019-01-03
Product Details - Arista 10:26 2017-08-18
Software Features Release Overview - Arista 28:42 2017-08-18
Highlighted Software Features - Arista EOS 45:26 2017-08-18
Highlighted Software Features - Arista 7150 8:18 2014-07-22
OpenFlow Support - Arista 13:36 2015-12-22
VXLAN Support - Arista 19:01 2017-08-18

1:45:03 Cisco Nexus Switches

This section covers Cisco’s Nexus series of data center switches and their Nexus-OS operating system.

Product Overview - Cisco Nexus Switches 9:00 2019-01-03
Product Details - Cisco 3:35 2014-09-12
FabricPath, vPC and DFA 17:37 2016-01-04
Software Release Overview - Nexus OS 34:25 2017-08-21
Highlighted Software Features - Nexus OS 19:45 2017-08-21
OpenFlow Support in Nexus Switches 4:20 2017-08-21
VXLAN Support in Nexus Switches 16:21 2017-08-21

1:38:03 Juniper QFX- and EX Series

This section covers Juniper’s QFX- and EX-series of data center switches and the data center features of Junos.

Product Overview - Juniper Data Center Switches 6:04 2019-01-03
Product Details - QFX- and EX-Series Switches 9:57 2016-01-04
Product Details - QFabric 8:08 2014-10-17
Virtual Chassis 10:30 2014-10-17
Junos Software Release Overview 19:42 2018-11-23
Highlighted Software Features 23:45 2018-11-23
OpenFlow Support on Juniper Data Center Switches 2:54 2016-01-04
VXLAN Support on Juniper Data Center Switches 17:03 2018-11-23

51:19:00 Relevant Webinars

All webinars in this section are included with your online course. Some of them are part of the core course content, others are included as reference material.

Data Center Fabric Architectures 20:27:00
Leaf-and-Spine Fabric Architectures 13:07:00
Data Center Infrastructure for Networking Engineers 10:40:00
Open Networking for Large-Scale Networks 2:22:00
Networks, Buffers, and Drops 1:12:00
ExpertExpress Case Studies 51:19:00
Introduction to Virtualized Networking 3:31:00
You started this section on %started% Mark completed