Sidebar

Data Models

ARF PDF MP4 ZIP

Home » Courses » Building Network Automation Solutions » Data Models

You have to be registered for this online course, or have this course selected as part of an active Expert ipSpace.net Subscription to access all course materials.

Data Models

The third module of the network automation online course describes data models and data stores. We'll start with an introduction, figure out what NETCONF, YANG and OpenConfig are, and finish with a data model abstraction case study.

More information...

1:53:10 Data Models and Data Model Transformation

This section introduces data models and helps you answer these questions:

  • How should I structure the data I need to describe devices in my network?
  • Should I describe devices or should I focus on network topology and services?
  • Can I use device-level data to create device configurations while presenting high-level data models to the operators?
  • How could I transform abstracted (high-level) network- and services data models into device-level data models?
  • How could I integrate data model transformation into my automation workflow?
Data Model Introduction 17:55 2021-04-21
Data Model Introduction QA 8:53 2021-04-21
Device-Level Data Models 15:33 2021-04-21
Optimizing a Device-Level Data Model 17:53 2021-04-21
Infrastructure and Services Data Models 11:40 2021-04-21
Data Model Transformations 17:45 2021-04-21
Data Model Transformation in Automation Workflow 23:31 2021-08-28
Slide Deck 3.3M 2021-01-20

Further Reading – Data Models

Network Automation Data Model Optimization
Data Model Transformation Concepts
Data Transformation Example (Jinja2 / Ansible / Makefile)
Complex Data Transformation Example: Lab Topology Building Tool
Source Code for the Lab Topology Building Tool

Further Reading – Data Transformation

Data Model Transformations in Network Automation Solutions
Worth Reading: Data Manipulation in Jinja2

1:20:31 Formatting, Describing, and Storing Data

After you built your data model, you have to create data structures (device, link, network, service, or customer data) in a format readable by humans and machines, describe the data model for documentation and validation purposes, and store the data somewhere.

This section dives deep into presentation formats (XML, JSON, YAML), data description languages (including YANG), and data stores, from text files and Excel to NoSQL databases.

Data Representation 18:56 2021-08-28
Data Model Descriptions 15:32 2021-08-28
YANG and OpenConfig 13:17 2021-08-28
Data Stores 18:56 2021-08-28
Selecting a Data Store 13:50 2021-08-28

Further Reading – Data Representation

Beware XML-to-JSON Information Loss (Junos with Ansible)
XML-to-JSON Information Loss, Cisco Nexus OS Edition
Fixing XML-to-JSON Conversion Challenges

Further Reading – Data Validation

Data Validation with JSON Schema
Simple Data Validation with YANG Using yanglint
Interesting Tool: Schema Enforcer
New Ansible Data Validation Module(s)
Automation Should Prevent Operator Errors
Validating Data in GitOps-Based Automation
Automation Solution: Testing Data Models

Further Reading – Data Stores

Text Files or Relational Database?
Using YAML Instead of Excel in Network Automation Solutions
Growing Beyond Ansible host_vars and group_vars
What’s the Big Deal with Validation?

Other Interesting Blog Posts

What’s the Big Deal with Validation?
What Is Continuous Integration?
Continuous Integration in Network Automation
Firewall Ruleset Automation with CI Pipeline
From Excel to Network Infrastructure as Code with Carl Buchmann

Sample YANG Data Models

YANG Module Classification (RFC 8199)
Service models explained (RFC 8309)
Customer-focused YANG model for L3VPN service delivery
Network-centric L3VPN YANG Model
YANG model for L2VPN service delivery

37:05 Single Source of Truth

After mastering the data model-, model transformation-, and data store concepts, you're ready for one of the most important network automation topics: single source of truth.

Network Automation Source-of-Truth 17:00 2021-12-17
Building a Source-of-Truth 20:05 2021-12-17
Slide Deck 783K 2021-11-01
Building the Network Automation Source of Truth
Building Network Automation Source-of-Truth (Part 2)
Creating Automation Source-of-Truth from Device Configurations
Device Configurations Are Not a Good Source of Truth
Read Network Device Information with REST API and Store It Into a Database
Building Automation Device Inventory with Open Source Tools
Big Red Button for Network Automation

Further Reading

Git as a Source of Truth for Network Automation
Hierarchical Configuration Up and Running

1:19:59 NETCONF, YANG and OpenConfig

The guest presentation by Marcel Wiget (Juniper) describes YANG data models, briefly introduces NETCONF, and shows how you can use OpenConfig or your custom data model (described in YANG) to configure Junos devices.

Vendor-Neutral YANG Data Models 1.8M 2017-01-20
Introduction to NETCONF and YANG 26:27 2017-01-27
OpenConfig and Custom YANG Data Models 14:24 2017-01-27
Using YANG Data Models on Junos 39:08 2017-01-27

58:07 Using YANG Data Models on Traditional Network Devices with napalm-yang

While configuring network devices with OpenConfig sounds really interesting, don't expect to be able to do it any time soon - the vendors are (as usual) very reluctant to add yet another layer of indirection on top of their already-bloated software.

David Barroso decided to fix that problem within NAPALM, and wrote a set of libraries (napalm-yang) that perform translation between OpenConfig (or any other) YANG data model and device configurations.

What Are YANG and OpenConfig 15:32 2017-10-13
napalm-yang Basics 10:29 2017-10-13
Advanced napalm-yang Features 10:01 2017-10-13
napalm-yang Integration with Ansible 22:05 2017-10-13
Slide deck: OpenConfig with NAPALM 2.7M 2017-10-09

Understanding Ansible

You won’t be able to follow the rest of this module without a thorough understanding of Ansible. If you haven’t completed the Ansible for Networking Engineers online course, please do it now, and focus on the data modeling part of DMVPN case study.

42:29 Abstract Everything

In this part of the Network Automation Use Cases webinar David Barroso walks you through layers of network automation abstraction: from abstracting vendor-specific APIs with NAPALM to abstracting device configuration differences, and finally abstracting device state into infrastructure data model.

Template Multi-Vendor Deployments 8:52 2017-07-21
Abstract Vendor API 8:42 2017-07-21
Automate Everything 6:35 2017-07-21
Abstract Everything 18:20 2017-07-21

Additional resources

Slide deck 4.1M 2016-11-18
Source code for examples used in this section

1:28:48 Intent-Based Networking

Most of the intent-based systems are nothing more than a fancy orchestration system with an abstraction layer. This section describes the many levels of abstraction you can implement in such a system.

What Is Intent-Based Networking 19:00 2022-04-14
Device Configuration as Intent 7:29 2022-04-14
Abstracting and Managing Device-Level Intent 10:18 2022-04-14
Replacing Intent Definition 6:38 2022-04-14
Network-Wide Intent 6:06 2022-04-14
Automated Remediation 12:42 2022-04-14
Back to Reality 16:29 2022-04-14
Questions and Answers 10:06 2022-04-14

Additional resources

Slide deck 2.3M 2022-03-16
Intent-Based Networking-related Blog Posts

Further Reading

Intent-Based Networking Taxonomy
The Maslow’s Pyramid of Needs of Intent-Based Networking by Benoît Claise
How Facebook Learned to Stop Worrying and Love the Network
Circular Dependencies Considered Harmful (World-Wide Facebook Outage)
Automation Solution: Deploy BGP Routing with YANG Data Models
Using Flow Tracking to Build Firewall Rulesets... and the Halting Problem
The Law of Leaky Abstractions (Joel on Software)
The ABC of Vendor Lock-In
Shift Network Spend From Premium Products to Premium People

Service-Oriented IETF Data Models

Service models explained (RFC 8309)
YANG model for L3VPN service delivery
YANG model for L2VPN service delivery

34:02 Transforming Data Models with Ansible

Orchestration systems with an abstraction layer usually use network- and services data models to describe the desired system functionality, and device (or node) data models to describe the target state of the system.

In every such system, someone has to perform the mapping between the high-level and low-level data models, and this section describes how you can do that with Ansible.

Transforming Data Models with Ansible 17:28 2017-10-13
Data Models - Discussion Questions 16:34 2017-10-13
Slide Deck: Transforming Data Models with Ansible 2.3M 2017-10-20

1:23:27 Free items Data Center Fabrics Data Model

In another network automation use case Dinesh Dutt talked about automated data center fabric deployments. Focus on how simple his data model is and how he builds a whole fabric from just a few configurable parameters.

Challenges of Data Center Fabric Deployments 20:02 2017-01-24
Separate Data from Code 12:56 2017-01-24
Ansible Tips and Tricks 9:19 2017-01-24
Validation 12:51 2017-01-24
Staging and Production 11:10 2017-01-24
Fabric Deployment Demo 17:09 2017-01-24

Additional resources

Slide deck 1.6M 2017-01-24
Source code for examples used in this section

Hands-on exercises

In the hands-on exercises you’ll build a data model for your network automation solution and store it in one or more YAML files.

Homework: Create an infrastructure or services data model
Submit your homework
Overview: Submitting Hands-On Exercise Solutions

Useful Tips and Tricks

Data (de)Duplication in Network Automation Data Models
Data Munging with Ansible and Jinja2 7.5K 2018-12-24

Optional Materials

1:18:10 Manipulating Network Information with Python

Hans Verkerk guided us you through some easy-to-understand scripts written in Python. These scripts abstract Cisco IOS configurations into Python data structures, enabling you to query the data gathered from network devices using standard Python utilities in a simple but effective way.

Storing Information into Python Data Structures 32:18 2020-02-23
Parsing Device Configuration Files 10:17 2020-02-23
Validating Campus Network Configurations 12:59 2020-02-23
Improved Device Configuration Parsing 5:10 2020-02-23
Migration Example 17:26 2020-02-23
Slide Deck 1.7M 2019-10-25

Additional Informations

Data Manipulation with Python Built-In Superheroes
Source code for live demos

Additional resources

Want to know even more about data models in network automation? You might enjoy these webinars and presentations.

NETCONF and YANG 4:25:00

If you want to know more about NETCONF, watch at least the first part of NETCONF and YANG webinar with David Gee (~ 1 hour if you skip the hands-on part).

You might also want to watch the NETCONF Deep Dive section to understand the NETCONF protocol details.

The Single Source of Truth for Automatic Network Configuration by Andy Davidson
Slides for "The Single Source of Truth for Automatic Network Configuration" presentation
%arc%
%arc%
%arc%
You started this section on %started% Mark completed