Validation, Error Handling and Testing


Home » Courses » Building Network Automation Solutions » Validation, Error Handling and Testing

You have to be registered for this online course, or have this course selected as part of an active Expert Subscription to access all course materials.

Validation, Error Handling and Testing

The fifth module of the network automation online course covers testing, debugging, error handling, and continuous integration.

More information...

Understanding Ansible

Have you completed the Ansible Deep Dive module of the Ansible for Networking Engineers online course? If haven't, please do it now, or you'll have problems understanding the material discussed in this module.

2:53:16 Validation, Error Handling and Unit Tests

In this section you’ll learn how to:

  • Write fail-safe playbooks
  • Recover from Ansible module failures and report them
  • Use Ansible logging and debugging tools
  • Debug individual components of your Ansible playbooks (tasks, plays or modules/plugins)
  • Build and automate unit tests
  • Validate input data and device state

The "Software Testing" part of this section has been significantly improved in 2022, the rest of the materials were delivered in one of the 2017 online courses.

The Importance of Testing and Validation 18:26 2023-02-11

1:15:19 Software Testing

Software Testing Overview 14:00 2023-02-11
Software Testing with Version Control 10:04 2023-02-11
Unit Tests 13:44 2023-02-18
Unit Tests in Network Automation 14:56 2023-02-18
Example - Nexus OS VLAN List 5:33 2023-02-18
Integration and System Tests 17:02 2023-02-18

33:08 Input Data Validation

Input Data Validation 20:18 2023-03-02
Example - VLAN Validation 5:19 2023-03-02
Writing Fail-Safe Playbooks 7:31 2023-03-02

46:23 Debugging Ansible Playbooks

Debugging Ansible playbooks and Jinja2 templates and expressions could turn into an excruciating experience. The guidelines and hints you'll get in this section will make your debugging process easier and faster.

Debugging Ansible Playbooks 15:40 2017-11-07
Snapshot Ansible Variables 8:52 2017-11-07
Debugging Jinja2 Templates 9:28 2017-11-07
Common Pitfalls 12:23 2017-11-07
Slide Deck 2.5M 2017-11-08

More Information

Slide Deck 3.6M 2023-01-18
Podcast: Network Testing with Kristian Larsson and David Barroso
Podcast: Test-Driven Network Development with Michael Kashin
Validating Data in GitOps-Based Automation
Checking Network Device Configurations in a GitOps CI Pipeline

XML-to-JSON Conversion Details

Beware XML-to-JSON Information Loss (Junos with Ansible)
XML-to-JSON Information Loss, Cisco Nexus OS Edition
Fixing XML-to-JSON Conversion Challenges

Real-Life Examples

Testing Data Models by Donald Johnson

1:48:00 Network Continuous Integration

Once you have a reliable test suite, you can start implementing continuous integration (automated merging of changes to the baseline code branch), delivery (having a deliverable environment at all times) and deployment (automated deployment of the deliverable environment).

51:45 Network CI/CD Principles

This section describes CI/CD principles, their applicability in network automation, and the hidden gotchas of trying to validate network configurations before deploying them.

Continuous Integration Concepts 13:05 2023-03-02
Continuous Integration with Git 13:22 2023-03-02
Continuous Integration in Network Automation 11:13 2023-03-02
Continuous Delivery and Deployment 14:05 2023-03-02

56:15 CI/CD with GitLab

In his Network Continuous Integration presentation Pete Lumbis explained the basics of Continuous Integration and Continuous Delivery (CI/CD) concepts and demonstrated how you can use GitLab and virtual labs to test network device configurations prior to production deployment.

Network CI-CD 2.5M 2017-02-07
Change Management Today and Tomorrow 9:05 2017-02-08
Continuous Integration with GitLab 24:40 2017-02-08
NetDevOps Toolbox 22:30 2017-02-08

1:10:17 Continuous Integration with GitLab CI

GitLab CI is one of the commonly-used networking-focused continuous integration tools due to its agent-based architecture.

In this section Pete Lumbis explained how he uses GitLab CI to test his network automation scripts and device configurations.

Continuous Integration Concepts 24:03 2017-11-08
GitLab CI 22:36 2017-11-08
GitLab Demo 23:38 2017-11-08
Slide deck 837K 2017-11-04

2:21:03 Testing Network Automation Systems

Kristian Larsson described a testing methodology based on the approach his team is using to test the network automation system they're building for Deutsche Telekom Terastream project.

He started with unit and system tests, described various approaches to device mocking, explained how to do end-to-end system tests, and shown how you can use vrnetlab (an open source virtual network lab environment focused on testing use cases) in Continuous Integration pipeline.

In the second part of his presentation he described how you could test correctness of BGP configurations with docker-based BGP feeds, and concluded with a discussion of robustness and feature flags.

Testing Overview 16:02 2019-04-10
Unit and System Tests 18:46 2019-04-10
End-to-End Tests 12:36 2019-04-10
vrnetlab 24:42 2019-04-10
Using vrnetlab 27:54 2019-04-10
Testing BGP 18:23 2019-04-10
Robustness, Safety and Feature Flags 22:40 2019-04-10
Slide Deck 19M 2019-04-09

Hands-on exercises

You’ll add data validation and error handling to the service deployment playbook you created during Week 4, and create unit tests to test your new code with a variety of invalid inputs.

Homework: Add logging, testing and validation to your solution
Submit your homework
Overview: Submitting Hands-On Exercise Solutions

Optional Self-Study Materials

1:13:31 Building a Testing Pipeline

When you automate, you want to make sure that you are not automating bugs and mistakes, thereby making them spread across your environment. That is why you want to have a proper testing pipeline in place!

In his presentation, Gabriele Gerbino gives some practical examples building a simple testing pipeline by using Travis-CI and VMs/containers to make sure that you can sleep better once you hit the DEPLOY red button.

Why and What Should You Test 19:18 2018-10-18
Build the Testing Infrastructure 29:59 2018-10-18
Execute the Tests 17:45 2018-10-18
Homework Assignments 6:29 2018-10-18
Slide deck 1.7M 2018-10-19

1:11:39 Writing Scalable and Reliable Software

David Barroso explained how to design and write an application focusing on three aspects; scalability, maintainability, and reliability.

He described design principles, tools, and patterns you can use to grow your applications without compromising long-term quality, and increase confidence and speed when developing new features or changing existing ones.

In the second half of the presentation, David illustrated these principles with a simple application that generates interface- and BGP configuration from a network data model.

Introduction 3:50 2021-03-03
Developer Tools 7:01 2021-03-03
Static Typing with Mypy 12:48 2021-03-03
Coding Example 29:35 2021-03-03
Testing 15:48 2021-03-03
Summary 2:37 2021-03-03
Slide Deck 3.6M 2020-10-29

Additional resources

Explore validation with NAPALM, test-driven development and distributed testing tools. You might also be interested in tools that make your Ansible development less error-prone.

Validating deployments with NAPALM

Validating deployments with NAPALM blog post
napalm-validate section in the Ansible course
Test-Driven Network Development with Michael Kashin
Distributed On-Demand Network Testing with Matt Oswalt


YAML validator
Ansible lint (best-practices checker)
Automated Ansible playbook/rules review
Structured Git pre-commit hooks

Other interesting resources

Unit testing Ansible modules
You started this section on %started% Mark completed