# DevNet Lab 17 -- Automated testing with pyATS and Genie
[toc]
---
### Scenario
In this lab, you will explore the basics of [**pyATS**](https://developer.cisco.com/pyats/) (pronounced "py" followed by each letter individually, "A" "T" "S") and Genie. The pyATS tool is an end-to-end testing ecosystem that specializes in data-driven and reusable testing, and is designed for agile, rapid development iterations.
Genie extends and builds upon pyATS for use in a networked environment. Examples of features provided by Genie include:
- Device connectivity, parsers, and APIs
- Cisco platform-agnostic Python object models for features like OSPF and BGP
- Pool of reusable test cases
- YAML-based test-runner engine
The true benefit of pyATS is the ability to quickly and easily test the **configuration state** and **operational state** of a device for compliance with the **desired state** of the network. While there are benefits to both types of testing, this lab is designed to familiarize you with operational state testing to provide a more complete view of your network's current status.
To illustrate **operational state** testing with pyats, we need to run two routers that are OSPF protocol neighbors exchanging networks. This will allow us to test for the presence of an IPv4 network prefix on the spoke router that is advertised by the hub router.
### Objectives
After completing these lab activities, you will be able to:
- **Set Up a Lab Environment**: Configure a lab environment with two virtual routers to test OSPFv3 configurations using pyATS and Genie.
- **Implement OSPFv3 Configurations**: Apply initial OSPFv3 configurations on both hub and spoke routers to establish network connectivity, and manually verify that OSPFv3 is working properly by examining routing tables and neighbor lists.
- **Develop custom Python tests**: Create Python scripts using pyATS to automate testing of network configurations and operational status, focusing on developing reusable test cases.
- **Execute automated tests**: Use pyATS to run automated tests to ensure that network configurations match expected states and that OSPFv3 is operating properly.
## Part 1: Prepare the lab environment
In this part, you will set up the lab environment. From the Devnet VM, we need to manage two routers that exchange IPv4 and IPv6 networks.
Therefore, we will start by declaring and configuring the hypervisor resources: six switch ports and two virtual routers. Then we set the initial configuration for each OSPFv3 routing daemon.
Here is a connection scheme table for both routers:
| Router interface | tap interface mode | VLAN(s) |
|:------ |:------- |:---- |
| GigabitEthernet1 | access mode | out-of-band |
| GigabitEthernet2 | access mode | isolated VLAN |
| GigabitEthernet3 | access mode | unused |
### Step 1: Declare and configure hypervisor switch ports
We use the [**switch-conf.py**](https://gitlab.inetdoc.net/labs/startup-scripts/-/blob/main/switch-conf.py?ref_type=heads) script with a YAML declaration file to configure the three switch ports needed by the virtual router.
Here is a template YAML declaration file named `switch.yaml` that configures 3 tap interfaces per router as described in the topology diagram:
```yaml=
ovs:
switches:
- name: dsw-host
ports:
- name: tapU # <-- SPOKE ROUTER G1 MGMT
type: OVSPort
vlan_mode: access
tag: _OOB_VLAN_ID_ # <-- YOUR OUT-OF-BAND VLAN ID
- name: tapV # <-- SPOKE ROUTER G2
type: OVSPort
vlan_mode: access
tag: _VVV_ # <-- YOUR PRIVATE VLAN ID FOR OSPF
- name: tapW # <-- SPOKE ROUTER G3
type: OVSPort
vlan_mode: access
tag: 999 # <-- UNUSED VLAN ID FOR THIS LAB
- name: tapX # <-- HUB ROUTER G1 MGMT
type: OVSPort
vlan_mode: access
tag: _OOB_VLAN_ID_ # <-- YOUR OUT-OF-BAND VLAN ID
- name: tapY # <-- HUB ROUTER G2
type: OVSPort
vlan_mode: access
tag: _VVV_ # <-- YOUR PRIVATE VLAN ID FOR OSPF
- name: tapZ # <-- HUB ROUTER G3
type: OVSPort
vlan_mode: access
tag: 999 # <-- UNUSED VLAN ID FOR THIS LAB
```
:::warning
As with all the labs in this series, be sure to replace all VLAN ID placeholders with your own identifiers.
:::
Here is a sample trace of the `switch-conf.py` script execution with some tap interface configuration changes:
```bash
switch-conf.py switch.yaml
```
```bash=
----------------------------------------
Switch dsw-host exists
>> Port tapU vlan_mode is already set to access
>> Port tapU tag is already set to _OOB_VLAN_ID_
>> Port tapV vlan_mode changed to access
>> Port tapV tag changed to _VVV_
>> Port tapW vlan_mode is already set to access
>> Port tapW tag is already set to 999
>> Port tapX vlan_mode is already set to access
>> Port tapX tag is already set to _OOB_VLAN_ID_
>> Port tapY vlan_mode changed to access
>> Port tapY tag changed to _VVV_
>> Port tapZ vlan_mode is already set to access
>> Port tapZ tag is already set to 999
----------------------------------------
```
### Step 2: Declare and start the two hub-and-spoke routers
As with switch ports, we start by creating a YAML declaration file for our virtual router. Here is an example declaration file named `lab17-routers.yaml`.
```yaml=
kvm:
vms:
- vm_name: lab17-spoke
os: iosxe
master_image: c8000v-universalk9.17.16.01a.qcow2
force_copy: false
tapnumlist: [U, V, W] # <-- YOUR TAP INTERFACE NUMBERS
- vm_name: lab17-hub
os: iosxe
master_image: c8000v-universalk9.17.16.01a.qcow2
force_copy: false
tapnumlist: [X, Y, Z] # <-- YOUR TAP INTERFACE NUMBERS
```
:::warning
Again, be sure to replace the placeholders in any `tapnumlist` list of tap interface numbers.
:::
Here is an example of running the `lab-startup.py` script:
```bash
lab-startup.py lab17-routers.yaml
```
```bash=
Copying /home/etudianttest/masters/c8000v-universalk9.17.16.01a.qcow2 to lab17-spoke.qcow2...
done.
Creating OVMF_CODE.fd symlink...
Creating lab17-spoke_OVMF_VARS.fd file...
Starting lab17-spoke...
Waiting a second for TPM socket to be ready.
---
~> Router name : lab17-spoke.qcow2
~> RAM size : 16384MB
~> SPICE VDI port number : 790U
~> telnet console port number : 700U
~> mgmt G1 tap interface : tapU, access mode
~> mgmt G1 IPv6 LL address : fe80::faad:caff:fefe:_UUUU_%vlan_OOB_VLAN_ID_
~> G2 tap interface : tapV, access mode
~> G3 tap interface : tapW, access mode
lab17-spoke started!
Copying /home/etudianttest/masters/c8000v-universalk9.17.16.01a.qcow2 to lab17-hub.qcow2...
done.
Creating lab17-hub_OVMF_VARS.fd file...
Starting lab17-hub...
Waiting a second for TPM socket to be ready.
---
~> Router name : lab17-hub.qcow2
~> RAM size : 16384MB
~> SPICE VDI port number : 790X
~> telnet console port number : 700X
~> mgmt G1 tap interface : tapX, access mode
~> mgmt G1 IPv6 LL address : fe80::faad:caff:fefe:_XXXX_%vlan_OOB_VLAN_ID_
~> G2 tap interface : tapY, access mode
~> G3 tap interface : tapZ, access mode
lab17-hub started!
```
:::info
Note that the routers may take some time to boot because the Cisco IOS-XE system is multi-layered.
:::
### Step 3: Check the two virtual routers for SSH communication
In this step, we need to verify that the two routers are up and accessible via SSH before proceeding with the minimal configuration.
From the screenshot above, we can gather the Link Local IPv6 address of the two routers and test them with the `ssh` command. So we initiate SSH connections from our Devnet VM shell.
```bash
ssh -q etu@fe80::faad:caff:fefe:_UUUU_%enp0s1 exit
```
```bash=
(etu@fe80::faad:caff:fefe:_UUUU_%enp0s1) Password:
Connection to fe80::faad:caff:fefe:_UUUU_%enp0s1 closed by remote host.
```
```bash
echo $?
```
```bash=
0
```
```bash
ssh -q etu@fe80::faad:caff:fefe:_XXXX_%enp0s1 exit
```
```bash=
(etu@fe80::faad:caff:fefe:_XXXX_%enp0s1) Password:
Connection to fe80::faad:caff:fefe:_XXXX_%enp0s1 closed by remote host.
```
```bash
echo $?
```
```bash=
0
```
### Step 4: Set initial router configurations
Now that we have access to our hub-and-spoke routers, we set up the initial configurations we want to run tests against.
First, open a new SSH connection and go to the router configuration terminal:
```console
rtrXXX#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
rtrXXX(config)#
```
Then copy and paste the spoke and hub lab configuration on each router.
- Spoke router configuration
```
hostname spoke
!
ipv6 unicast-routing
!
username genieuser privilege 15 secret -genie-
!
router ospfv3 100
router-id 0.0.0.2
address-family ipv4 unicast
area 0
exit-address-family
address-family ipv6 unicast
area 0
exit-address-family
!
interface GigabitEthernet2
no shutdown
ip address 10.0.1.2 255.255.255.0
ipv6 enable
ipv6 address 2001:db8:1::2/64
ospfv3 100 ipv4 area 0
ospfv3 100 ipv6 area 0
!
interface Loopback0
ip address 10.0.2.2 255.255.255.255
ipv6 enable
ipv6 address 2001:db8:2::2/128
ospfv3 100 ipv4 area 0
ospfv3 100 ipv6 area 0
!
end
```
- Hub router configuration
```
hostname hub
!
ipv6 unicast-routing
!
username genieuser privilege 15 secret -genie-
!
router ospfv3 100
router-id 0.0.0.1
address-family ipv4 unicast
area 0
exit-address-family
address-family ipv6 unicast
area 0
exit-address-family
!
interface GigabitEthernet2
no shutdown
ip address 10.0.1.1 255.255.255.0
ipv6 enable
ipv6 address 2001:db8:1::1/64
ospfv3 100 ipv4 area 0
ospfv3 100 ipv6 area 0
!
interface Loopback0
ip address 172.16.0.1 255.255.255.255
ipv6 enable
ipv6 address 2001:db8:ac::1/128
ospfv3 100 ipv4 area 0
ospfv3 100 ipv6 area 0
!
! Lo1 is the missing entry in the OSPFv3 configuration
!
interface Loopback1
ip address 172.16.1.1 255.255.255.255
ipv6 enable
ipv6 address 2001:db8:ac:1::1/128
!
end
```
Here are the common configuration items between the two configuration files:
- identical authentication credentials with the privileged user **genieuser**
- IPv6 routing
- OSPFv3 (process ID 100) with:
- Dual-stack support for both IPv4 and IPv6 address families
- All interfaces configured in area 0
- Loopback0 interface with IPv4/IPv6 addressing
:::warning
Note that the Loopback1 interface on the hub router is not enabled for OSPFv3. This is intentional because we want a test to fail.
:::
### Step 5: View the routing tables on each router
As a final step in this part, we will manually verify that OSPFv3 is working between the hub and spoke routers by looking at the presence of the IPv4 and IPv6 networks in the routing tables.
- Spoke router tables
```
sh ip route
```
```bash=
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C 10.0.1.0/24 is directly connected, GigabitEthernet2
L 10.0.1.2/32 is directly connected, GigabitEthernet2
C 10.0.2.2/32 is directly connected, Loopback0
172.16.0.0/32 is subnetted, 1 subnets
O 172.16.0.1 [110/1] via 10.0.1.1, 00:39:38, GigabitEthernet2
```
```
sh ipv6 route
```
```bash=
IPv6 Routing Table - default - 5 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
B - BGP, R - RIP, H - NHRP, HG - NHRP registered
Hg - NHRP registration summary, HE - NHRP External, I1 - ISIS L1
I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary, D - EIGRP
EX - EIGRP external, ND - ND Default, NDp - ND Prefix, DCE - Destination
NDr - Redirect, O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1
OE2 - OSPF ext 2, ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
a - Application, m - OMP
C 2001:DB8:1::/64 [0/0]
via GigabitEthernet2, directly connected
L 2001:DB8:1::2/128 [0/0]
via GigabitEthernet2, receive
LC 2001:DB8:2::2/128 [0/0]
via Loopback0, receive
O 2001:DB8:AC::1/128 [110/1]
via FE80::FAAD:CAFF:FEFE:B, GigabitEthernet2
L FF00::/8 [0/0]
via Null0, receive
```
- Hub router tables
```
sh ip route
```
```bash=
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C 10.0.1.0/24 is directly connected, GigabitEthernet2
L 10.0.1.1/32 is directly connected, GigabitEthernet2
O 10.0.2.2/32 [110/1] via 10.0.1.2, 00:43:49, GigabitEthernet2
172.16.0.0/32 is subnetted, 2 subnets
C 172.16.0.1 is directly connected, Loopback0
C 172.16.1.1 is directly connected, Loopback1
```
```
sh ipv6 route
```
```bash=
IPv6 Routing Table - default - 6 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
B - BGP, R - RIP, H - NHRP, HG - NHRP registered
Hg - NHRP registration summary, HE - NHRP External, I1 - ISIS L1
I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary, D - EIGRP
EX - EIGRP external, ND - ND Default, NDp - ND Prefix, DCE - Destination
NDr - Redirect, O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1
OE2 - OSPF ext 2, ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
a - Application, m - OMP
C 2001:DB8:1::/64 [0/0]
via GigabitEthernet2, directly connected
L 2001:DB8:1::1/128 [0/0]
via GigabitEthernet2, receive
O 2001:DB8:2::2/128 [110/1]
via FE80::FAAD:CAFF:FEFE:8, GigabitEthernet2
LC 2001:DB8:AC::1/128 [0/0]
via Loopback0, receive
LC 2001:DB8:AC:1::1/128 [0/0]
via Loopback1, receive
L FF00::/8 [0/0]
via Null0, receive
```
One last check is done by looking at the OSPFv3 neighbor list.
```
sh ospfv3 neighbor
```
```bash=
OSPFv3 100 address-family ipv4 (router-id 0.0.0.1)
Neighbor ID Pri State Dead Time Interface ID Interface
0.0.0.2 1 FULL/DR 00:00:39 6 GigabitEthernet2
OSPFv3 100 address-family ipv6 (router-id 0.0.0.1)
Neighbor ID Pri State Dead Time Interface ID Interface
0.0.0.2 1 FULL/DR 00:00:32 6 GigabitEthernet2
```
## Part 2: Prepare the automated test environment
In this part, you will set up a new Python virtual environment dedicated to pyATS.
### Step 1: Create the working directory and Python virtual environment
1. Ensure the `$HOME/labs/lab17` directory exist and navigate to this folder
```bash
mkdir -p $HOME/labs/lab17 && cd $HOME/labs/lab17
```
2. Install the **pyats** Python virtual environement
```bash
python3 -m venv pyats
source ./pyats/bin/activate
pip3 install pyats[full]
```
3. Verify that pyATS has been successfully installed
```bash
pyats --help
```
```bash=
Usage:
pyats <command> [options]
Commands:
clean runs the provided clean file
create create scripts and libraries from template
develop Puts desired pyATS packages into development mode
diff Command to diff two snapshots saved to file or directory
dnac Command to learn DNAC features and save to file (Prototype)
learn Command to learn device features and save to file
logs command enabling log archive viewing in local browser
migrate utilities for migrating to future versions of pyATS
parse Command to parse show commands
run runs the provided script and output corresponding results.
secret utilities for working with secret strings.
shell enter Python shell, loading a pyATS testbed file and/or pickled data
undevelop Removes desired pyATS packages from development mode
validate utilities that help to validate input files
version commands related to version display and manipulation
General Options:
-h, --help Show help
Run 'pyats <command> --help' for more information on a command.
```
If we need to update `pyats` after cloning the git repository, we can run these commands:
```
pyats version check
```
```
pyats version update --yes
```
### Step 2: Create a testbed YAML file
The pyATS and Genie tools use a YAML file to know which devices to connect to and what the correct credentials are. This file is called the testbed file. Genie has built-in functionality to create the testbed file for you.
We start using the `genie` command by looking at the help options
Enter the command `genie --help` to see all the available commands. For additional help on any command, use the `<command>` parameter, as shown below, for the create command. Note that `testbed` is one of the options for the create command.
```bash
genie --help
```
```bash=
Usage:
genie <command> [options]
Commands:
create Create Testbed, parser, triggers, ...
develop Puts desired pyATS packages into development mode
diff Command to diff two snapshots saved to file or directory
dnac Command to learn DNAC features and save to file (Prototype)
learn Command to learn device features and save to file
parse Command to parse show commands
run Run Genie triggers & verifications in pyATS runtime environment
shell enter Python shell, loading a pyATS testbed file and/or pickled data
undevelop Removes desired pyATS packages from development mode
General Options:
-h, --help Show help
Run 'genie <command> --help' for more information on a command.
```
To create a new **testbed file**, we need to use the `--create` option, which has its own help section.
```bash
genie create --help
```
```bash=
Usage:
genie create <subcommand> [options]
Subcommands:
parser create a new Genie parser from template
testbed create a testbed file automatically
trigger create a new Genie trigger from template
General Options:
-h, --help Show help
-v, --verbose Give more output, additive up to 3 times.
-q, --quiet Give less output, additive up to 3 times, corresponding to WARNING, ERROR,
and CRITICAL logging levels
```
Before we can create our first testbed file, we need to perform two tasks:
- Add a secrets management section to the main `pyats` configuration file
- Collect the two router IPv4 out-of-band interface addresses
We start by adding a `[secrets]` section to the pyats configuration file:
```bash
cat << 'EOF' >> ~/.pyats/pyats.conf
[secrets]
string.representer = pyats.utils.secret_strings.FernetSecretStringRepresenter
EOF
```
Then we generate the pyats secret:
```bash
pyats secret keygen
```
```bash=
Newly generated key :
WVy...
```
Finally we append the secret to the pyats configuration file.
```bash
echo "string.key = WVy..." >> ~/.pyats/pyats.conf
```
Now that our first task is done, we can check the contents of the configuration file.
```bash
cat ~/.pyats/pyats.conf
```
```bash=
[secrets]
string.representer = pyats.utils.secret_strings.FernetSecretStringRepresenter
string.key = WVy... # <-- YOUR OWN SECRET
```
For the second task, we need to connect to the routers and get the name and out-of-band interface address.
```bash
ssh etu@fe80::faad:caff:fefe:_UUUU_%enp0s1
```
```bash=
spoke#sh ip int brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 198.18.CCC.UUU YES DHCP up up
GigabitEthernet2 10.0.1.2 YES manual up up
GigabitEthernet3 unassigned YES unset administratively down down
Loopback0 10.0.2.2 YES manual up up
```
```bash
ssh etu@fe80::faad:caff:fefe:_XXXX_%enp0s1
```
```bash=
hub#sh ip int brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 198.18.CCC.XXX YES DHCP up up
GigabitEthernet2 10.0.1.1 YES manual up up
GigabitEthernet3 unassigned YES unset administratively down down
Loopback0 172.16.0.1 YES manual up up
Loopback1 172.16.1.1 YES manual up up
```
To create your testbed YAML file, the `genie` command needs parameters.
The `--output` parameter creates the `testbed.yaml` file.
The `--encode-password` parameter will encode the passwords in the YAML file.
The `interactive` parameter means that you will be asked a series of questions.
Provide the following answers to create the `testbed.yml` file.
- Device hostname
: Must match the hostname of the device, which for this example is **rtrXXX**
- IP address
: Must match your router out-of-band interface IPv4 address
- Username
: Local username used for ssh, which is **genieuser**
- Default password
: Local username password used for ssh, which is **-genie-**
- Enable password
: Leave blank. There is no privileged password configured on the router
- Protocol
: SSH along with the key exchange group expected by the router
- OS
: OS on the router is IOS-XE
Finally, we are ready to run the `genie create` command:
```bash
genie create testbed interactive --output testbed.yaml --encode-password
```
```bash=
Start creating Testbed yaml file ...
Do all of the devices have the same username? [y/n] y
Common Username: genieuser
Do all of the devices have the same default password? [y/n] y
Common Default Password (leave blank if you want to enter on demand):
Do all of the devices have the same enable password? [y/n] y
Common Enable Password (leave blank if you want to enter on demand):
Device hostname: spoke
IP (ip, or ip:port): 198.18.CCC.UUU
Protocol (ssh, telnet, ...): ssh
OS (iosxr, iosxe, ios, nxos, linux, ...): iosxe
More devices to add ? [y/n] y
Device hostname: hub
IP (ip, or ip:port): 198.18.CCC.XXX
Protocol (ssh, telnet, ...): ssh
OS (iosxr, iosxe, ios, nxos, linux, ...): iosxe
More devices to add ? [y/n] n
Testbed file generated:
testbed.yaml
```
:::warning
The interactive menu of this Genie testbed YAML file creation asks for a password twice.
1. The first password is the password of the **genieuser** user account created when the configuration was inserted on each router: **-genie-**.
2. The second password is left blank because we are not using the enable IOS-XE level, as our genieuser already has the maximum level of privileges.
:::
Edit the `testbed.yaml` file. Make a note of your entries in the YAML file. Your SSH password is encrypted. The lines that ask for the enable password need to be **commented out** because the `genieuser` has the highest privilege level.
Here is a sample copy of the `testbed.yaml` file:
```yaml=
devices:
hub:
connections:
cli:
ip: 198.18.CCC.UUU
protocol: ssh
credentials:
default:
password: '%ENC{gAAA...Fyw==}'
username: genieuser
# enable:
# password: '%ASK{}'
os: iosxe
type: cat8k
spoke:
connections:
cli:
ip: 198.18.CCC.XXX
protocol: ssh
credentials:
default:
password: '%ENC{gAAAA...qBw==}'
username: genieuser
# enable:
# password: '%ASK{}'
os: iosxe
type: cat8k
```
### Step 3: Use Genie to parse output from the `show ip interface brief` command into JSON
:::info
Converting unstructured output data from Cisco IOS-XE into structured data is critical for operational test automation. By structuring data, organizations can streamline their test processes, reduce manual errors, and increase the reliability of network automation tasks. This transformation supports more accurate and efficient network management.
:::
Using your testbed YAML file, invoke Genie to parse the unstructured output of the `show ip interface brief` command into structured JSON.
This command includes the IOS command to be parsed (`show ip interface brief`), the YAML testbed file (`testbed.yaml`), which contains the device list.
```bash
genie parse "show ip interface brief" --testbed-file testbed.yaml --devices spoke hub
```
The results of this command are two JSON-formatted dictionaries of network interfaces: the structured data representation of the network configuration.
- Spoke router interfaces
```json=
{
"interface": {
"GigabitEthernet1": {
"interface_is_ok": "YES",
"ip_address": "198.18.54.7",
"method": "DHCP",
"protocol": "up",
"status": "up"
},
"GigabitEthernet2": {
"interface_is_ok": "YES",
"ip_address": "10.0.1.2",
"method": "manual",
"protocol": "up",
"status": "up"
},
"GigabitEthernet3": {
"interface_is_ok": "YES",
"ip_address": "unassigned",
"method": "unset",
"protocol": "down",
"status": "administratively down"
},
"Loopback0": {
"interface_is_ok": "YES",
"ip_address": "10.0.2.2",
"method": "manual",
"protocol": "up",
"status": "up"
}
}
}
```
- Hub router interfaces
```json=
{
"interface": {
"GigabitEthernet1": {
"interface_is_ok": "YES",
"ip_address": "198.18.54.10",
"method": "DHCP",
"protocol": "up",
"status": "up"
},
"GigabitEthernet2": {
"interface_is_ok": "YES",
"ip_address": "10.0.1.1",
"method": "manual",
"protocol": "up",
"status": "up"
},
"GigabitEthernet3": {
"interface_is_ok": "YES",
"ip_address": "unassigned",
"method": "unset",
"protocol": "down",
"status": "administratively down"
},
"Loopback0": {
"interface_is_ok": "YES",
"ip_address": "172.16.0.1",
"method": "manual",
"protocol": "up",
"status": "up"
},
"Loopback1": {
"interface_is_ok": "YES",
"ip_address": "172.16.1.1",
"method": "manual",
"protocol": "up",
"status": "up"
}
}
}
```
## Part 3: Create a first pyATS Python testing script
In this part, we will demonstrate how network engineers can automate the collection of device information and store it in a machine-readable format for further analysis or documentation.
### Step 1: Review Testing Fundamentals
Tests should follow a classic pattern:
1. Arrange
: Prepare your environment for the test. This task may involve collecting data, setting variables, connecting to a device, or a combination of steps.
2. Act
: Perform an action to affect the environment. For example, call a function or invoke an API.
3. Assert
: Test whether an expected condition is met. If so, the test passes. Otherwise, the test fails.
4. Cleanup
: An optional stage to perform any necessary post-test tasks.
Creating tests for network infrastructure with pyATS is no different. Before creating your first tests, review the usage of Python assertions and how they will be used for testing your network.
An assertion is used to test that a condition evaluates to true. The syntax is:
> assert statement to test, "Message to be printed if False"
When an assertion evaluates to `False`, an exception (error) of type AssertionError will be raised (created). If the assertion evaluates to `True`, no message will be printed and the testing will continue.
pyATS automatically handles exceptions of type AssertionError and interprets the exception as a test failure (FAILED). If the assertion results in True, the test will be marked as PASSED.
- Example of an assertion that evaluates to False:
```python
python -c "assert True is False, 'True does not equal False.'"
```
```python=
Traceback (most recent call last):
File "<string>", line 1, in <module>
AssertionError: True does not equal False.
```
Verify that Python exited abnormally by checking the exit code returned to the shell. Zero (0) is a successful exit status, any other value is considered a failure:
```bash
echo $?
```
```bash=
1
```
- Example of an assertion that evaluates to True:
```python
python -c "assert True is True, 'This message should never be printed'"
```
You should just see another prompt indicating that the assertion was true and therefore no exception was raised. If you want to see that the script exit code was 0 (successful), you can run the following code to see the exit code:
```bash
echo $?
```
```bash=
0
```
Now that we understand the importance of assertions in testing, we are ready for the next steps: **Arrange** and then **Act**.
### Step 2: Create a Python script to parse `show version` as the "Arrange" phase
We want a Python script using the pyATS framework to connect to a network device named **spoke** defined in the testbed YAML file. After setting up logging and loading the testbed configuration, this script connects to the device and learns the platform characteristics. The script then executes the `show version` command on the router, parses the structured output using pyATS's parsing capabilities, and saves the results as formatted JSON in a file named `show_version.json`.
Here is our first Python script code, in which you need to edit the device name.
```python=
#!/usr/bin/env python
import json
import logging
import sys
from pyats.topology import loader
# Set up logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
# Load the testbed file
testbed = loader.load("testbed.yaml")
device = testbed.devices.spoke # <-- YOUR OWN ROUTER HOSTNAME
# Connect to the device and learn the current platform attributes
device.connect(learn_tokens=True, overwrite_testbed_tokens=True)
# Open a connection to the device
device.connect()
# Parse the 'show version' command and save to file
with open("show_version.json", "w") as f:
json.dump(device.parse("show version"), f, indent=2)
```
Let's run our new pyATS Python code named `01_parse_show_version.py`:
```bash
python3 01_parse_show_version.py
```
The console output of this command is far too long to reproduce here. It shows that the Python script has learned and collected all the characteristics of the router platform. These operations are similar to the **gather_facts** phase of an Ansible playbook. This console output provides timestamps and unstructured IOS-XE data.
Here is a snippet showing the configuration tokens learned after connecting to the target device.
| Token Type | Defined in Testbed | Learned from Device | Used for this job |
|--------------|--------------------|---------------------|-------------------|
| os | iosxe | iosxe | iosxe |
| os_flavor | None | None | None |
| version | None | 17.16.1a | 17.16.1a |
| platform | cat8k | cat8k | cat8k |
| model | None | c8000v | c8000v |
| submodel | None | None | None |
| pid | None | C8000V | C8000V |
| chassis_type | None | None | None |
The Python script writes the result of the `show version` command to the `show_version.json` file. This JSON file contains the structured data from the command output.
```json=
{
"version": {
"xe_version": "17.16.01a",
"version_short": "17.16",
"platform": "Virtual XE",
"version": "17.16.1a",
"image_id": "X86_64_LINUX_IOSD-UNIVERSALK9-M",
"label": "RELEASE SOFTWARE (fc1)",
"os": "IOS-XE",
"location": "IOSXE",
"image_type": "production image",
"copyright_years": "1986-2024",
"compiled_date": "Thu 19-Dec-24 18:08",
"compiled_by": "mcpre",
"rom": "IOS-XE ROMMON",
"hostname": "rtrXXX",
"uptime": "2 days, 15 hours, 38 minutes",
"uptime_this_cp": "2 days, 15 hours, 39 minutes",
"returned_to_rom_by": "reload",
"returned_to_rom_at": "17:23:18 WEST Tue Jan 14 2025",
"system_restarted_at": "18:39:09 WEST Thu Mar 13 2025",
"system_image": "bootflash:packages.conf",
"last_reload_reason": "reload",
"license_type": "Perpetual",
"chassis": "C8000V",
"main_mem": "1889922",
"processor_type": "VXE",
"rtr_type": "C8000V",
"chassis_sn": "9J3WSJZRVVP",
"router_operating_mode": "Autonomous",
"number_of_intfs": {
"Gigabit Ethernet": "3"
},
"mem_size": {
"non-volatile configuration": "32768",
"physical": "16308192"
},
"disks": {
"bootflash:.": {
"disk_size": "28100576",
"type_of_disk": "virtual hard disk"
}
},
"curr_config_register": "0x2102"
}
}
```
## Part 4: Configuration State Testing
In this part, we will create a pyATS script to test the configuration state of OSPFv3 on our hub and spoke routers.
We focus on testing the configuration state of OSPFv3 on our hub and spoke routers using pyATS. The script `02_ospfv3_config_state_test.py` demonstrates a comprehensive approach to configuration testing, covering all three phases of the testing process:
Arrange
: The script sets up the test environment by connecting to the devices in the testbed and learning their configurations. This is done in the `CommonSetup` class and the `setup` method of the `TestOSPFv3ConfigState` class.
Act
: While there is no explicit "act" phase in configuration testing (since we don't change the device state), the script effectively "acts" by parsing and extracting the relevant OSPFv3 configuration from the learned device configuration.
Assert
: The bulk of the script focuses on assertions, verifying various aspects of the OSPFv3 configuration. This includes verifying that the OSPFv3 process is configured, validating the presence of IPv4 and IPv6 address families, and ensuring that OSPFv3 is properly configured on specific interfaces. These assertions are implemented in the test methods of the TestOSPFv3ConfigState class.
This script provides a framework for ensuring that the OSPFv3 configuration on the devices matches the expected state, covering key aspects of the protocol's setup across multiple network elements.
### Step 1: Create the configuration test script code
Here is the test script code:
```python=
#!/usr/bin/env python
import json
import logging
from genie.harness.base import Trigger
from pyats import aetest
# Logging configuration
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class CommonSetup(aetest.CommonSetup):
"""Common setup tasks for the test script"""
@aetest.subsection
def connect_to_devices(self, testbed):
"""Connect to all devices in the testbed"""
testbed.connect(log_stdout=False)
@aetest.subsection
def prepare_testcases(self, testbed):
"""Prepare testcases by generating parameters for each device"""
# Devices to test
devices = ["spoke", "hub"]
device_list = []
for device_name in devices:
if device_name in testbed.devices:
device = testbed.devices[device_name]
# Learn device configuration
device_config = device.learn("config")
# Add the device to our list
device_list.append(
{
"device_name": device_name,
"device": device,
"running_config": device_config,
}
)
# Use loop.mark with uid_keys to properly pass parameters
aetest.loop.mark(
TestOSPFv3ConfigState, device_info=device_list, uid_keys=("device_name",)
)
class TestOSPFv3ConfigState(Trigger):
"""
Test trigger for OSPFv3 configuration state
"""
@aetest.setup
def setup(self, device_info):
"""
Setup for this specific device test instance
"""
self.device_name = device_info["device_name"]
self.device = device_info["device"]
self.running_config = device_info["running_config"]
logger.info(f"Starting tests for device: {self.device_name}")
# Extract OSPFv3 configuration
expected_config_section = "router ospfv3 100"
logger.info(f"Verifying the presence of '{expected_config_section}'")
self.ospfv3_config = self.running_config.get(expected_config_section, None)
logger.info(
f"OSPFv3 Configuration for {self.device_name}:\n{json.dumps(self.ospfv3_config, indent=2)}"
)
@aetest.test
def test_ospfv3_process_configured(self):
"""
Verifies that the OSPFv3 process is configured on the device.
"""
# trunk-ignore(bandit/B101)
assert (
self.ospfv3_config is not None
), f"The OSPFv3 process is not configured on {self.device_name}"
@aetest.test
def test_ospfv3_address_families(self):
"""
Verifies that IPv4 and IPv6 address families are configured for OSPFv3.
"""
for address_family in ["ipv4", "ipv6"]:
logger.info(
f"Verifying address family {address_family} on {self.device_name}"
)
expected_config = f"address-family {address_family} unicast"
logger.info(f"Checking for the presence of '{expected_config}'")
# trunk-ignore(bandit/B101)
assert (
expected_config in self.ospfv3_config
), f"Address family {address_family} is not configured for OSPFv3 on {self.device_name}"
@aetest.test
def test_ospfv3_interfaces(self):
"""
Verifies OSPFv3 configuration on interfaces.
"""
interfaces_to_check = ["GigabitEthernet2", "Loopback0"]
for interface in interfaces_to_check:
logger.info(f"Verifying interface {interface} on {self.device_name}")
interface_config = self.running_config.get(f"interface {interface}", {})
logger.info(
f"Interface {interface} configuration on {self.device_name}:\n{json.dumps(interface_config, indent=2)}"
)
# trunk-ignore(bandit/B101)
assert (
"ospfv3 100 ipv4 area 0" in interface_config
), f"OSPFv3 IPv4 is not configured on {interface} for {self.device_name}"
# trunk-ignore(bandit/B101)
assert (
"ospfv3 100 ipv6 area 0" in interface_config
), f"OSPFv3 IPv6 is not configured on {interface} for {self.device_name}"
class CommonCleanup(aetest.CommonCleanup):
"""Common cleanup tasks for the test script"""
@aetest.subsection
def disconnect_devices(self, testbed):
"""Disconnect from all devices in the testbed"""
testbed.disconnect()
if __name__ == "__main__":
from genie.testbed import load
testbed = load("testbed.yaml")
aetest.main(testbed=testbed)
```
The script output is very verbose, but we are able to identify the list of individual tests that passed or failed.
```bash
python3 02_test_ospfv3_config.py
```
Here is a snippet of the test output without timestamps:
```bash=
AETEST-INFO: +------------------------------------------------------------------------------+
AETEST-INFO: | Detailed Results |
AETEST-INFO: +------------------------------------------------------------------------------+
INFO:pyats.aetest.reporter.default: SECTIONS/TESTCASES RESULT
AETEST-INFO: SECTIONS/TESTCASES RESULT
INFO:pyats.aetest.reporter.default:--------------------------------------------------------------------------------
AETEST-INFO: --------------------------------------------------------------------------------
INFO:pyats.aetest.reporter.default:.
|-- common_setup PASSED
| |-- connect_to_devices PASSED
| `-- prepare_testcases PASSED
|-- TestOSPFv3ConfigState[device_info={'device_name':_'spoke',_'dev... PASSED
| |-- setup PASSED
| |-- test_ospfv3_process_configured PASSED
| |-- test_ospfv3_address_families PASSED
| `-- test_ospfv3_interfaces PASSED
|-- TestOSPFv3ConfigState[device_info={'device_name':_'hub',_'devic... PASSED
| |-- setup PASSED
| |-- test_ospfv3_process_configured PASSED
| |-- test_ospfv3_address_families PASSED
| `-- test_ospfv3_interfaces PASSED
`-- common_cleanup PASSED
`-- disconnect_devices PASSED
AETEST-INFO: .
AETEST-INFO: |-- common_setup PASSED
AETEST-INFO: | |-- connect_to_devices PASSED
AETEST-INFO: | `-- prepare_testcases PASSED
AETEST-INFO: |-- TestOSPFv3ConfigState[device_info={'device_name':_'spoke',_'dev... PASSED
AETEST-INFO: | |-- setup PASSED
AETEST-INFO: | |-- test_ospfv3_process_configured PASSED
AETEST-INFO: | |-- test_ospfv3_address_families PASSED
AETEST-INFO: | `-- test_ospfv3_interfaces PASSED
AETEST-INFO: |-- TestOSPFv3ConfigState[device_info={'device_name':_'hub',_'devic... PASSED
AETEST-INFO: | |-- setup PASSED
AETEST-INFO: | |-- test_ospfv3_process_configured PASSED
AETEST-INFO: | |-- test_ospfv3_address_families PASSED
AETEST-INFO: | `-- test_ospfv3_interfaces PASSED
AETEST-INFO: `-- common_cleanup PASSED
AETEST-INFO: `-- disconnect_devices PASSED
INFO:pyats.aetest.reporter.default:+------------------------------------------------------------------------------+
| Summary |
+------------------------------------------------------------------------------+
AETEST-INFO: +------------------------------------------------------------------------------+
AETEST-INFO: | Summary |
AETEST-INFO: +------------------------------------------------------------------------------+
INFO:pyats.aetest.reporter.default: Number of ABORTED 0
INFO:pyats.aetest.reporter.default: Number of BLOCKED 0
INFO:pyats.aetest.reporter.default: Number of ERRORED 0
INFO:pyats.aetest.reporter.default: Number of FAILED 0
INFO:pyats.aetest.reporter.default: Number of PASSED 4
INFO:pyats.aetest.reporter.default: Number of PASSX 0
INFO:pyats.aetest.reporter.default: Number of SKIPPED 0
INFO:pyats.aetest.reporter.default: Total Number 4
INFO:pyats.aetest.reporter.default: Success Rate 100.0%
INFO:pyats.aetest.reporter.default:--------------------------------------------------------------------------------
AETEST-INFO: --------------------------------------------------------------------------------
```
### Step 2: Insert another configuration state test
From the previous step, we know that our configuration state test script works. However, it seems that the test for the OSPF daemon router_id is missing. So we need to add this new test to the script code.
Locate the `TestOSPFv3ConfigState` class and add a `test_ospfv3_router_id_configured` method after the `test_ospfv3_process_configured` method. Here is the code for the new method:
```python=
@aetest.test
def test_ospfv3_router_id_configured(self):
"""
Verifies that the router-id is configured for OSPFv3.
"""
logger.info(f"Verifying router-id configuration on {self.device_name}")
# Check if router-id configuration exists in the OSPFv3 config
router_id_configured = False
router_id_value = None
for config_line in self.ospfv3_config:
if config_line.startswith("router-id"):
router_id_configured = True
router_id_value = config_line.split(" ")[1]
break
logger.info(f"Router-ID on {self.device_name}: {router_id_value}")
# trunk-ignore(bandit/B101)
assert (
router_id_configured
), f"Router-ID is not configured for OSPFv3 on {self.device_name}"
# trunk-ignore(bandit/B101)
assert (
router_id_value is not None
), f"Router-ID value is not set for OSPFv3 on {self.device_name}"
```
Run the Python test script again and locate the test tree output.
```bash
python3 02_ospfv3_config_state_test.py
```
Here is the output snippet showing that our new test is being processed:
```bash=
INFO:pyats.aetest.reporter.default:.
|-- common_setup PASSED
| |-- connect_to_devices PASSED
| `-- prepare_testcases PASSED
|-- TestOSPFv3ConfigState[device_info={'device_name':_'spoke',_'dev... PASSED
| |-- setup PASSED
| |-- test_ospfv3_process_configured PASSED
| |-- test_ospfv3_router_id_configured PASSED
| |-- test_ospfv3_address_families PASSED
| `-- test_ospfv3_interfaces PASSED
|-- TestOSPFv3ConfigState[device_info={'device_name':_'hub',_'devic... PASSED
| |-- setup PASSED
| |-- test_ospfv3_process_configured PASSED
| |-- test_ospfv3_router_id_configured PASSED
| |-- test_ospfv3_address_families PASSED
| `-- test_ospfv3_interfaces PASSED
`-- common_cleanup PASSED
`-- disconnect_devices PASSED
```
## Part 5: Operational State Testing
Configuration state tests are very useful, and are common in network automation. For example, Ansible tests that a configuration stanza exists (or doesn't exist) when performing idempotence checks in a playbook run.
Configuration state testing has one important limitation: Configuration state testing does not indicate if a feature is operating as expected!
A real-world example of this condition can be demonstrated using a smartphone. At the end of the day, the battery is low and needs to be charged. You plug the phone into the charger and walk away, expecting a full battery when you return.
- The **desired state** is that the phone is charging.
- The **configured state** is that the phone is charging.
- The **operational state** is that the phone is not charging, because the state of the power supply was not tested!
It is important that you test the **operational state** of your network. A state is never achieved unless it is configured, and the configuration can change over time in ways that affect the operational state.
For this part, we will take a multi-file approach to pyATS testing, as this improves maintainability, flexibility, and scalability.
Job File
: Job files are the entry point for executing tests. They define which triggers to execute and how to configure the test environment. This separation ensures that the test logic remains independent of the execution details.
Trigger File
: Trigger files contain the actual test cases. They define what to test and how to test it. Triggers can be reused in different jobs, making them very versatile.
Data File
: Data files provide a way to parameterize tests. They allow you to specify which devices to test, which attributes to check, and other test-specific details without modifying the trigger code.
### Step 1: Create the job file
Here is the code for the job file named `03_op_state_job.py`. As mentioned above, it specifies trigger (`TestOSPFv3State`) and data files.
```python=
#!/usr/bin/env python
"""
OSPFv3 operational state test job file
"""
# Import the gRun function to run job with datafiles
from genie.harness.main import gRun
def main():
"""
Main function that runs the job using gRun
"""
# Run the job using gRun with trigger datafile
gRun(
trigger_uids=["TestOSPFv3State"],
trigger_datafile="ospfv3_op_state_datafile.yml",
subsection_datafile="subsection_datafile.yml",
loglevel="INFO",
)
if __name__ == "__main__":
main()
```
### Step 2: Create the trigger file
The Python script named `03_ospfv3_op_state_test.py` is the heart of the operational state tests. The purpose here is to evaluate the OSPF routing protocol relationship between the two hub and spoke routers. Therefore, it contains the following tests:
OSPFv3 Neighbor Relationships (test_ospfv3_neighbors)
: - Verifies that OSPFv3 neighbor relationships are established
- Checks for the presence of "FULL" state neighbors in the output
- Confirms at least one neighbor is in "FULL" state
- Applies to both hub and spoke devices
OSPFv3 Route Installation (test_ospfv3_routes)
: - For spoke device: Checks that the route to hub's Loopback0 IPv6 address exists in routing table
- For hub device: Checks that the route to spoke's Loopback0 IPv6 address exists in routing table
- Verifies routes are properly installed via OSPFv3
- Uses show ipv6 route ospf command to verify route presence
IPv4 Route Presence Check (test_missing_route_ipv4)
: - Verifies that the route to hub's Loopback1 IPv4 address is present in the spoke's routing table
- Uses `show ip route ospfv3` command to check route presence
- Test passes if the route is present, fails if it's absent
- Only applicable to the spoke device
IPv6 Route Presence Check (test_missing_route_ipv6)
: - Verifies that the route to hub's Loopback1 IPv6 address is present in the spoke's routing table
- Uses show ipv6 route ospf command to check route presence
- Test passes if the route is present, fails if it's absent
- Only applicable to the spoke device
```python=
#!/usr/bin/env python
import logging
from genie.harness.base import Trigger
from pyats import aetest
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class TestOSPFv3State(Trigger):
"""
Test trigger for OSPFv3 operational state
"""
@aetest.test
def test_ospfv3_neighbors(self, uut):
"""
Verifies that the device has established OSPFv3 neighbor relationships.
"""
neighbor_list_output = uut.execute("show ospfv3 neighbor")
logger.info(f"OSPFv3 Neighbors Output on {uut.name}:\n{neighbor_list_output}")
has_full_neighbors = "FULL" in neighbor_list_output
# trunk-ignore(bandit/B101)
assert (
has_full_neighbors
), f"No OSPFv3 neighbor with FULL state found on {uut.name}"
neighbor_count = neighbor_list_output.count("FULL")
logger.info(f"Number of OSPFv3 neighbors in FULL state: {neighbor_count}")
# trunk-ignore(bandit/B101)
assert (
neighbor_count > 0
), f"No OSPFv3 neighbor with FULL state found on {uut.name}"
@aetest.test
def test_ospfv3_routes(self, uut):
"""
Verifies OSPFv3 routes are properly installed in the routing table.
"""
try:
if uut.name == "spoke":
# Test spoke has a route to hub's Loopback0
expected_route_ipv6 = getattr(
uut, "hub_loopback0_ipv6", "2001:DB8:AC::1"
).split("/")[0]
spoke_raw_output = uut.execute("show ipv6 route ospf")
logger.info(f"IPv6 OSPF routes on {uut.name}:\n{spoke_raw_output}")
# trunk-ignore(bandit/B101)
assert (
expected_route_ipv6 in spoke_raw_output
), f"Route to hub's Loopback0 ({expected_route_ipv6}) not found on spoke"
logger.info(
f"Route to hub's Loopback0 ({expected_route_ipv6}) found on spoke"
)
elif uut.name == "hub":
# Test hub has a route to spoke's Loopback0
expected_route_ipv6 = getattr(
uut, "spoke_loopback0_ipv6", "2001:DB8:2::2"
).split("/")[0]
hub_raw_output = uut.execute("show ipv6 route ospf")
logger.info(f"IPv6 OSPF routes on {uut.name}:\n{hub_raw_output}")
# trunk-ignore(bandit/B101)
assert (
expected_route_ipv6 in hub_raw_output
), f"Route to spoke's Loopback0 ({expected_route_ipv6}) not found on hub"
logger.info(
f"Route to spoke's Loopback0 ({expected_route_ipv6}) found on hub"
)
except Exception as e:
logger.error(f"Error in test_ospfv3_routes: {e}")
self.failed(f"Test failed due to error: {e}")
@aetest.test
def test_missing_route_ipv4(self, uut):
"""
Verifies the presence of specific IPv4 routes in the OSPF routing table.
This method tests that the route to the hub's Loopback1 is present
in the spoke's IPv4 OSPF routing table.
"""
if uut.name != "spoke":
self.skipped("This test only applies to the spoke device")
return
# Use the command for IPv4
spoke_raw_output = uut.execute("show ip route ospfv3")
logger.info(f"IPv4 OSPF routes on {uut.name}:\n{spoke_raw_output}")
# Verify the presence of IPv4 route to hub's Loopback1
# Use getattr with a default value for the attribute
hub_loopback1_ipv4 = getattr(uut, "hub_loopback1_ipv4", "172.16.1.1")
# trunk-ignore(bandit/B101)
assert (
hub_loopback1_ipv4 in spoke_raw_output
), f"IPv4 route to hub's Loopback1 ({hub_loopback1_ipv4}) not found in spoke routing table when it should be present"
logger.info(
f"IPv4 route to hub's Loopback1 ({hub_loopback1_ipv4}) correctly present in spoke routing table"
)
@aetest.test
def test_missing_route_ipv6(self, uut):
"""
Verifies the presence of specific IPv6 routes in the OSPF routing table.
This method tests that the route to the hub's Loopback1 is present
in the spoke's IPv6 OSPF routing table.
"""
if uut.name != "spoke":
self.skipped("This test only applies to the spoke device")
return
# Use the command for IPv6
spoke_raw_output = uut.execute("show ipv6 route ospf")
logger.info(f"IPv6 OSPF routes on {uut.name}:\n{spoke_raw_output}")
# Verify the presence of IPv6 route to hub's Loopback1
# Use getattr with a default value for the attribute
hub_loopback1_ipv6 = getattr(uut, "hub_loopback1_ipv6", "2001:DB8:AC:1::1/128")
# Extract only the address without the prefix if needed
if "/" in hub_loopback1_ipv6:
hub_loopback1_ipv6 = hub_loopback1_ipv6.split("/")[0]
# trunk-ignore(bandit/B101)
assert (
hub_loopback1_ipv6 in spoke_raw_output
), f"IPv6 route to hub's Loopback1 ({hub_loopback1_ipv6}) not found in spoke routing table when it should be present"
logger.info(
f"IPv6 route to hub's Loopback1 ({hub_loopback1_ipv6}) correctly present in spoke routing table"
)
if __name__ == "__main__":
from genie.harness.main import gRun
from genie.testbed import load
# Load the testbed
testbed = load("testbed.yaml")
# Run with trigger datafile
gRun(
trigger_uids=["TestOSPFv3State"],
trigger_datafile="ospfv3_op_state_datafile.yml",
subsection_datafile="subsection_datafile.yml",
)
```
### Step 3: Create the data files
In teh context of this file datas are split in two files:
- `subsection_datafile.yml`
This file defines the standard setup and cleanup procedures for test execution, handling the connection to devices at the beginning of the tests and ensuring proper disconnection at the end, which helps to maintain consistent initialization and cleanup of the test environment across test runs.
```yaml=
---
setup:
sections:
connect:
method: genie.harness.commons.connect
order:
- connect
cleanup:
sections:
disconnect:
method: genie.harness.commons.disconnect
order:
- disconnect
```
- `ospfv3_op_state_datafile.yml`
This file defines the test parameters and configuration for OSPFv3 Operational State Verification, specifying the test class to run, the target devices (hub and spoke), and device-specific attributes such as process IDs and interface IP addresses required for route verification tests.
```yaml=
---
extends: ospfv3_base_datafile.yml
TestOSPFv3State:
source:
class: 03_ospfv3_op_state_test.TestOSPFv3State
groups: [test_ospfv3_op_state]
devices: [spoke, hub]
devices_attributes:
spoke:
ospfv3_process_id: 100
hub_loopback0_ipv4: 172.16.0.1
hub_loopback0_ipv6: 2001:DB8:AC::1/128
hub_loopback1_ipv4: 172.16.1.1
hub_loopback1_ipv6: 2001:DB8:AC:1::1/128
hub:
ospfv3_process_id: 100
spoke_loopback0_ipv4: 172.16.2.2
spoke_loopback0_ipv6: 2001:DB8:2::2/128
```
:::warning
The Loopback1 interface on the hub router is the one that is intentionally misconfigured. Although it is addressed, its route is not known to the spoke router because this interface has not been added to OSPF area 0. This leads to a test failure when running the pyATS job.
:::
### Step 4: Run the pyATS job
Now that all the job, trigger, and data files are in place, it is time to run the pyATS job with all the defined tests.
```bash
pyats run job 03_op_state_job.py --testbed-file testbed.yaml
```
As usual, the command output is very verbose and it would take too much space in this document to reproduce it. So here is a snippet of the task results.
```bash=
+--------------------------------------------------------------------+
| Task Result Details |
+--------------------------------------------------------------------+
Task-1: genie_testscript FAILED
|-- common_setup PASSED
| `-- connect PASSED
|-- TestOSPFv3State.spoke FAILED
| |-- test_ospfv3_neighbors PASSED
| |-- test_ospfv3_routes PASSED
| |-- test_missing_route_ipv4 FAILED
| `-- test_missing_route_ipv6 FAILED
|-- TestOSPFv3State.hub PASSED
| |-- test_ospfv3_neighbors PASSED
| |-- test_ospfv3_routes PASSED
| |-- test_missing_route_ipv4 SKIPPED
| `-- test_missing_route_ipv6 SKIPPED
`-- common_cleanup PASSED
`-- disconnect PASSED
```
As expected, the tests on the spoke router are marked as failed. Specifically, the IPv4 and IPV6 missing route tests failed. We can get a more detailed view of the logs by using the built-in web service.
```bash
pyats logs view
```
```bash=
Logfile: /home/etu/.pyats/archive/25-03/03_op_state_job.2025Mar20_08:03:47.204062.zip
View at:
http://localhost:37567/
Press Ctrl-C to exit
```
When we open the web page, we see the same view of the task results.
If we click on the title of the failed test, a right panel appears and we get the details of the failure condition.
Here is a text copy of this particular failed test mentioning that an IPv4 route is missing on the spoke router.
```bash=
440: 2025-03-20T08:04:01: +------------------------------------------------------------------------------+
441: 2025-03-20T08:04:01: | Starting section test_missing_route_ipv4 |
442: 2025-03-20T08:04:01: +------------------------------------------------------------------------------+
443: 2025-03-20T08:04:01: +++ spoke with via 'cli': executing command 'show ip route ospfv3' +++
show ip route ospfv3
spoke#
463: 2025-03-20T08:04:01: IPv4 OSPF routes on spoke:
464: 2025-03-20T08:04:01: Caught an assertion failure while executing section test_missing_route_ipv4:
465: 2025-03-20T08:04:01: Traceback (most recent call last):
466: 2025-03-20T08:04:01: File "/home/etu/labs/lab17/03_ospfv3_op_state_test.py", line 104, in test_missing_route_ipv4
467: 2025-03-20T08:04:01: hub_loopback1_ipv4 in spoke_raw_output
468: 2025-03-20T08:04:01: AssertionError: IPv4 route to hub's Loopback1 (172.16.1.1) not found in spoke routing table when it should be present
469: 2025-03-20T08:04:01: The result of section test_missing_route_ipv4 is => FAILED
```
## Part 6: Fix the hub router configuration to pass tests
Although configuring devices is not the primary goal of pyATS tests, it is possible to fix our intentionally broken configuration with a new set of job, trigger, and data files. We will create these new files in this part and then re-run the job from the previous part to validate our automated test discovery journey.
### Step 1: Create the fix job, trigger, and data files
As we did in the previous part, we have 3 files to create:
The job script is named `04_fix_config_job.py`:
```python=
#!/usr/bin/env python
"""
OSPFv3 configuration fix job file
This job runs the trigger to fix OSPFv3 configuration on hub router by
adding Loopback1 interface to area 0 for both IPv4 and IPv6 address families.
"""
# Import the gRun function to run job with datafiles
from genie.harness.main import gRun
def main():
"""
Main function that runs the job using gRun
"""
# Run the job using gRun with trigger datafile
gRun(
trigger_uids=["FixOSPFv3"],
trigger_datafile="ospfv3_config_fix_datafile.yml",
subsection_datafile="subsection_datafile.yml",
loglevel="INFO",
)
if __name__ == "__main__":
main()
```
The trigger script named `04_ospfv3_config_fix.py` defines the **FixOSPFv3** class which contains 4 methods:
- `verify_hub_loopback1_interface()`
: - Confirms that the Loopback1 interface exists on the hub router
- Executes the command show ip interface brief | include Loopback1
- Verifies the presence of "Loopback1" in the command output
- Fails the test if the interface is not found
- Skips the test if executed on non-hub devices
- `configure_loopback1_ospfv3_ipv4()`
: - Configures OSPFv3 for IPv4 on the Loopback1 interface
- Identifies the OSPFv3 process ID from device attributes (defaults to 100)
- Applies configuration commands to add Loopback1 to OSPFv3 area 0 for IPv4
- Skips the test if executed on non-hub devices
- `configure_loopback1_ospfv3_ipv6()`
: - Configures OSPFv3 for IPv6 on the Loopback1 interface
- Identifies the OSPFv3 process ID from device attributes (defaults to 100)
- Applies configuration commands to add Loopback1 to OSPFv3 area 0 for IPv6
- Skips the test if executed on non-hub devices
- `verify_configuration()`
: - Validates that the OSPFv3 configuration has been successfully applied to Loopback1
- Checks the running configuration with show running-config interface Loopback1
- Verifies both IPv4 and IPv6 OSPFv3 configurations are present
- Uses assertions to confirm proper configuration with the correct process ID
- Skips the test if executed on non-hub devices
```python=
#!/usr/bin/env python
"""
pyATS Trigger to fix OSPFv3 configuration on the hub router
This script adds Loopback1 to OSPFv3 area 0 for both IPv4 and IPv6 address families,
which enables the hub's Loopback1 routes to be advertised to the spoke router.
"""
import logging
from genie.harness.base import Trigger
from unicon.core.errors import SubCommandFailure
from pyats import aetest
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class FixOSPFv3(Trigger):
"""
pyATS Trigger class to fix the OSPFv3 configuration on the hub router.
This class ensures Loopback1 interface is added to OSPFv3 area 0 for
both IPv4 and IPv6 address families.
"""
@aetest.test
def verify_hub_loopback1_interface(self, uut):
"""
Verify that the Loopback1 interface exists on the hub router.
Args:
uut: Device object from the testbed
"""
if uut.name != "hub":
self.skipped("This trigger is only applicable to the hub device")
return
# Check if Loopback1 exists
try:
output = uut.execute("show ip interface brief | include Loopback1")
logger.info(f"Loopback1 interface status: {output}")
if "Loopback1" not in output:
self.failed("Loopback1 interface not found on hub router")
logger.info("Loopback1 interface exists on hub router")
except SubCommandFailure as err:
self.failed(f"Failed to verify Loopback1 interface: {err}")
@aetest.test
def configure_loopback1_ospfv3_ipv4(self, uut):
"""
Configure OSPFv3 on Loopback1 for IPv4 address family.
Args:
uut: Device object from the testbed
"""
if uut.name != "hub":
self.skipped("This trigger is only applicable to the hub device")
return
# Get the OSPFv3 process ID
ospf_process_id = getattr(uut, "ospfv3_process_id", 100)
try:
# Configure OSPFv3 for IPv4 on Loopback1 area 0
config_commands = [
"interface Loopback1",
f"ospfv3 {ospf_process_id} ipv4 area 0",
"exit",
]
logger.info(
f"Configuring OSPFv3 for IPv4 on Loopback1 with process ID {ospf_process_id}"
)
uut.configure(config_commands)
logger.info("Successfully configured OSPFv3 for IPv4 on Loopback1")
except SubCommandFailure as err:
self.failed(f"Failed to configure OSPFv3 IPv4 on Loopback1: {err}")
@aetest.test
def configure_loopback1_ospfv3_ipv6(self, uut):
"""
Configure OSPFv3 on Loopback1 for IPv6 address family.
Args:
uut: Device object from the testbed
"""
if uut.name != "hub":
self.skipped("This trigger is only applicable to the hub device")
return
# Get the OSPFv3 process ID
ospf_process_id = getattr(uut, "ospfv3_process_id", 100)
try:
# Configure OSPFv3 for IPv6 on Loopback1 area 0
config_commands = [
"interface Loopback1",
f"ospfv3 {ospf_process_id} ipv6 area 0",
"exit",
]
logger.info(
f"Configuring OSPFv3 for IPv6 on Loopback1 with process ID {ospf_process_id}"
)
uut.configure(config_commands)
logger.info("Successfully configured OSPFv3 for IPv6 on Loopback1")
except SubCommandFailure as err:
self.failed(f"Failed to configure OSPFv3 IPv6 on Loopback1: {err}")
@aetest.test
def verify_configuration(self, uut):
"""
Verify that the configuration has been applied correctly.
Args:
uut: Device object from the testbed
"""
if uut.name != "hub":
self.skipped("This trigger is only applicable to the hub device")
return
# Get the OSPFv3 process ID
ospf_process_id = getattr(uut, "ospfv3_process_id", 100)
try:
# Verify configuration
output = uut.execute(
f"show running-config interface Loopback1 | include ospfv3 {ospf_process_id}"
)
logger.info(f"Loopback1 OSPFv3 configuration: {output}")
# Check if both IPv4 and IPv6 are configured
ipv4_configured = f"ospfv3 {ospf_process_id} ipv4 area 0" in output
ipv6_configured = f"ospfv3 {ospf_process_id} ipv6 area 0" in output
# Assert both are configured
# trunk-ignore(bandit/B101)
assert (
ipv4_configured
), f"OSPFv3 IPv4 not configured on Loopback1 with process ID {ospf_process_id}"
# trunk-ignore(bandit/B101)
assert (
ipv6_configured
), f"OSPFv3 IPv6 not configured on Loopback1 with process ID {ospf_process_id}"
logger.info("Loopback1 OSPFv3 configuration verified successfully")
except SubCommandFailure as err:
self.failed(f"Failed to verify OSPFv3 configuration on Loopback1: {err}")
if __name__ == "__main__":
from genie.harness.main import gRun
from genie.testbed import load
# Load the testbed
testbed = load("testbed.yaml")
# Run with trigger datafile
gRun(
trigger_datafile="ospfv3_config_fix_datafile.yml",
trigger_uids=["FixOSPFv3"],
testbed=testbed,
subsection_datafile="subsection_datafile.yml",
)
```
Finally, the `ospfv3_config_fix_datafile.yml` datafile specifies the class of tests to use.
```python=
---
extends: ospfv3_base_datafile.yml
FixOSPFv3:
source:
class: 04_ospfv3_config_fix.FixOSPFv3
groups: [fix_ospfv3_config]
devices: [hub]
devices_attributes:
hub:
ospfv3_process_id: 100
```
### Step 2: Run the fix pyATS job
We are ready to run the new job when all three files are in place.
```bash
pyats run job 04_fix_config_job.py --testbed-file testbed.yaml
```
As we did in the previous part, we access the results via the built-in web service.
```bash
pyats logs view
```
The screenshot above shows the configuration command being used to add the Loopback1 interface to OSPFv3 area 0.
### Step 3: Re-run the operational state test job
This new job run shows that all of the network prefix exchange **operational state** tests for our two routers are now successful.
```bash
pyats run job 03_op_state_job.py --testbed-file testbed.yaml
```
The results dashboard shows that all tests have passed.
The two routes advertised by the hub router are present in the spoke router table.
```
spoke#
502: 2025-03-20T08:47:47: IPv4 OSPF routes on spoke:
503: 2025-03-20T08:47:47: Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
504: 2025-03-20T08:47:47: D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
505: 2025-03-20T08:47:47: N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
506: 2025-03-20T08:47:47: E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
507: 2025-03-20T08:47:47: n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
508: 2025-03-20T08:47:47: i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
509: 2025-03-20T08:47:47: ia - IS-IS inter area, * - candidate default, U - per-user static route
510: 2025-03-20T08:47:47: H - NHRP, G - NHRP registered, g - NHRP registration summary
511: 2025-03-20T08:47:47: o - ODR, P - periodic downloaded static route, l - LISP
512: 2025-03-20T08:47:47: a - application route
513: 2025-03-20T08:47:47: + - replicated route, % - next hop override, p - overrides from PfR
514: 2025-03-20T08:47:47: & - replicated local route overrides by connected
515: 2025-03-20T08:47:47:
516: 2025-03-20T08:47:47: Gateway of last resort is not set
517: 2025-03-20T08:47:47:
518: 2025-03-20T08:47:47: 172.16.0.0/32 is subnetted, 2 subnets
519: 2025-03-20T08:47:47: O 172.16.0.1 [110/1] via 10.0.1.1, 1d22h, GigabitEthernet2
520: 2025-03-20T08:47:47: O 172.16.1.1 [110/1] via 10.0.1.1, 00:08:00, GigabitEthernet2
521: 2025-03-20T08:47:47: IPv4 route to hub's Loopback1 (172.16.1.1) correctly present in spoke routing table
522: 2025-03-20T08:47:47: The result of section test_missing_route_ipv4 is => PASSED
```
## Conclusion
This document outlines a comprehensive approach to automating network testing using pyATS and Genie, with a focus on an OSPFv3 protocol exchange between two routers.
Although pyATS is a Cisco tool, the need for automation and the methods used here are universally applicable to various network environments. The lab setup and test procedures demonstrate how automation can streamline network validation and efficiently ensure compliance with desired states. This approach is beneficial for any network infrastructure, regardless of vendor.