397 views
# IaC Lab 1 -- Use Ansible to build new Debian GNU/Linux Virtual Machines [toc] --- > Copyright (c) 2024 Philippe Latu. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". GitLab repository: https://gitlab.inetdoc.net/iac/lab01 ## Background / Scenario In this lab, you will explore the basics of using Ansible to build and customise Debian GNU/Linux virtual machines. This is a first illustration of the **Infrastructure as Code** (IaC) **push** method where the DevNet VM (controlling server) uses Ansible to build a new target system *almost* from scratch. ![IaC lab 1 scenario](https://md.inetdoc.net/uploads/c68d6422-36c3-45ea-95b9-7f3a613c3704.png) The main stages of the scenario are as follows: 1. We start at the Hypervisor shell level by pulling a virtual machine base image from [cloud.debian.org](https://cloud.debian.org/images/cloud/). Then we resize the main partition, and customize the virtual machine image files copied from the cloud image. 2. From the network point of view, virtual machines are customized to use **Virtual Routing and Forwarding** (VRF). The main idea here is to set up a dedicated network to manage the VM from Ansible playbooks located on the DevNet VM. This management network must be isolated from other networks that the Debian VM would use. 3. Once the Debian VM has been properly customized, it can be started and configured with any Ansible playbooks. We can connect them to any network or VLAN by configuring new interfaces. In this context, the Debian virtual machines must be connected to a switch port in **trunk** mode to enable the use of VLAN tags to identify the relevant broadcast domains. # Part 1: Configure Ansible on the DevNet VM First, we need to configure Ansible and check that we have access to the hypervisor from the DevNet VM via SSH. ## Step 1: Create the Ansible directory and configuration file 1. Make the `~/iac/lab01` directory for example and navigate to this folder ```bash mkdir -p ~/iac/lab01 && cd ~/iac/lab01 ``` 2. Check that **ansible** is installed There are two main ways to set up a new Ansible workspace. Packages and Python virtual environments are both viable options. Both methods have their advantages and disadvantages. - If we wish to use the Ansible package provided by the Linux distribution, the tool will be immediately available. However, difficulties may arise with modules that are not up to date or aligned with Python repositories. With an Ubuntu distribution, the package information is as follows: ```bash apt show ansible | head -n 10 ``` ```bash= Package: ansible Version: 7.7.0+dfsg-1 Priority: optional Section: universe/admin Origin: Ubuntu Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Original-Maintainer: Lee Garrett <debian@rocketjump.eu> Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 265 MB Depends: ansible-core (>= 2.11.5-1~), python3:any, openssh-client | python3-paramiko (>= 2.6.0), python3-distutils, python3-dnspython, python3-httplib2, python3-jinja2, python3-netaddr, python3-yaml ``` - To utilize the latest versions of tools in Python, we can create a virtual environment by following these steps: Still with an Ubuntu distribution: ```bash= cat << EOF > requirements.txt ansible ansible-lint EOF ``` ```bash= python3 -m venv ansible source ./ansible/bin/activate pip3 install -r requirements.txt ``` 3. Create a new `ansible.cfg` file in the `lab01` directory from the shell prompt {%gist platu/e5a71ea5fdc67fa5478e7310eab81d5a%} # Part 2: Designing the Declarative Part Now that the Ansible automation tool is installed, it is time to plan the desired state of our laboratory infrastructure. This is the most challenging aspect of the task as we begin with a blank screen. A starting point must be chosen, followed by a description of the expected results. The approach chosen is to start at the bare-metal hypervisor level and then move on to the virtual machine and its network configuration. - At system startup, all provisioned tap interfaces are owned by the type 2 hypervisor. - These tap interfaces connect virtual machines to Open vSwitch switches, just like a patch cable. In addition, the tap interface names are used to designate the switch ports. All configuration instructions for switch ports use these tap interface names. > In our context, we use a switch port in trunk mode to forward the traffic of multiple broadcast domains or VLANs. Each frame flowing through this trunk port uses an IEEE 802.1Q tag to identify the broadcast domain to which it belongs. - Each virtual machine uses a tap interface number to connect its network interface to the switch port. > In our context, we want the virtual machine network interface to use a dedicated virtual routing and forwarding (VRF) table for automation operations. Doing so, the virtual system network traffic and the automation traffic are completely independant and isolated. - The VLAN used for automation operations is referred to as the **Out of Band** network, as it is exclusively reserved for management traffic and does not carry any user traffic. - All the other VLANs used for laboratory traffic are referred to as the **In Band** network, as all user traffic flows through them. Now that all statements are in place, they must be translated into YAML description files that reflect the desired state. ## Step 1: The inventory directory and its content When using Ansible, it is important to differentiate between the inventory directory and the host variables directory. We can start by creating the `inventory` and `host_vars` directories ```bash mkdir ~/iac/lab01/{inventory,host_vars} ``` The inventory directory files contain the necessary information, such as host names, groups, and attributes, required to establish network connections with these hosts. Here is a copy of the `inventory/hosts.yml` file. {%gist platu/84c2dd1d414e8f9b295924e38079b942%} The YAML description above contains two groups: **hypervisors** and **VMs**. Within the hypervisors group, Bob is currently the only member present with the necessary SSH network connection parameters. The VMs group comprises two members, namely **vmXXX** and **vmYYY**. At this stage, we do not know much except for the fact that we are going to instantiate two virtual machines. The SSH network connection parameters for all virtual machines will be provided after they are started and the dynamic inventory Python script is executed. ## Step 2: The host_vars directory and its content The content of the host_vars directory will now be examined. YAML description files for each host of the lab infrastructure can be found there. Here are copies of: - `host_vars/bob.yml` - `host_vars/vmXXX.yml` - `host_vars/vmYYY.yml` :warning: Be sure to edit this inventory file and replace the **XXX** and **YYY** placeholders with the corresponding real names. {%gist platu/1312cb965502325f08bc5581f032192f%} The hypervisor YAML file `bob.yml` contains a list of tap interfaces to be configured, as specified in the design. The configuration parameters for each tap interface listed include the switch name, trunk mode port, and the list of allowed VLANs in the trunk. Other parameters relate to the image pull source that is common to all virtual machines in this lab. The YAML files for virtual machines reference the hypervisor tap interface connection and contain the network interface **In Band** VLAN configuration parameters. Notice that IPv4 addresses are calculated using the tap interface number. The intent is to avoid students to use the same IPv4 address for different virtual machines. # Part 3: Access the hypervisor from the DevNet virtual machine using Ansible Now that the minimum inventory and host variables are in place, it is necessary to verify hypervisor accessibility before pulling virtual machine images. In this part, a vault is created to store all secrets on the DevNet virtual machine user home directory. As stated in the beginning of this lab, we are not utilizing an external service to handle confidential information. ## Step 1: Check SSH access from DevNet VM to the Hypervisor We start with a shell test connection before to set up the configuration for **Ansible**. One more time, be sure to change tap interface number to match your resource allocation. ```bash ssh etudianttest@fe80:XXX::1%enp0s1 ``` ```bash= ssh -p 2222 etudianttest@fe80:XXX::1%enp0s1 The authenticity of host '[fe80:XXX::1%enp0s1]:2222 ([fe80:XXX::1%enp0s1]:2222)' can't be established. ED25519 key fingerprint is SHA256:xnQumIo5mnWNJh+Tcak0XmEAkxptGHpiFqd0BNLOMgo. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '[fe80:XXX::1%enp0s1]:2222' (ED25519) to the list of known hosts. Linux bob 6.6.15-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.6.15-2 (2024-02-04) x86_64 ``` ## Step 2: Create a new vault file Back to the DevNet VM console, create a new vault file called `iac_lab01_passwd.yml` and enter the unique vault password which will be used for all users passwords to be stored. ```bash ansible-vault create $HOME/iac_lab01_passwd.yml ``` ```bash= New Vault password: Confirm New Vault password: ``` This will open the default editor which is defined by the `$EDITOR` environment variable. There we enter a variable name which will designate the password for Web server VM user account. ```bash hypervisor_user: XXXXXXXXXX hypervisor_pass: YYYYYYYYYY vm_user: etu vm_pass: ZZZZZZZZZ ``` ## Step 3: Verify Ansible communication with the Hypervisor Now, we are able to use the `ping` ansible module to commincate with the `bob` entry defined in the inventory file. ```bash ansible bob -m ping --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= bob | SUCCESS => { "changed": false, "ping": "pong" } ``` As the ansible ping is successful, we can go on with playbooks to build new virtual machines. # Part 4: Designing the procedural part In [Part 2](#Part-2-Designing-the-Declarative-Part), lab design choices were translated into declarative YAML files to assess the desired state of the two virtual machines' network connections. Now, we need to use this declarative information in procedures to effectively build, customize, and run the virtual machines. In order to be able to build and launch virtual machines, we first need to prepare directories and get access to virtual machines launch scripts. Next, we need to check that the switch port to which a virtual machine will be connected is in **trunk** mode. ## Step 1: Preparation stage at the Hypervisor level This is a copy of the first Ansible Playbook, which includes two categories of procedures. - In the initial phase, the tasks ensure that all the required directories and symlinks are in place in the user's home directory to run a virtual machine. These operations are familiar to all students as they are provided at the top of each Moodle course page as shell instructions. - In the second phase, the configuration of the tap switch ports is adjusted to match the attributes specified in the YAML file. In this particular lab context, each switch port is configured in trunk mode with a restricted list of allowed VLANs. The playbook is designed for reusability and includes instructions for configuring switch ports in either access or trunk mode. {%gist platu/ebf4e99f243766db3b612759c97d7c62%} The playbook uses two Ansible modules: **file** and **shell**. - [ansible.builtin.file module](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html) manages all types of files and their properties - [ansible.builtin.shell module](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/shell_module.html) executes shell commands on target hosts. The most interesting aspects of the shell tasks are: - Using messages sent to standard output to determine whether or not a configuration change has occurred. Look for the `changed_when:` keyword in the playbook code. - Using item attributes to know whether or not to run a task. Look for the `when:` keyword in the playbook code. - Using item attributes to also know if a task has failed or not. Look for the `failed_when:` keyword in the same code. When we run the playbook, we get the following output. ```bash ansible-playbook prepare.yml --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= PLAY [PREPARE LAB ENVIRONMENT] ************************************************* TASK [Gathering Facts] ********************************************************* ok: [bob] TASK [ENSURE SYMLINK TO MASTERS DIRECTORY EXISTS] ****************************** ok: [bob] TASK [ENSURE VM DIRECTORY EXISTS] ********************************************** ok: [bob] TASK [ENSURE SYMLINK TO SCRIPTS DIRECTORY EXISTS] ****************************** ok: [bob] TASK [ENSURE LAB DIRECTORY EXISTS] ********************************************* ok: [bob] TASK [CHECK IF TAP INTERFACES ARE ALREADY USED BY ANOTHER USER] **************** ok: [bob] => (item={'name': 'tap4', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmXXX', 'switch': 'dsw-host'}) ok: [bob] => (item={'name': 'tap5', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmYYY', 'switch': 'dsw-host'}) TASK [CONFIGURE TAP INTERFACES SWITCH CONNECTION] ****************************** ok: [bob] => (item={'name': 'tap4', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmXXX', 'switch': 'dsw-host'}) ok: [bob] => (item={'name': 'tap5', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmYYY', 'switch': 'dsw-host'}) TASK [CONFIGURE TAP INTERFACES IN ACCESS MODE] ********************************* skipping: [bob] => (item={'name': 'tap4', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmXXX', 'switch': 'dsw-host'}) skipping: [bob] => (item={'name': 'tap5', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmYYY', 'switch': 'dsw-host'}) skipping: [bob] TASK [CONFIGURE TAP INTERFACES IN TRUNK MODE] ********************************** ok: [bob] => (item={'name': 'tap4', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmXXX', 'switch': 'dsw-host'}) ok: [bob] => (item={'name': 'tap5', 'vlan_mode': 'trunk', 'trunks': [0, 28, 230], 'link': 'vmYYY', 'switch': 'dsw-host'}) PLAY RECAP ********************************************************************* bob : ok=8 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 ``` ## Step 2: Develop scripts to resize, customize, and create inventory entry Bash and Python scripts were chosen to separate operations. ### Increase virtual machine storage capacity Once the generic cloud image file is pulled from [cloud.debian.org](), we first have to resize its main partition to increase storage space. Here is the `resize.sh` script which is based on `qemu-img` and `virt-resize` commands. The function takes two input parameters: the filename of the virtual machine image and the desired storage capacity to be added. {%gist platu/80abe91df20868cc8e49e1de0f23a8ed%} ### Customize virtual machine The second script, called `customize.sh`, is more complex, since it performs several tasks. All these tasks are controlled by the `virt-customize` command. Below is a list of the processing performed on the four input parameters. - Create a user account, set its password and add it to system groups `sudo` and `adm` - Set the timezone - Prepare and copy all the network configuration files into the virtual machine `/etc/systemd/network/` virtual machine directory - Reconfigure SSH service to allow password connection on port 2222 Here is the `customize.sh` script source code with its input parameters: - virtual machine image filename - user name to be used for Ansible SSH connections - user password extracted from Ansible vault - Out of band VLAN identifier {%gist platu/1fd65398266c1cb10db581018acdfb10%} ### Build dynamic inventory After launching virtual machines, we must update the Ansible inventory with a new YAML file created from the output messages of the launch operations. The YAML entry should include the name of the virtual machine and its IPv6 local link address within the Out of Band VLAN. Here is the source code of the `build_lab_inventory.py`: {%gist platu/1c5c5c3070da7038140007936f70fd42%} ## Step 3: Create an Ansible playbook to synthetise scripts operations. We are now ready to pull virtual machine image, resize main partition, customize and add inventory. In this step, we develop a playbook called `pull_customize_run.yml` that calls the scripts developed in the previous step. {%gist platu/986366adc99e558a3bf73fd43d52e03c%} The playbook contains 7 tasks: DOWNLOAD DEBIAN CLOUD IMAGE QCOW2 FILE : Use the **get_url** module to download image file from cloud.debian.org. COPY CLOUD IMAGE TO VM IMAGE FILE FOR ALL VMS : After pulling the reference image, we need to create a copy for each virtual machine defined in the Ansible inventory. RESIZE VIRTUAL MACHINE FILESYSTEM : Use the **script** module to run the `resize.sh` script on the hypervisor with the parameter list provided by the Ansible variables. CUSTOMIZE VIRTUAL MACHINE IMAGE : Use the **script** module to run the `customize.sh` script on the hypervisor with the parameter list provided by the Ansible variables. The main interest of the **script** module comes from the fact that we don't have to worry about copying a script from the DevNet VM to the hypervisor before running it. LAUNCH VIRTUAL MACHINE : Use the **shell** module to call the `ovs-startup.sh` script which is already present on the hypervisor. The necessary directories and symlinks were set up when the `prepare.yml` playbook was run. From this point on, the new virtual machine is active and running. FORCE LAB INVENTORY REBUILD : Use the **file** module to delete launch output trace and previous inventory files in the DevNet working directory to ensure that the dynamic inventory is up to date as the **fetch** module used in the next task does not provide an update facility. FETCH LAUNCH OUTPUT MESSAGES : Use the **fetch** module to collect launch output messages from the hypervisor to the DevNet VM. These messages are useful for creating the new inventory entry that gives Ansible access to the virtual machine. BUILD LAB INVENTORY : Use the **command** module to run the Python script that extracts the virtual machine name and IPv6 local link address of the virtual machine and creates the inventory entry in YAML format. Notice the **delegate_to** option which allows to run the script on the DevNet VM itself. ## Step 4: Run the `pull_customize_run.yml` playbook Here is a sample output of the playbook execution. ```bash ansible-playbook pull_customize_run.yml --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= PLAY [PULL AND CUSTOMIZE CLOUD IMAGE] ************************************* TASK [Gathering Facts] **************************************************** ok: [bob] TASK [DOWNLOAD DEBIAN CLOUD IMAGE QCOW2 FILE] ***************************** changed: [bob] TASK [COPY CLOUD IMAGE TO VM IMAGE FILE FOR ALL VMS] ********************** changed: [bob] => (item=vmXXX) changed: [bob] => (item=vmYYY) TASK [RESIZE VIRTUAL MACHINE FILESYSTEM] ********************************** changed: [bob] => (item=vmXXX) changed: [bob] => (item=vmYYY) TASK [CUSTOMIZE VIRTUAL MACHINE IMAGE] ************************************ changed: [bob] => (item=vmXXX) changed: [bob] => (item=vmYYY) TASK [LAUNCH VIRTUAL MACHINE] ********************************************* changed: [bob] => (item=vmXXX) changed: [bob] => (item=vmYYY) TASK [FORCE LAB INVENTORY REBUILD] **************************************** changed: [bob -> localhost] => (item=trace/launch_output.txt) changed: [bob -> localhost] => (item=inventory/lab.yml) TASK [FETCH LAUNCH OUTPUT MESSAGES] *************************************** changed: [bob] TASK [BUILD LAB INVENTORY] ************************************************ changed: [bob -> localhost] PLAY RECAP **************************************************************** bob : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` ## Step 5: Check Ansible SSH access to the target virtual machines Here we use the **ping** Ansible module directly from the command line. ```bash ansible vms -m ping --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` > We use the **vms** group entry defined in the main inventory file set in Part 1: `hosts.yml` ```bash= vmXXX | SUCCESS => { "changed": false, "ping": "pong" } vmYYY | SUCCESS => { "changed": false, "ping": "pong" } ``` We can also check the inventory contains **vmXXX** and **vmYYY** entries with their own parameters. ```bash ansible-inventory --yaml --list ``` ```yaml= all: children: hypervisors: hosts: bob: ansible_host: fe80:1c::1%enp0s1 ansible_ssh_pass: '{{ hypervisor_pass }}' ansible_ssh_port: 2222 ansible_ssh_user: '{{ hypervisor_user }}' cloud_url: cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.qcow2 filesystem_resize: 32G image_name: debian-13-amd64.qcow2 lab_name: iac_lab01 oob_vlan: 28 taps: - link: vmXXX name: tap4 switch: dsw-host trunks: - 0 - 28 - 230 vlan_mode: trunk - link: vmYYY name: tap5 switch: dsw-host trunks: - 0 - 28 - 230 vlan_mode: trunk vms: hosts: vmXXX: ansible_become: 'true' ansible_become_password: '{{ vm_pass }}' ansible_host: fe80::baad:caff:fefe:4%enp0s1 ansible_port: 2222 ansible_ssh_pass: '{{ vm_pass }}' ansible_ssh_user: '{{ vm_user }}' inband_vlan: 230 interfaces: - interface_id: '{{ inband_vlan }}' interface_type: vlan ipv4_address: 10.0.{{ 228 + tap_number|int // 256 }}.{{ tap_number|int % 256 }}/22 ipv4_dns: 172.16.0.2 ipv4_gateway: 10.0.228.1 patches: enp0s1: tap4 ram: 1024 tap_number: '{{ patches.enp0s1 | regex_replace(''tap'', '''') }}' vmYYY: ansible_become: 'true' ansible_become_password: '{{ vm_pass }}' ansible_host: fe80::baad:caff:fefe:5%enp0s1 ansible_port: 2222 ansible_ssh_pass: '{{ vm_pass }}' ansible_ssh_user: '{{ vm_user }}' inband_vlan: 230 interfaces: - interface_id: '{{ inband_vlan }}' interface_type: vlan ipv4_address: 10.0.{{ 228 + tap_number|int // 256 }}.{{ tap_number|int % 256 }}/22 ipv4_dns: 172.16.0.2 ipv4_gateway: 10.0.228.1 patches: enp0s1: tap5 ram: 1024 tap_number: '{{ patches.enp0s1 | regex_replace(''tap'', '''') }}' ``` This completes the **Infrastructure as Code** part of creating virtual machines from scratch. We can now control and configure these new virtual machines from the DevNet VM. # Part 5: Add an In band VLAN connection to the virtual machines Before adding any new in-band network access to our virtual machines, we analyze the current network configuration and its addresses within the out-of-band VLAN. Then, we create another Ansible playbook to provide access to the selected in-band VLANs and their addressing plan. ## Step 1: Look at the actual network configuration Now that we have access to our newly built virtual machines, let's examine the current network configuration. The main point here is to show that even though we can see the management addresses of the dedicated interface, **the default routing tables are empty**. ### What are the addresses assigned to the out-of-band VLAN? We just have to list addresses with the `ip` command using the **command** ansible module on the DevNet virtual machine. ```bash ansible vms -m command -a "ip addr ls" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmXXX | CHANGED | rc=0 >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether b8:ad:ca:fe:00:04 brd ff:ff:ff:ff:ff:ff inet6 fe80::baad:caff:fefe:4/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 3: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000 link/ether 62:55:83:00:0f:41 brd ff:ff:ff:ff:ff:ff 4: mgmt@enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master mgmt-vrf state UP group default qlen 1000 link/ether b8:ad:ca:fe:00:04 brd ff:ff:ff:ff:ff:ff inet 198.18.29.4/23 metric 1024 brd 198.18.29.255 scope global dynamic mgmt valid_lft 85276sec preferred_lft 85276sec inet6 2001:678:3fc:1c:baad:caff:fefe:4/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 85933sec preferred_lft 13933sec inet6 fe80::baad:caff:fefe:4/64 scope link proto kernel_ll valid_lft forever preferred_lft forever vmYYY | CHANGED | rc=0 >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether b8:ad:ca:fe:00:05 brd ff:ff:ff:ff:ff:ff inet6 fe80::baad:caff:fefe:5/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 3: mgmt-vrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000 link/ether 86:61:67:8a:b4:1a brd ff:ff:ff:ff:ff:ff 4: mgmt@enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master mgmt-vrf state UP group default qlen 1000 link/ether b8:ad:ca:fe:00:05 brd ff:ff:ff:ff:ff:ff inet 198.18.29.5/23 metric 1024 brd 198.18.29.255 scope global dynamic mgmt valid_lft 85278sec preferred_lft 85278sec inet6 2001:678:3fc:1c:baad:caff:fefe:5/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 85933sec preferred_lft 13933sec inet6 fe80::baad:caff:fefe:5/64 scope link proto kernel_ll valid_lft forever preferred_lft forever ``` - Lines starting with '2:' indicate that the main network interface `enp0s1` doesn't have any address, except a link-local IPv6 address, which is not relevant. - Lines starting with "3:" indicate that the `mgmt-vrf` interface defines a **virtual routing and forwarding** context. - Lines starting with '4:' indicate that the `mgmt@enp0s1` interface is the management interface used by Ansible SSH connections to manage the target virtual machines system. The management network, also known as the out-of-band VLAN, is a separate and independent network from the target system's network configuration. This is obvious when examining the routing tables. ### What network entries are provided by the routing tables? As with the network addresses, the `ip` command and ansible **command** module are used. The default IPv4 routing table on the target system is empty. ```bash ansible vms -m command -a "ip route ls" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmYYY | CHANGED | rc=0 >> vmXXX | CHANGED | rc=0 >> ``` We get the same result with IPv6 default routing table. ```bash ansible vms -m command -a "ip -6 route ls" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmYYY | CHANGED | rc=0 >> fe80::/64 dev enp0s1 proto kernel metric 256 pref medium vmXXX | CHANGED | rc=0 >> fe80::/64 dev enp0s1 proto kernel metric 256 pref medium ``` We have to specify the VRF context to get routing tables network entries for the management out-of-band network. ```bash ansible vms -m command -a "ip route ls vrf mgmt-vrf" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmXXX | CHANGED | rc=0 >> default via 198.18.28.1 dev mgmt proto dhcp src 198.18.29.4 metric 1024 172.16.0.2 via 198.18.28.1 dev mgmt proto dhcp src 198.18.29.4 metric 1024 198.18.28.0/23 dev mgmt proto kernel scope link src 198.18.29.4 metric 1024 198.18.28.1 dev mgmt proto dhcp scope link src 198.18.29.4 metric 1024 vmYYY | CHANGED | rc=0 >> default via 198.18.28.1 dev mgmt proto dhcp src 198.18.29.5 metric 1024 172.16.0.2 via 198.18.28.1 dev mgmt proto dhcp src 198.18.29.5 metric 1024 198.18.28.0/23 dev mgmt proto kernel scope link src 198.18.29.5 metric 1024 198.18.28.1 dev mgmt proto dhcp scope link src 198.18.29.5 metric 1024 ``` IPv4 network configurationn is provided by DHCP. ```bash ansible vms -m command -a "ip -6 route ls vrf mgmt-vrf" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmXXX | CHANGED | rc=0 >> 2001:678:3fc:1c::/64 dev mgmt proto ra metric 1024 expires 86131sec hoplimit 64 pref medium fe80::/64 dev mgmt proto kernel metric 256 pref medium multicast ff00::/8 dev mgmt proto kernel metric 256 pref medium default via fe80::801:42ff:feb7:13ba dev mgmt proto ra metric 1024 expires 1531sec hoplimit 64 pref medium vmYYY | CHANGED | rc=0 >> 2001:678:3fc:1c::/64 dev mgmt proto ra metric 1024 expires 86131sec hoplimit 64 pref medium fe80::/64 dev mgmt proto kernel metric 256 pref medium multicast ff00::/8 dev mgmt proto kernel metric 256 pref medium default via fe80::801:42ff:feb7:13ba dev mgmt proto ra metric 1024 expires 1531sec hoplimit 64 pref medium ``` IPv6 network configuration is provided by SLAAC Router Advertisment (RA). ## Step 2: Adding a new network access to an In-Band VLAN In the previous step, we had the confirmation that management network access is isolated in a dedicated namespace. We are now ready to add access to new in-band VLANs that will be available to the `ansible_user` for lab operations. It's time to complete the virtual machines addressing plan and create a new network configuration playbook. ### Declare In-Band VLAN addressing First, we need to choose among all the VLANs provided by the private cloud infrastructure. For instance, let us consider VLAN 230, which is addressed in the following manner: - IPv4 static addressing with default gateway: 10.0.228.1/22 - IPv6 SLAAC addressing We can print virtual machine group `vms` inventory entries as they were defined in the declarative phase of the lab design ([Part 2](#Part-2-Designing-the-Declarative-Part)). We can recall here that **IPv4 addresses are calculated** with each virtual machine tap interface number **to avoid duplicates**. ```bash ansible-inventory --yaml --limit vms --list ``` ```yaml= all: children: vms: hosts: vmXXX: ansible_become: 'true' ansible_become_password: '{{ vm_pass }}' ansible_host: fe80::baad:caff:fefe:4%enp0s1 ansible_port: 2222 ansible_ssh_pass: '{{ vm_pass }}' ansible_ssh_user: '{{ vm_user }}' inband_vlan: 230 interfaces: - interface_id: '{{ inband_vlan }}' interface_type: vlan ipv4_address: 10.0.{{ 228 + tap_number|int // 256 }}.{{ tap_number|int % 256 }}/22 ipv4_dns: 172.16.0.2 ipv4_gateway: 10.0.228.1 patches: enp0s1: tap4 ram: 1024 tap_number: '{{ patches.enp0s1 | regex_replace(''tap'', '''') }}' vmYYY: ansible_become: 'true' ansible_become_password: '{{ vm_pass }}' ansible_host: fe80::baad:caff:fefe:5%enp0s1 ansible_port: 2222 ansible_ssh_pass: '{{ vm_pass }}' ansible_ssh_user: '{{ vm_user }}' inband_vlan: 230 interfaces: - interface_id: '{{ inband_vlan }}' interface_type: vlan ipv4_address: 10.0.{{ 228 + tap_number|int // 256 }}.{{ tap_number|int % 256 }}/22 ipv4_dns: 172.16.0.2 ipv4_gateway: 10.0.228.1 patches: enp0s1: tap5 ram: 1024 tap_number: '{{ patches.enp0s1 | regex_replace(''tap'', '''') }}' ``` ### Create a playbook to set up the new network access The tasks outlined in this new playbook called `add_vlan.yml` involve adding two new network configuration files and a VLAN entry to the main interface configuration file of the virtual machines. If necessary, reload all network interfaces. {%gist platu/fa8c5b231c006b362cf08a25624ef8e9%} The following Ansible modules are involved in managing network configuration files: [ansible.builtin.stat](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/stat_module.html) : Retrieve file or file system status In our lab context, we need to check that the netdev and network files exist before we can create or edit them. The **stat** module is used to collect the status of the files. Then we can decide if the files need to be created or not. [ansible.builtin.copy](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/copy_module.html) : Copy files to remote locations In our lab context, we are not actually copying files, but providing content to be copied to the virtual machine's /etc/systemd/network directory files. This copy operation only occurs if the target files do not already exist. [ansible.builtin.lineinfile](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/lineinfile_module.html) : Manage lines in text files This module is used to edit files depending on the network configuration attributes provided in the YAML host variable files. If the VLAN default gateway is declared, it will be added to the interface network file. The same applies to the DNS resolver. ## Step3: Run the `add_vlan.yml` playbook and test the new network access When running this playboook, we get the following output: ```bash ansible-playbook add_vlan.yml --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= PLAY [ADD IN BAND VLAN ACCESS] ************************************************* TASK [Gathering Facts] ********************************************************* ok: [vmYYY] ok: [vmXXX] TASK [CHECK IF SYSTEMD NETDEV FILE EXISTS FOR IN BAND VLAN] ******************** ok: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) ok: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) TASK [CREATE SYSTEMD NETDEV FILE FOR IN BAND VLAN] ***************************** changed: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) changed: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) TASK [CHECK IF SYSTEMD NETWORK FILE EXISTS FOR IN BAND VLAN] ******************* ok: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) ok: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) TASK [CREATE SYSTEMD NETWORK FILE FOR IN BAND VLAN] **************************** changed: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) changed: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) TASK [UPDATE DEFAULT GATEWAY IF DEFINED] *************************************** changed: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) changed: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) TASK [UPDATE DNS RESOLVER ADDRESS IF DEFINED] ********************************** changed: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) changed: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) TASK [UPDATE MAIN INTERFACE FILE] ********************************************** changed: [vmXXX] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.4/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) changed: [vmYYY] => (item={'interface_type': 'vlan', 'interface_id': 230, 'ipv4_address': '10.0.228.5/22', 'ipv4_gateway': '10.0.228.1', 'ipv4_dns': '172.16.0.2'}) RUNNING HANDLER [RESTART SYSTEMD NETWORKD] ************************************* changed: [vmYYY] changed: [vmXXX] PLAY RECAP ********************************************************************* vmXXX : ok=9 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vmYYY : ok=9 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` When testing the new in-band network access, we follow standard procedures: ### List IPv4 and IPv6 addresses of the new VLAN interface ```bash ansible vms -m command -a "ip addr ls dev vlan230" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmXXX | CHANGED | rc=0 >> 5: vlan230@enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:ad:ca:fe:00:04 brd ff:ff:ff:ff:ff:ff inet 10.0.228.4/22 brd 10.0.231.255 scope global vlan230 valid_lft forever preferred_lft forever inet6 2001:678:3fc:e6:baad:caff:fefe:4/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 2591939sec preferred_lft 604739sec inet6 fe80::baad:caff:fefe:4/64 scope link proto kernel_ll valid_lft forever preferred_lft forever vmYYY | CHANGED | rc=0 >> 5: vlan230@enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:ad:ca:fe:00:05 brd ff:ff:ff:ff:ff:ff inet 10.0.228.5/22 brd 10.0.231.255 scope global vlan230 valid_lft forever preferred_lft forever inet6 2001:678:3fc:e6:baad:caff:fefe:5/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 2591939sec preferred_lft 604739sec inet6 fe80::baad:caff:fefe:5/64 scope link proto kernel_ll valid_lft forever preferred_lft forever ``` ### List the default IPv4 and IPv6 routing tables entries ```bash ansible vms -m command -a "ip route ls" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmYYY | CHANGED | rc=0 >> default via 10.0.228.1 dev vlan230 proto static 10.0.228.0/22 dev vlan230 proto kernel scope link src 10.0.228.5 vmXXX | CHANGED | rc=0 >> default via 10.0.228.1 dev vlan230 proto static 10.0.228.0/22 dev vlan230 proto kernel scope link src 10.0.228.4 ``` ```bash ansible vms -m command -a "ip -6 route ls" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmYYY | CHANGED | rc=0 >> 2001:678:3fc:e6::/64 dev vlan230 proto ra metric 512 expires 2591808sec mtu 9000 hoplimit 64 pref high fe80::/64 dev enp0s1 proto kernel metric 256 pref medium fe80::/64 dev vlan230 proto kernel metric 256 pref medium default via fe80:e6::1 dev vlan230 proto ra metric 512 expires 1608sec mtu 9000 hoplimit 64 pref high vmXXX | CHANGED | rc=0 >> 2001:678:3fc:e6::/64 dev vlan230 proto ra metric 512 expires 2591808sec mtu 9000 hoplimit 64 pref high fe80::/64 dev enp0s1 proto kernel metric 256 pref medium fe80::/64 dev vlan230 proto kernel metric 256 pref medium default via fe80:e6::1 dev vlan230 proto ra metric 512 expires 1608sec mtu 9000 hoplimit 64 pref high ``` ### Run ICMP tests to a public Internet address ```bash ansible vms -m command -a "ping -c3 9.9.9.9" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmXXX | CHANGED | rc=0 >> PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data. 64 bytes from 9.9.9.9: icmp_seq=1 ttl=51 time=37.6 ms 64 bytes from 9.9.9.9: icmp_seq=2 ttl=51 time=26.5 ms 64 bytes from 9.9.9.9: icmp_seq=3 ttl=51 time=22.4 ms --- 9.9.9.9 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 22.367/28.827/37.627/6.445 ms vmYYY | CHANGED | rc=0 >> PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data. 64 bytes from 9.9.9.9: icmp_seq=1 ttl=51 time=22.6 ms 64 bytes from 9.9.9.9: icmp_seq=2 ttl=51 time=22.2 ms 64 bytes from 9.9.9.9: icmp_seq=3 ttl=51 time=22.5 ms --- 9.9.9.9 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 22.206/22.455/22.614/0.178 ms ``` ```bash ansible vms -m command -a "ping -c3 2620:fe::fe" --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= vmXXX | CHANGED | rc=0 >> PING 2620:fe::fe (2620:fe::fe) 56 data bytes 64 bytes from 2620:fe::fe: icmp_seq=1 ttl=59 time=146 ms 64 bytes from 2620:fe::fe: icmp_seq=2 ttl=59 time=39.3 ms 64 bytes from 2620:fe::fe: icmp_seq=3 ttl=59 time=39.9 ms --- 2620:fe::fe ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 39.333/74.932/145.585/49.959 ms vmYYY | CHANGED | rc=0 >> PING 2620:fe::fe (2620:fe::fe) 56 data bytes 64 bytes from 2620:fe::fe: icmp_seq=1 ttl=59 time=259 ms 64 bytes from 2620:fe::fe: icmp_seq=2 ttl=59 time=39.8 ms 64 bytes from 2620:fe::fe: icmp_seq=3 ttl=59 time=39.5 ms --- 2620:fe::fe ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 39.549/112.924/259.404/103.577 ms ``` All the above ICMP and ICMPv6 tests show that there no packet loss and that packet routing is fully functional with IPv4 and IPv6. # Part 6: Virtual machines system configuration Last but not least, we can use the in-band VLAN network access to run a few system configuration tasks on the virtual machines. ## Step 1: Create an operating system configuration playbook We want to tune the following operating system parts: - Localization - Time zone - Package management with proper signature checks - A bash shell alias called `ll` Therefore, we create a `system_bits.yml` playbook file. {%gist platu/8e966ea0efe021721c5c3ec3c443ced1 %} Fortunately, Ansible modules such as **locale_gen** and **timezone** simplify the operations. The most complex part of this playbook uses the **replace** module with regular expressions. The line `Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg`must be added after each `Components: main` of the `/etc/apt/sources.list.d/debian.sources` only if it's not already there. Finally, the **lineinfile** module is called to create a `$HOME/.bash_aliases` file for the `ansible_user` account with the correct attributes. ## Step 2: Run the `system_bits.yml` playbook ```bash ansible-playbook system_bits.yml --ask-vault-pass --extra-vars @$HOME/iac_lab01_passwd.yml Vault password: ``` ```bash= PLAY [CONFIGURE SYSTEM BITS AND PIECES] **************************************** TASK [Gathering Facts] ********************************************************* ok: [vmYYY] ok: [vmXXX] TASK [CONFIGURE VM LOCALES] **************************************************** changed: [vmYYY] changed: [vmXXX] TASK [CONFIGURE VM TIMEZONE] *************************************************** changed: [vmXXX] changed: [vmYYY] TASK [INSTALL DEBIAN KEYRING] ************************************************** changed: [vmYYY] changed: [vmXXX] TASK [UPDATE DEBIAN SOURCES] *************************************************** changed: [vmXXX] changed: [vmYYY] TASK [ADD LL BASH ALIAS] ******************************************************* changed: [vmXXX] changed: [vmYYY] PLAY RECAP ********************************************************************* vmXXX : ok=6 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vmYYY : ok=6 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` This lab concludes with the final configuration step. At this point, we are prepared to initiate new automated networking manipulations.