# IaC Lab 2 -- Using GitLab CI to run Ansible playbooks an build new Debian GNU/Linux Virtual Machines [toc] --- > Copyright (c) 2024 Philippe Latu. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". GitLab repository: https://gitlab.inetdoc.net/iac/lab02 ## Background / Scenario This lab will introduce you to the basics of using GitLab CI/CD to automate the building of Debian GNU/Linux virtual machines through a pipeline. We are transitioning from the development area ([Lab 1](https://md.inetdoc.net/s/f-mfjs-kQ)) to the testing area (this Lab) since all Ansible playbooks are available in a Git repository stored in a GitLab instance. To perform this task using the usual **Infrastructure as Code** path, a GitLab docker/podman container can be used to initiate a new Ansible installation from scratch. Here we have made a different questionable choice. To avoid using multiple cloud services, we have opted for a minimal GitLab CI/CD setup with a shell runner and the following two arguments: - Ansible vault secrets are already stored on the DevNet virtual machine and we do not use any cloud provider. - The dynamic inventory is generated from a python script run by a playbook and does not rely on any external service.  # Part 1: Install and configure GitLab Runner on the DevNet virtual machine The diagram above illustrates the need to establish communication between two distinct entities: the GitLab service, which is hosted on a separate machine from the hypervisor, and the GitLab runner, which must be installed on the DevNet virtual machine. To begin, we will install the GitLab runner and configure it. Next, we will create a new runner from the GitLab web interface. The **token** generated during this process will be required for runner registration. ## Step 1: Install gitlab-runner package The GitLab Runner is available as a Debian package. To install it, a Debian package repository must be added to the sources along with a signing key to check package integrity. 1. Download and save the GPG public signing key ```bash= curl -fsSL https://packages.gitlab.com/runner/gitlab-runner/gpgkey |\ sudo gpg --dearmor -o /usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg ``` 2. Add the new repository source file ```bash= echo "deb [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] \ https://packages.gitlab.com/runner/gitlab-runner/ubuntu/ jammy main" |\ sudo tee runner_gitlab-runner.list ``` 3. Update package list and install gitlab-runner ```bash= sudo apt update && sudo apt install gitlab-runner ``` 4. Check gitlab-runner service status ```bash systemctl status gitlab-runner.service ``` ```bash= ● gitlab-runner.service - GitLab Runner Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; preset: enabled) Active: active (running) since Thu 2024-03-07 09:56:44 CET; 2min 9s ago Main PID: 10258 (gitlab-runner) Tasks: 9 (limit: 9455) Memory: 11.3M (peak: 14.1M) CPU: 142ms CGroup: /system.slice/gitlab-runner.service └─10258 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner mars 07 09:56:44 devnet24 systemd[1]: Started gitlab-runner.service - GitLab Runner. mars 07 09:56:44 devnet24 gitlab-runner[10258]: Runtime platform arch=amd64 os=linux pid=10258 revision=782c6ecb version=16.9.1 mars 07 09:56:44 devnet24 gitlab-runner[10258]: Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0 max_builds=0 mars 07 09:56:44 devnet24 gitlab-runner[10258]: Running in system-mode. mars 07 09:56:44 devnet24 gitlab-runner[10258]: mars 07 09:56:44 devnet24 gitlab-runner[10258]: Created missing unique system ID system_id=s_a345ff83db78 mars 07 09:56:44 devnet24 gitlab-runner[10258]: Configuration loaded builds=0 max_builds=1 mars 07 09:56:44 devnet24 gitlab-runner[10258]: listen_address not defined, metrics & debug endpoints disabled builds=0 max_builds=1 mars 07 09:56:44 devnet24 gitlab-runner[10258]: [session_server].listen_address not defined, session endpoints disabled builds=0 max_builds=1 mars 07 09:56:44 devnet24 gitlab-runner[10258]: Initializing executor providers builds=0 max_builds=1 ``` From the command results above, we obtain some interesting information about the GitLab Runner service: - The service is up and running - It runs with the gitlab-runner **user** identity - The service configuration file is: `/etc/gitlab-runner/config.toml` - The **listen_address** is not defined. So we need to define it! ## Step 2: Configure gitlab-runner service listen and advertise addresses The initial content of the **gitlab-runner** service configuration file comes with the follwing content: ```toml concurrent = 1 check_interval = 0 shutdown_timeout = 0 [session_server] session_timeout = 1800 ``` We need to add two new address entries in the `[session_server]` category. For convenience, we choose the DevNet IPv6 address as our `advertise_address`. ```bash cat <<EOF | sudo tee -a /etc/gitlab-runner/config.toml listen_address = "[::]:8093" advertise_address = "[2001:678:3fc:VVV:baad:caff:fefe:XXX]:8093" EOF ``` :warning: Edit the IPv6 advertised address in the command above to match that of your own DevNet virtual machine. After editing the configuration file, the service must be restarted and checked to ensure it is listening on port 8093. ```bash sudo systemctl restart gitlab-runner.service ``` ```bash sudo lsof -i tcp:8093 ``` ```bash= COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME gitlab-ru 8871 root 3u IPv6 42430 0t0 TCP *:8093 (LISTEN) ``` To create a new runner, access the GitLab web interface. The runner creation process will provide a unique token, which will be used to register the gitlab-runner service to the GitLab instance on the DevNet virtual machine. ## Step 3: Create a new runner from the GitLab website The standard practice is to **create a runner at the group level**. This allows the runner to be shared among all projects (i.e. Git repositories) that belong to the group. We start by selecting a group and then create a new runner for this group. 1. Select a group  2. From the left panel, select Runners  3. Complete the new group runner form In our context, we have selected the following form items: - Linux as the runner platform - Run untagged jobs - Runner description - Maximum job timeout set to 600 Click on the option **Create runner** located at the bottom of the page..  3. Create runner The key information on this page is the token that establishes the relationship between the GitLab service and the runner on the DevNet virtual machine.  ## Step 4: Register the DevNet runner to the GitLab service Return to the DevNet virtual machine and execute the `gitlab-runner` command to finalize the registration process. ```bash sudo gitlab-runner register --non-interactive \ --executor "shell" \ --url "https://gitlab.inetdoc.net/" \ --token "glrt-XXXXXXXXXXXXXXXXXXXX" ``` ```bash= Runtime platform arch=amd64 os=linux pid=35760 revision=782c6ecb version=16.9.1 Running in system-mode. Verifying runner... is valid runner=Hc23EDu4b Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml" ``` After completing the registration, we can verify on the **runners page** that the relationship has been correctly established.  # Part 2: Setup a first GitLab CI test pipeline Now that the relationship between the GitLab instance and its runner has been established, we are ready to test a first **Continuous Integration** (CI) *atomic* pipeline. Following the [Lab 1](https://md.inetdoc.net/s/f-mfjs-kQ) scenario we will run the Ansible ad-hoc command with the ping module to verify connectivity between the DevNet virtual machine and the Hypervisor. ## Step 1: Copy secrets to the gitlab-runner user account As mentioned in the lab introduction, a centralized identity and secret management solution is not being used to limit architecture complexity. All identities and secrets are stored on the DevNet virtual machine. In the **Continuous Integration** pipeline context, the `ansible` command is executed using the `gitlab-runner` user account. Therefore, the vault and its password must be copied from the user account used during the initial development phase (ie. the [Lab 1](https://md.inetdoc.net/s/f-mfjs-kQ)) to the `gitlab-runner` user account. Here are the instructions: ```bash= echo "ThisVaultSecret" >$HOME/.vault.passwd chmod 600 $HOME/.vault.passwd sudo cp $HOME/iac_lab02_passwd.yml \ $HOME/.vault.passwd \ /home/gitlab-runner/ sudo chown gitlab-runner:gitlab-runner \ /home/gitlab-runner/iac_lab02_passwd.yml \ /home/gitlab-runner/.vault.passwd ``` - The initial two commands save the secret for opening the vault in the `.vault.passwd` file with limited access permissions. - The next command copy both the vault file and its secret file to the `gitlab-runner` user home directory. - Finally, these two files must be owned by the `gitlab-runner` user. To complete the task of transferring identity and secrets, it is necessary to mention that the shell executor will use the `ANSIBLE_VAULT_PASSWORD_FILE` variable to access the vault password. These are the instructions that set this variable for each new shell opening: ```bash= echo "export ANSIBLE_VAULT_PASSWORD_FILE=\$HOME/.vault.passwd" |\ sudo tee /home/gitlab-runner/.profile sudo chown gitlab-runner:gitlab-runner /home/gitlab-runner/.profile ``` - The `ANSIBLE_VAULT_PASSWORD_FILE` is saved in the `.profile` under the `gitlab-runner` user home directory. - The ownership of the `.profile` file, which contains commands that are executed each time a shell is opened, must be assigned to the `gitlab-runner` user. ## Step 2: Create a first .gitlab-ci.yml file For this initial test of Continuous Integration, we have created a minimal `.gitlab-ci.yml` file. This will prove that integration is properly configured and running. ```yaml= stages: - ping hypervisors-ping: stage: ping script: - ansible hypervisors -m ping --extra-vars @$HOME/iac_lab02_passwd.yml ``` What are the stages of a CI pipeline in GitLab? : Stages in a GitLab CI pipeline act as organizational units that group related jobs together. They play a key role in controlling the flow and execution order of your CI/CD process. In this first CI test, we only have one single stage called **ping**. What are the jobs of a CI pipeline in GitLab? : Jobs are the smallest executable units in a GitLab CI pipeline. They represent individual, atomic tasks that execute specific actions within a CI/CD process. In our context, there is a single job named **hypervisors-ping** that belongs to the **ping** stage and calls Ansible using the **script** command. ## Step 3: Continuous Integration pipeline execution After committing and pushing the `.gitlab-ci.yml` to the Git repository, the runner on the DevNet virtual machine executes the pipeline jobs. The results are available on the GitLab web service. From the **Lab02** project page left panel, select **Pipelines** under the **Build** main item.  Under the **Build** section, selecting **Jobs** will display the results of the Ansible command.  The same results are also available through the VSCode GitLab extension panel.  Now that we have successfully executed a first continuous integration job, we can go further by adding stages and running Ansible playbooks as jobs. ## Step4: Set up a new SSH configuration file for the gitlab-runner user account As we use IPv6 link local addresses to manage each virtual machine or virtual router, there may be address conflicts with SSH connections from the gitlab-runner user account. When a virtual machine is reinstantiated, its SSH server keys are renewed, even though its management interface link local address was previously used and stored in the SSH `known_hosts` file. This causes an error each time a new SSH connection is attempted. To overcome this problem, we need to specify that IPv6 link local addresses must not be saved when an SSH connection is made to a new virtual machine or router. This is done by adding instructions to the `.ssh/config` file of the gitlab-runner user. ```bash= sudo mkdir -p /home/gitlab-runner/.ssh sudo chmod 700 /home/gitlab-runner/.ssh sudo touch /home/gitlab-runner/.ssh/config cat << EOF | sudo tee -a /home/gitlab-runner/.ssh/config Host fe80::* CheckHostIP no StrictHostKeyChecking no UserKnownHostsFile=/dev/null EOF ``` With these configuration instructions in place, we won't encounter SSH connection errors during the **Gathering Facts** playbook task. # Part 3: Build a pipeline with Ansible playbooks The next step is to execute all the Ansible playbooks created in [Lab 1](https://md.inetdoc.net/s/f-mfjs-kQ) by defining new **stages** and **jobs** in the `.gitlab-ci.yml` file. This will enable us to achieve full automation of virtual machine creation, completing our Infrastructure as Code scenario. ## Step 1: Add a new prepare stage to the pipeline As a reminder, the purpose of the `prepare.yml` Ansible playbook in [Lab 1](https://md.inetdoc.net/s/f-mfjs-kQ#Step-1-Prepare-directories-on-the-Hypervisor) is to set up all the required directories and symlinks for running virtual machines on Hypervisors. In addition, it checks the configuration of switch ports that will be utilized by the virtual machines that are to be created. In this section, we add a new stage named **hypervisors-prepare** to the `.gitlab-ci.yml` file with a dependency relationship to the previous **hypervisors-ping** stage. Here is a copy of the new version of the `.gitlab-ci.yml` file: ```yaml= stages: - ping - prepare hypervisors-ping: stage: ping script: - ansible hypervisors -m ping --extra-vars @$HOME/iac_lab02_passwd.yml hypervisors-prepare: stage: prepare needs: - hypervisors-ping script: - ansible-playbook prepare.yml --extra-vars @$HOME/iac_lab02_passwd.yml ``` What is the role of the `needs` keyword in a job? : This feature enables the specification of job dependencies, where a job must successfully complete (or sometimes be allowed to fail) other jobs before it can begin execution. This dependency management helps prevent downstream jobs from failing due to missing artifacts or incomplete tasks from earlier jobs. It leads to a more reliable and predictable CI pipeline execution. The GitLab web service provides a graphical illustration of this dependency relationship.  Following the same process we add two more stages called **vms_build** and **vms_add_vlan** to the `.gitlab-ci.yml` file. And the last one fails. ## Step 2: Illustrate pipeline failure without artifacts As we can read about DevOps paradigms: "Failure is normal!". > The key pillar of DevOps Philosophy is to accept failures as normal. It reveals that failures are normal and should be expected. Organizations should consider it another opportunity to strengthen their infrastructure and build it even better. So what about the failure of our own pipeline? Here is a copy of the `.gitlab_ci.yml` file with the new stages added. ```yaml= stages: - ping - prepare - pull_customize_run - vms_setup hypervisors-ping: stage: ping script: - ansible hypervisors -m ping --extra-vars @$HOME/iac_lab02_passwd.yml hypervisors-prepare: stage: prepare needs: - hypervisors-ping script: - ansible-playbook prepare.yml --extra-vars @$HOME/iac_lab02_passwd.yml vms_build: stage: pull_customize_run needs: - hypervisors-prepare script: - ansible-playbook pull_customize_run.yml --extra-vars @$HOME/iac_lab02_passwd.yml vms_add_vlan: stage: vms_setup needs: - vms_build script: - ansible-playbook add_vlan.yml --extra-vars @$HOME/iac_lab02_passwd.yml ``` The last stage **vms_add_vlan** fails on SSH connections to the newly created virtual machines. How to find error messages?  One way is to click on the failed stage on the GitLab pipeline web page.  Another way is to use the VSCode extension to display the same error messages.  Error messages indicate that the virtual machine names are not resolved and SSH connection attempts fail. The VScode screenshot above shows that the `lab.yaml` inventory file is missing from the `inventory` directory. The first guess is that the dynamic inventory build Python script may have failed during the **vms_build** stage. In fact, this assumption is wrong. Let's try to find out what happened by looking at the job console results. ```shell= Reinitialized existing Git repository in /home/gitlab-runner/builds/Hc23EDu4b/0/iac/lab02/.git/ Checking out 537e08a2 as detached HEAD (ref is main)... Removing inventory/lab.yml Removing trace/launch_output.save Removing trace/launch_output.txt ``` Here we can see that files have been deleted in the early steps of the **vms_add_vlan** job. So when the Ansible playbook runs, the inventory is incomplete because the `inventory/lab.yml` file was deleted before it started. The current state of our pipeline is that there is no retention of files and directories created during a job. This leads to the concept of *artifacts*. What are artifacts in a GitLab continuous integration (CI) pipeline? : Artifacts function as a mechanism to store and share files generated during the pipeline execution. These artifacts, typically files or directories, are saved on the GitLab server after a specific job finishes running. Subsequent jobs within the pipeline can then download and utilize these artifacts. To fix our issue, we need to add an `artifacts` list to the **vms_build** job that shares the `inventory` directory contents with other jobs. The new version of **vms_build** can be: ```yaml= vms_build: stage: pull_customize_run artifacts: paths: - inventory needs: - hypervisors-prepare script: - ansible-playbook pull_customize_run.yml --extra-vars @$HOME/iac_lab02_passwd.yml ``` ## Step 3: Run the complete CI pipeline To conclude this lab, we will successfully run the complete GitLab continuous integration pipeline. Here is a screenshot of the GitLab website showing that all stages have been successfully completed.  Here is another screenshot showing that the job artifacts are stored on the GitLab server.  And, of course, the two new virtual machines are now fully functional and available to start a brand new lab.