# IaC Lab 2 -- Using GitLab CI to run Ansible playbooks and build new Debian GNU/Linux Virtual Machines [toc] --- > Copyright (c) 2026 Philippe Latu. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". https://inetdoc.net GitLab repository: https://gitlab.inetdoc.net/iac/lab02/ ### Scenario This lab will introduce you to the basics of using GitLab CI/CD to automate the building of Debian GNU/Linux virtual machines through a pipeline. We move from the development area ([Lab 01](https://md.inetdoc.net/s/f-mfjs-kQ)) to the testing area (this lab) since all Ansible playbooks are available in a Git repository stored on a GitLab instance. We took a questionable architectural shortcut here. To avoid using multiple cloud services, we chose a minimal GitLab CI/CD setup with a shell runner and the following two constraints: - Ansible vault secrets are already stored on the DevNet virtual machine user account and we do not use a cloud provider. - A playbook generates the dynamic inventory from the JSON-formatted VM launch output messages, so it does not rely on any external service.  ### Objectives After completing the manipulation steps in this document, you will be able to: - Install and configure GitLab Runner on the DevNet virtual machine to enable CI/CD pipeline execution. - Create and test a basic `.gitlab-ci.yml` file to ensure proper integration between GitLab and the runner. - Set up a multi-stage CI pipeline that incorporates Ansible playbooks for automating virtual machine creation and configuration. - Implement artifact management to ensure proper file retention and sharing between pipeline stages. - Successfully run a complete CI pipeline that automates the entire process of creating and configuring Debian GNU/Linux virtual machines. ## Part 1: Install and configure GitLab Runner on the DevNet virtual machine The diagram above illustrates the need to establish communication between two different entities: the GitLab service, which is hosted on a separate machine from the hypervisor, and the GitLab runner, which must be installed on the DevNet virtual machine. First, we will install and configure the GitLab runner. Next, we will create a new runner using the GitLab web interface. The **token** generated during this process will be required to register this new runner. ### Step 1: Install gitlab-runner package The GitLab Runner is available as a Debian package. To install it, a Debian package repository must be added to the sources along with a signing key to check package integrity. 1. Download and save the GPG public signing key ```bash= curl -fsSL https://packages.gitlab.com/runner/gitlab-runner/gpgkey |\ sudo gpg --dearmor -o /usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg ``` 2. Add the new repository source file ```bash= echo "deb [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] \ https://packages.gitlab.com/runner/gitlab-runner/debian/ trixie main" |\ sudo tee /etc/apt/sources.list.d/runner_gitlab-runner.list ``` 3. Update package list and install gitlab-runner ```bash= sudo apt update && sudo apt -y install gitlab-runner ``` 4. Check gitlab-runner service status ```bash systemctl status gitlab-runner.service ``` ```bash= systemctl status gitlab-runner.service ● gitlab-runner.service - GitLab Runner Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; preset: enabled) Active: active (running) since Sun 2026-03-22 09:07:45 CET; 53s ago Invocation: 5c42a7f0136f4ead9e4c6c9547206354 Main PID: 16190 (gitlab-runner) Tasks: 12 (limit: 9422) Memory: 23.1M (peak: 23.8M) CPU: 150ms CGroup: /system.slice/gitlab-runner.service └─16190 /usr/bin/gitlab-runner run --config /etc/gitlab-runner/config.toml --working-directory /home/gitlab-runner --service gitlab-runner --user gitlab-runner mars 22 09:07:46 devnet26 gitlab-runner[16190]: Runtime platform arch=amd64 os=linux pid=16190 revision=ac71f4d8 version=18.10.0 mars 22 09:07:46 devnet26 gitlab-runner[16190]: Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0 max_builds=0 mars 22 09:07:46 devnet26 gitlab-runner[16190]: Running in system-mode. mars 22 09:07:46 devnet26 gitlab-runner[16190]: mars 22 09:07:46 devnet26 gitlab-runner[16190]: Created missing unique system ID system_id=s_7b7ec03a419a mars 22 09:07:46 devnet26 gitlab-runner[16190]: Usage logger disabled builds=0 max_builds=1 mars 22 09:07:46 devnet26 gitlab-runner[16190]: Configuration loaded builds=0 max_builds=1 mars 22 09:07:46 devnet26 gitlab-runner[16190]: listen_address not defined, metrics & debug endpoints disabled builds=0 max_builds=1 mars 22 09:07:46 devnet26 gitlab-runner[16190]: [session_server].listen_address not defined, session endpoints disabled builds=0 max_builds=1 mars 22 09:07:46 devnet26 gitlab-runner[16190]: Initializing executor providers builds=0 max_builds=1 ``` The results of the above commands give us some interesting information about the GitLab Runner service: - The service is up and running - It runs under the user identity `gitlab-runner` - The service configuration file is: `/etc/gitlab-runner/config.toml` - The **listen_address** is not defined. So we need to define it! ### Step 2: Configure gitlab-runner service listen and advertise addresses The initial content of the **gitlab-runner** service configuration file is as follows: ```bash sudo cat /etc/gitlab-runner/config.toml ``` ```toml= concurrent = 1 check_interval = 0 shutdown_timeout = 0 [session_server] session_timeout = 1800 ``` We must add two new address entries in the `[session_server]` category. For convenience, we choose the DevNet IPv6 address as our `advertise_address`. ```bash cat <<EOF | sudo tee -a /etc/gitlab-runner/config.toml listen_address = "[::]:8093" advertise_address = "[2001:678:3fc:VVV:baad:caff:fefe:XXX]:8093" EOF ``` :::warning Edit the IPv6 advertised address in the above command to match that of your own DevNet virtual machine. ::: After editing the configuration file, you must restart the service and verify that it is listening on port 8093. ```bash sudo systemctl restart gitlab-runner.service ``` ```bash sudo lsof -i tcp:8093 ``` ```bash= COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME gitlab-ru 16611 root 3u IPv6 826382 0t0 TCP *:8093 (LISTEN) ``` ### Step 3: Create a new runner from the GitLab web interface To create a new runner, access the GitLab web interface. The runner creation process **provides a unique token** that is used to register the gitlab-runner service with the GitLab instance on the DevNet virtual machine. The standard practice is to **create a runner at the group level**. This allows the runner to be shared by all the projects (i.e. Git repositories) that belong to the group. We will start by selecting a group and then create a new runner for that group. 1. Select a group  2. From the left panel, select Runners  3. Complete the new group runner form :::info In our context, we choose the following form elements: - Linux as the runner execution platform - Run untagged jobs - Runner description - Maximum job timeout set to 600 ::: Click on the option **Create runner** located at the bottom of the page..  4. Create runner The most important information on this page is the token that establishes the relationship between the GitLab service and the runner on the DevNet virtual machine.  ### Step 4: Register the DevNet runner to the GitLab service Return to the DevNet virtual machine and execute the `gitlab-runner` command to finalize the registration process. ```bash sudo gitlab-runner register --non-interactive \ --executor "shell" \ --url "https://gitlab.inetdoc.net/" \ --token "glrt-XXXXXXXXXXXXXXXXXXXX" ``` ```bash= Runtime platform arch=amd64 os=linux pid=16965 revision=ac71f4d8 version=18.10.0 Running in system-mode. Verifying runner... is valid correlation_id=XXXXXXXXXXXXXXXXXXXXXXXXXX runner=XXXXXXXXX runner_name=devnet26 Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml" ``` Once the registration is complete, we can verify that the relationship has been correctly established on the **Runners Page**.  ## Part 2: Set up a first GitLab CI test pipeline Now that the relationship between the GitLab instance and its runner has been established, we are ready to test a first **Continuous Integration** (CI) *atomic* pipeline. Following the [**Lab 01**](https://md.inetdoc.net/s/f-mfjs-kQ) scenario we will run the Ansible ad hoc command with the ping module to verify connectivity between the DevNet virtual machine and the hypervisor. ### Step 1: Set up the Ansible toolchain for the gitlab-runner user Just as you configured your developer account in [**Lab 01**](https://md.inetdoc.net/s/f-mfjs-kQ), you now need to enable the `gitlab-runner` user to run Ansible playbooks. Create and execute the following `setup_ansible_toolchain.sh` script to build its environment. ```bash= #!/usr/bin/env bash # Purpose: Sets up 'uv' and an Ansible virtual environment for the current user. # This script should be executed directly by the target user (e.g., developer or gitlab-runner). set -euo pipefail if [[ ${USER} == "gitlab-runner" ]]; then readonly PROJECT_DIR="${HOME}" readonly VENV_DIR="${PROJECT_DIR}/.venv" else readonly PROJECT_DIR="${PWD}" readonly VENV_DIR="${PROJECT_DIR}/.venv" fi readonly PROJECT_FILE="pyproject.toml" echo "Starting Ansible toolchain setup for user: ${USER}" # 1. Verify project metadata exists cd "${PROJECT_DIR}" if [[ ! -f ${PROJECT_FILE} ]]; then cat <<EOF >"${PROJECT_DIR}/${PROJECT_FILE}" [project] name = "lab02" version = "0.1.0" description = "IaC Ansible labs" readme = "README.md" requires-python = ">=3.13" dependencies = ["ansible>=13.4.0", "ansible-lint>=26.3.0"] EOF else echo "${PROJECT_FILE} found in the ${PROJECT_DIR} directory." fi # 2. Install 'uv' locally for the user if not present if ! command -v uv >/dev/null 2>&1; then echo "Installing 'uv'..." curl -LsSf https://astral.sh/uv/install.sh | sh # Add standard astral installation paths to PATH for immediate use export PATH="${HOME}/.local/bin:${PATH}" else echo "'uv' is already installed." fi # 3. Create the virtual environment only if it does not already exist if [[ -d ${VENV_DIR} ]]; then echo "Existing virtual environment detected at ${VENV_DIR}." else echo "Creating new virtual environment at ${VENV_DIR}..." uv venv "${VENV_DIR}" fi # 4. Activate the target virtual environment if needed, then sync project dependencies if [[ ${VIRTUAL_ENV-} == "${VENV_DIR}" ]]; then echo "Virtual environment is already activated. Syncing dependencies from ${PROJECT_FILE}..." else echo "Activating virtual environment at ${VENV_DIR}..." # shellcheck source=/dev/null source "${VENV_DIR}/bin/activate" fi echo "Upgrading dependencies from ${PROJECT_FILE}..." uv sync --active --upgrade echo "--------------------------------------------------------" echo "Toolchain setup complete! Virtual environment is at:" echo "${VENV_DIR}" echo "To activate manually, run: source ${VENV_DIR}/bin/activate" echo "--------------------------------------------------------" ``` The `setup_ansible_toolchain.sh` script sets up a uv-managed Python virtual environment and installs the Ansible toolchain for the current user. It is designed to be run by either a developer or the gitlab-runner service account without modification. It performs the following tasks: Detect the execution context: : If the script is run as the gitlab-runner user, PROJECT_DIR is set to $HOME so that the virtual environment is installed in a stable, session-independent location. For any other user, PROJECT_DIR defaults to the current working directory. Verify or create the project metadata file: : The script changes to `PROJECT_DIR` and checks whether a `pyproject.toml` file already exists. If not, it generates a minimal one that declares the project dependencies, namely **ansible** and **ansible-lint**, with their required minimum versions. Install `uv` if not already present: : The script checks whether the **`uv`** package manager is available on `PATH`. If it is missing, it downloads and installs it from the official Astral installer and immediately adds it to the current session's `PATH`. Create the Python virtual environment: : If no virtual environment exists yet at `${PROJECT_DIR}/.venv`, the script creates one using uv venv. If one is already present, it reports that and skips creation. Activate the virtual environment and synchronise dependencies: : If the virtual environment is not already active, the script sources its activation script. It then runs `uv sync --active --upgrade` to install or upgrade all declared dependencies from `pyproject.toml`. When the script file is created just run it with the `gitlab-runner` user identity. ```bash sudo -u gitlab-runner bash setup_ansible_toolchain.sh ``` ```bash= Starting Ansible toolchain setup for user: gitlab-runner Installing 'uv'... downloading uv 0.10.12 x86_64-unknown-linux-gnu no checksums to verify installing to /home/gitlab-runner/.local/bin uv uvx everything's installed! Creating new virtual environment at /home/gitlab-runner/.venv... Using CPython 3.13.12 interpreter at: /usr/bin/python3 Creating virtual environment at: .venv Activate with: source .venv/bin/activate Activating virtual environment at /home/gitlab-runner/.venv... Upgrading dependencies from pyproject.toml... Resolved 33 packages in 756ms Prepared 31 packages in 4.63s Installed 31 packages in 299ms + ansible==13.4.0 + ansible-compat==25.12.1 + ansible-core==2.20.3 + ansible-lint==26.3.0 + attrs==26.1.0 + black==26.3.1 + bracex==2.6 + cffi==2.0.0 + click==8.3.1 + cryptography==46.0.5 + distro==1.9.0 + filelock==3.25.2 + jinja2==3.1.6 + jsonschema==4.26.0 + jsonschema-specifications==2025.9.1 + markupsafe==3.0.3 + mypy-extensions==1.1.0 + packaging==26.0 + pathspec==1.0.4 + platformdirs==4.9.4 + pycparser==3.0 + pytokens==0.4.1 + pyyaml==6.0.3 + referencing==0.37.0 + resolvelib==1.2.1 + rpds-py==0.30.0 + ruamel-yaml==0.19.1 + ruamel-yaml-clib==0.2.15 + subprocess-tee==0.4.2 + wcmatch==10.1 + yamllint==1.38.0 -------------------------------------------------------- Toolchain setup complete! Virtual environment is at: /home/gitlab-runner/.venv To activate manually, run: source /home/gitlab-runner/.venv/bin/activate -------------------------------------------------------- ``` With the Ansible toolchain installed and the virtual environment activated under the `gitlab-runner` account, the runner is now ready to execute Ansible playbooks as part of a GitLab CI pipeline. ### Step 2: Copy secrets to the gitlab-runner user account As mentioned in the lab introduction, a centralized identity and secrets management solution is not used to limit the complexity of the architecture. All identities and secrets are stored in the DevNet virtual machine. In the context of the **continuous integration pipeline**, the `ansible` command is executed using the `gitlab-runner` user account. Therefore, the vault and its password must be copied from the user account used during the initial development phase (i.e., the [**Lab 01**](https://md.inetdoc.net/s/f-mfjs-kQ)) to the `gitlab-runner` user account. Here is a copy of the [**share_secrets.sh**](https://gitlab.inetdoc.net/iac/lab02/-/blob/main/share_secrets.sh) Bash script: ```bash= #!/usr/bin/env bash # The purpose of this script is to copy secrets from the developer's home # directory to the GitLab Runner's home directory, and inject the runner's # SSH public key into the Ansible Vault. set -euo pipefail # Constants readonly HYPERVISOR_NAME="eve" readonly DEV_USER="etu" readonly RUNNER_USER="gitlab-runner" readonly RUNNER_HOME="/home/${RUNNER_USER}" readonly DEV_HOME="/home/${DEV_USER}" readonly RUNNER_SSH_DIR="${RUNNER_HOME}/.ssh" readonly VAULT_PASSWD_FILE=".vault_pass.txt" readonly VAULT_FILE=".iac_passwd" readonly ANSIBLE_VENV="${RUNNER_HOME}/.venv" # Function to handle errors error_exit() { echo "ERROR: $1" >&2 exit 1 } # Function to copy and set permissions securely in one atomic step secure_copy() { local src="$1" local dest="$2" local mode="${3:-0600}" [[ -f ${src} ]] || error_exit "Source file ${src} not found" install -m "${mode}" -o "${RUNNER_USER}" -g "${RUNNER_USER}" "${src}" "${dest}" } # Function to safely append the runner's public key to the Ansible Vault update_vault_runner_key() { local vault_file="$1" local pass_file="$2" local pub_key="$3" local vault_bin="${ANSIBLE_VENV}/bin/ansible-vault" local tmp_file # Fallback to system ansible-vault if the venv isn't found if [[ ! -x ${vault_bin} ]]; then vault_bin="$(command -v ansible-vault || true)" [[ -n ${vault_bin} ]] || error_exit "ansible-vault not found. Please run the toolchain setup script first." fi tmp_file="$(mktemp)" echo "Decrypting vault to check for vm_runnerkey..." "${vault_bin}" view --vault-password-file "${pass_file}" "${vault_file}" >"${tmp_file}" || { rm -f "${tmp_file}" error_exit "Failed to decrypt ${vault_file}" } # Check if the key already exists if ! grep -qE '^[[:space:]]*vm_runnerkey[[:space:]]*:' "${tmp_file}"; then # JSON-escape the value cleanly local escaped_key escaped_key="$(printf '%s' "${pub_key}" | sed 's/\\/\\\\/g; s/"/\\"/g')" printf '\nvm_runnerkey: "%s"\n' "${escaped_key}" >>"${tmp_file}" # Re-encrypt replacing the original file "${vault_bin}" encrypt "${tmp_file}" \ --vault-password-file "${pass_file}" \ --output "${vault_file}" || { rm -f "${tmp_file}" error_exit "Failed to re-encrypt ${vault_file}" } # Restore proper ownership and permissions since root overwrote it chown "${RUNNER_USER}":"${RUNNER_USER}" "${vault_file}" chmod 0600 "${vault_file}" echo "vm_runnerkey successfully added to the vault." else echo "vm_runnerkey is already present in the vault. No changes made." fi rm -f "${tmp_file}" } # Check if running as root [[ ${EUID} -eq 0 ]] || error_exit "This script must be run as root" echo "Setting up secrets for GitLab Runner..." # Setup Vault files secure_copy "${DEV_HOME}/${VAULT_PASSWD_FILE}" "${RUNNER_HOME}/${VAULT_PASSWD_FILE}" secure_copy "${DEV_HOME}/${VAULT_FILE}" "${RUNNER_HOME}/${VAULT_FILE}" # Setup profile PROFILE="${RUNNER_HOME}/.profile" echo "export ANSIBLE_VAULT_PASSWORD_FILE=${RUNNER_HOME}/${VAULT_PASSWD_FILE}" >"${PROFILE}" chown "${RUNNER_USER}":"${RUNNER_USER}" "${PROFILE}" chmod 0644 "${PROFILE}" # Setup SSH directory using install (-d flag creates directories) install -d -m 0700 -o "${RUNNER_USER}" -g "${RUNNER_USER}" "${RUNNER_SSH_DIR}" # Copy SSH configuration file if [[ -f "${DEV_HOME}/.ssh/config" ]]; then secure_copy "${DEV_HOME}/.ssh/config" "${RUNNER_SSH_DIR}/config" 0644 fi # Enable SSH passwordless configuration echo "Setting up dedicated SSH identity for gitlab-runner..." if [[ ! -f "${RUNNER_SSH_DIR}/id_ed25519" ]]; then # Run ssh-keygen as the gitlab-runner user su - "${RUNNER_USER}" -c "ssh-keygen -t ed25519 -C 'gitlab-runner@devnet-ci' -f ${RUNNER_SSH_DIR}/id_ed25519 -N ''" echo "New SSH key pair generated for gitlab-runner." else echo "SSH key pair already exists for gitlab-runner. Skipping generation." fi # Read the generated public key into a bash variable RUNNER_PUB_KEY=$(cat "${RUNNER_SSH_DIR}/id_ed25519.pub") # Inject the public key into the runner's Ansible Vault update_vault_runner_key "${RUNNER_HOME}/${VAULT_FILE}" "${RUNNER_HOME}/${VAULT_PASSWD_FILE}" "${RUNNER_PUB_KEY}" # Push the public key to the hypervisor using the developer's existing SSH access echo "Authorizing gitlab-runner on the hypervisor (${HYPERVISOR_NAME})..." # Execute SSH as the developer, explicitly passing the config (-F) and identity (-i) files. # Use curly braces { } to group the grep/echo logic safely. REMOTE_CMD="mkdir -p ~/.ssh && chmod 700 ~/.ssh" REMOTE_CMD+=" && { grep -q -F \"${RUNNER_PUB_KEY}\" ~/.ssh/authorized_keys" REMOTE_CMD+=" || echo \"${RUNNER_PUB_KEY}\" >> ~/.ssh/authorized_keys; }" REMOTE_CMD+=" && chmod 600 ~/.ssh/authorized_keys" sudo -u "${DEV_USER}" ssh \ -F "${DEV_HOME}/.ssh/config" \ -i "${DEV_HOME}/.ssh/id_ed25519" \ -o StrictHostKeyChecking=accept-new \ "${HYPERVISOR_NAME}" \ "${REMOTE_CMD}" # Verify that the key was added successfully sudo -u "${RUNNER_USER}" ssh \ -o StrictHostKeyChecking=accept-new \ "${HYPERVISOR_NAME}" \ 'echo Key is working!' echo "Passwordless SSH access established successfully!" exit 0 ``` This script copies authentication and configuration files from the developer account to the GitLab Runner account, injects the runner's SSH public key into the Ansible Vault, and configures passwordless SSH access for the runner on the hypervisor. #### Requirements - Must be run as root. - Developer account must exist. - GitLab Runner must be installed and using the gitlab-runner user. - The Ansible toolchain (virtual environment) must already be installed. #### What the script does 1. Copies vault files: - `.iac_passwd`: the Ansible vault file. - `.vault_pass.txt`: the vault password file. 2. Sets up the environment for Ansible: - Creates a `.profile` file for `gitlab-runner` that exports `ANSIBLE_VAULT_PASSWORD_FILE` with the correct path. - Ensures appropriate ownership and permissions on the profile and vault files. 3. Creates and secures the SSH environment for gitlab-runner: - Creates the .ssh directory with restrictive permissions. - Copies the SSH config file from the developer account if it exists. 4. Generates a dedicated SSH key pair for gitlab-runner: - If no `id_ed25519` key exists, generates a new ed25519 key pair as gitlab-runner (non-interactively, empty passphrase). - If a key already exists, keeps the existing identity and skips regeneration. 5. Injects the gitlab-runner public key into the Ansible Vault: - Decrypts the vault file (`.iac_passwd`) using ansible-vault view to check whether a vm_runnerkey variable is already present. - If the key is absent, appends a new `vm_runnerkey` variable containing the runner's ed25519 public key, then re-encrypts the vault in place using ansible-vault encrypt. - Restores correct ownership (`gitlab-runner`) and permissions (`0600`) on the vault file after re-encryption. - If the key is already present, skips the update to ensure idempotency. 6. Enables SSH passwordless access to the hypervisor: - Uses the developer’s existing SSH access to append the runner’s public key to `~/.ssh/authorized_keys` on the hypervisor, creating and securing the file if needed. - Ensures idempotency by adding the key only if it is not already present. - Verifies that `gitlab-runner` can log in to the hypervisor via SSH without a password. #### Security measures - Uses strict file modes (`0600` for sensitive files, `0700` for .ssh directories). - Sets correct ownership for all files and directories to the `gitlab-runner` user. - Uses a dedicated SSH key pair for `gitlab-runner`, avoiding reuse of the developer’s private key. - Fails fast with clear error messages on missing files or incorrect usage. #### Usage ```bash sudo bash ./share_secrets.sh ``` ```bash= Setting up secrets for GitLab Runner... Setting up dedicated SSH identity for gitlab-runner... SSH key pair already exists for gitlab-runner. Skipping generation. Decrypting vault to check for vm_runnerkey... Encryption successful vm_runnerkey successfully added to the vault. Authorizing gitlab-runner on the hypervisor (hypervisor_name)... Key is working! Passwordless SSH access established successfully! ``` ### Step 3: Create a first .gitlab-ci.yml file For this initial test of Continuous Integration, we have to create a minimal [**.gitlab-ci.yml**](https://gitlab.inetdoc.net/iac/lab02/-/blob/main/.gitlab-ci-00.yml?ref_type=heads) file. This will prove that the integration is properly configured and running. ```yaml= variables: VAULT_FILE: /home/gitlab-runner/.iac_passwd default: before_script: # Inject the persistent virtual environment into the execution path # This automatically activates Ansible for all jobs below - export PATH="/home/gitlab-runner/.venv/bin:$PATH" stages: - Ping Ping hypervisors: stage: Ping script: - ansible hypervisors -m ping --extra-vars "@${VAULT_FILE}" ``` The key design decision here is to share a persistent virtual environment across all pipeline jobs. Rather than rebuilding the Python virtual environment on every pipeline run, the `setup_ansible_toolchain.sh` script ([Step 1](https://md.inetdoc.net/#Step-1-Set-up-the-Ansible-toolchain-for-the-gitlab-runner-user)) pre-installs it once at `/home/gitlab-runner/.venv`. The default: `before_script` block then injects its `bin/` directory into `PATH` before every job, making Ansible and all other tools immediately available without any build step. What are the stages of a CI pipeline in GitLab? : Stages in a GitLab CI pipeline act as organizational units that group related tasks together. They play a key role in controlling the flow and execution order of your CI/CD process. In this first CI test, we only have one stage called **Ping**. What are the jobs of a CI pipeline in GitLab? : Jobs are the smallest executable units in a GitLab CI pipeline. They represent individual, atomic tasks that perform specific actions within a CI/CD process. In our context, there is a single job called **Ping** hypervisors, which belongs to the Ping stage and calls Ansible using the **script** command. :::info Make sure that CI/CD is enabled for your GitLab project. Go to project **Settings**, then **Visibility, project features, permissions**, and scroll down to **CI/CD**. Finally, do not forget to **Save changes** at the bottom of the category.  ::: ### Step 4: Continuous Integration pipeline execution After committing and pushing the `.gitlab-ci.yml` to the Git repository, the runner on the DevNet virtual machine executes the pipeline jobs. The results are available via the GitLab web service. From the left panel of the [**Lab02**](https://gitlab.inetdoc.net/iac/lab02) project page, select **Pipelines** under the **Build** main item.  Under the **Build** section, selecting the **Ping** job will display the results of the Ansible command.  The same results are also available through the VSCode GitLab extension panel. ```console= Running with gitlab-runner 18.10.0 (ac71f4d8) on devnet26 tFYHRcNtF, system ID: s_7b7ec03a419a Preparing the "shell" executor 00:00 Using Shell (bash) executor... Preparing environment 00:00 Running on devnet26... Getting source from Git repository 00:01 Gitaly correlation ID: 01KMFAZZB1RHKTRPJ7KNQ9NRC3 Fetching changes with git depth set to 20... Dépôt Git vide initialisé dans /home/gitlab-runner/builds/tFYHRcNtF/0/iac/lab02/.git/ Created fresh repository. Checking out b127e095 as detached HEAD (ref is main)... Skipping Git submodules setup Executing "step_script" stage of the job script 00:01 $ export PATH="/home/gitlab-runner/.venv/bin:$PATH" $ ansible hypervisors -m ping --extra-vars "@${VAULT_FILE}" eve | SUCCESS => { "changed": false, "ping": "pong" } Cleaning up project directory and file based variables 00:01 Job succeeded ``` In this second part, you prepared secrets, defined a minimal two-stage pipeline, and ran an initial Ansible-based CI job with GitLab Runner, giving you a first, concrete experience of GitLab CI in action. You are now ready to extend this foundation with richer pipelines that build and configure new Debian virtual machines automatically in the next parts of the lab. ## Part 3: Build a pipeline with Ansible playbooks The next step is to run all the Ansible playbooks created in [**Lab 01**](https://md.inetdoc.net/s/f-mfjs-kQ) by defining new **stages** and **jobs** in the `.gitlab-ci.yml` file. This will allow us to fully automate the creation of virtual machines and complete our **Infrastructure as Code** scenario. ### Step 1: Add a new prepare stage to the pipeline As a reminder, the purpose of the `01_prepare.yml` Ansible playbook in [**Lab 01**](https://md.inetdoc.net/s/f-mfjs-kQ#Step-1-Preparation-stage-at-the-Hypervisor-level) is to set up all the required directories and symlinks needed to run virtual machines on hypervisors. It also configures the switch ports that will be used by the virtual machines to be created. In this section, we add a new stage called **Prepare** to the `.gitlab-ci.yml` file with a dependency relationship to the previous **Ping** stage. Here is a copy of the new version of the [**.gitlab-ci.yml**](https://gitlab.inetdoc.net/iac/lab02/-/blob/main/.gitlab-ci-01.yml?ref_type=heads) file: ```yaml= variables: VAULT_FILE: /home/gitlab-runner/.iac_passwd default: before_script: # Inject the persistent virtual environment into the execution path # This automatically activates Ansible for all jobs below - export PATH="/home/gitlab-runner/.venv/bin:$PATH" stages: - Ping - Prepare Ping hypervisors: stage: Ping script: - ansible hypervisors -m ping --extra-vars "@${VAULT_FILE}" Prepare lab env: stage: Prepare artifacts: paths: - trace needs: - Ping hypervisors script: - ansible-playbook 01_prepare.yaml --extra-vars "@${VAULT_FILE}" ``` What is the role of the `needs` keyword in a job? : This feature allows us to specify job dependencies, where a job must successfully complete (or sometimes be allowed to fail) other jobs before it can begin execution. This dependency management helps prevent downstream jobs from failing due to missing artifacts or incomplete tasks from earlier jobs. It results in a more reliable and predictable CI pipeline execution. What is the role of the `artifacts` keyword in a job? : Artifacts are files or directories produced by a job that GitLab stores on the server after the job completes. Subsequent jobs within the same pipeline can then download them, enabling state to be shared between stages. Here is a short description of the key tasks: Ping Hypervisors : - Stage: Ping - Task: Checks connectivity to hypervisors using Ansible ping module. Prepare Lab Environment : - Stage: Prepare - Task: Runs an Ansible playbook to set up the lab environment. Security Note : Sensitive information is stored in a vault file (`/home/gitlab-runner/.iac_passwd`) and passed to Ansible using the `--extra-vars` option. The GitLab web service provides a graphical illustration of this dependency relationship.  ### Step 2: Illustrate pipeline failure without artifacts A core DevOps paradigm is that 'Failure is normal!'. Organizations should expect failures and view them as opportunities to strengthen and improve their infrastructure. So what about the failure of our own pipeline? Here is an intentionally broken version of the [**.gitlab-ci.yml**](https://gitlab.inetdoc.net/iac/lab02/-/blob/main/.gitlab-ci-02.yml?ref_type=heads) file with all the stages added. Lines 32 to 36 are commented out to avoid persisting artifacts between "Build and launch vms" and "Configure vms" jobs. ```yaml= variables: VAULT_FILE: /home/gitlab-runner/.iac_passwd default: before_script: # Inject the persistent virtual environment into the execution path # This automatically activates Ansible for all jobs below - export PATH="/home/gitlab-runner/.venv/bin:$PATH" stages: - Ping - Prepare - Pull_customize_run - Configure Ping hypervisors: stage: Ping script: - ansible hypervisors -m ping --extra-vars "@${VAULT_FILE}" Prepare lab env: stage: Prepare artifacts: paths: - trace needs: - Ping hypervisors script: - ansible-playbook 01_prepare.yaml --extra-vars "@${VAULT_FILE}" Build and launch vms: stage: Pull_customize_run # artifacts: # paths: # - inventory # - trace needs: - Prepare lab env script: - ansible-playbook 02_pull_customize_run.yaml --extra-vars "@${VAULT_FILE}" Configure vms: stage: Configure needs: - Build and launch vms script: - ansible-playbook 03_system_bits.yaml --extra-vars "@${VAULT_FILE}" ``` The last **'Configure'** stage fails due to SSH connection errors to the newly created virtual machines. So how do you find the error messages?  One way is to click on the failed stage on the GitLab pipeline web page.  Another way is to use the VSCode extension to view the same error messages. ```console= Running with gitlab-runner 18.10.0 (ac71f4d8) on devnet26 tFYHRcNtF, system ID: s_7b7ec03a419a Preparing the "shell" executor 00:00 Using Shell (bash) executor... Preparing environment 00:00 Running on devnet26... Getting source from Git repository 00:01 Gitaly correlation ID: 01KMFBXEXQZR41SBK9DDFAWFRS Fetching changes with git depth set to 20... Existing Git repository reinitialized in /home/gitlab-runner/builds/tFYHRcNtF/0/iac/lab02/.git/ Checking out 1c2be676 as detached HEAD (ref is main)... Delete inventory/vm_vm4.yaml Delete inventory/vm_vm5.yaml Delete trace/ Skipping Git submodules setup Executing "step_script" stage of the job script 05:12 $ export PATH="/home/gitlab-runner/.venv_ansible/bin:$PATH" $ ansible-playbook 03_system_bits.yaml --extra-vars "@${VAULT_FILE}" PLAY [CONFIGURE SYSTEM BITS AND PIECES] **************************************** TASK [WAIT FOR VMS TO BECOME ACCESSIBLE] *************************************** [ERROR]: Task failed: Action failed: timed out waiting for ping module test: Failed to connect to the host via ssh: ssh: Could not resolve hostname vm4: Name or service not known Origin: /home/gitlab-runner/builds/tFYHRcNtF/0/iac/lab02/03_system_bits.yaml:15:7 13 14 pre_tasks: 15 - name: WAIT FOR VMS TO BECOME ACCESSIBLE ^ column 7 fatal: [vm4]: FAILED! => {"changed": false, "elapsed": 310, "msg": "timed out waiting for ping module test: Failed to connect to the host via ssh: ssh: Could not resolve hostname vm4: Name or service not known"} [ERROR]: Task failed: Action failed: timed out waiting for ping module test: Failed to connect to the host via ssh: ssh: Could not resolve hostname vm5: Name or service not known Origin: /home/gitlab-runner/builds/tFYHRcNtF/0/iac/lab02/03_system_bits.yaml:15:7 13 14 pre_tasks: 15 - name: WAIT FOR VMS TO BECOME ACCESSIBLE ^ column 7 fatal: [vm5]: FAILED! => {"changed": false, "elapsed": 310, "msg": "timed out waiting for ping module test: Failed to connect to the host via ssh: ssh: Could not resolve hostname vm5: Name or service not known"} PLAY RECAP ********************************************************************* vm4 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 vm5 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Cleaning up project directory and file based variables 00:00 ERROR: Job failed: exit status 1 ``` Error messages indicate that the virtual machine names are not resolved and SSH connection attempts fail. The VSCode screenshot above shows that the `vm_vm4.yaml` and `vm_vm5.yaml` inventory files are missing from the `inventory` directory. One might initially assume that the dynamic inventory build failed during the **'Pull_customize_run'** stage. However, this assumption is incorrect. Let's try to find out what happened by looking at the job console results below. ```bash= Getting source from Git repository 00:01 Gitaly correlation ID: 01KMFBXEXQZR41SBK9DDFAWFRS Fetching changes with git depth set to 20... Existing Git repository reset in /home/gitlab-runner/builds/tFYHRcNtF/0/iac/lab02/.git/ Checking out 1c2be676 as detached HEAD (ref is main)... Delete inventory/vm_vm4.yaml Delete inventory/vm_vm5.yaml Delete trace/ ``` Here we can see that files and directories were deleted during the Git workspace initialisation at the start of the **Configure** stage. So when the Ansible playbook runs, the inventory is incomplete because the `inventory/vm_vm4.yaml` and `inventory/vm_vm5.yaml` files were deleted before it started. At this point, the pipeline has no mechanism to retain files and directories produced during a job. What are artifacts in a GitLab Continuous Integration (CI) pipeline? : Artifacts serve as a mechanism for storing and sharing files generated during pipeline execution. These artifacts, typically files or directories, are stored on the GitLab server after a particular job completes. Subsequent jobs within the pipeline can then download and use these artifacts. To fix our problem, we need to add an `artifacts` keyword list to the **Build and launch vms** job, which will share the contents of the `inventory` directory with other jobs. The new version of **Build and launch vms** may be: ```yaml= Build and launch vms: stage: Pull_customize_run artifacts: paths: - inventory - trace needs: - Prepare lab env script: - ansible-playbook 02_pull_customize_run.yaml --extra-vars "@${VAULT_FILE}" ``` ### Step 3: Run the complete CI pipeline At the end of this lab, we will successfully run the complete GitLab continuous integration pipeline. Here is a screenshot of the GitLab website showing that all stages have been successfully completed.  Here are two other screenshots to show access to the artifacts stored in the GitLab service. - Click **Browse** on the job page <img src="https://md.inetdoc.net/uploads/3b481915-639b-4029-9937-7adfcfdfdfde.png" width="240"/> - Develop the inventory folder  And, of course, the two new virtual machines are now fully functional and available to start a brand new lab. ## Conclusion In summary, this lab demonstrates the power of Infrastructure as Code (IaC) and Continuous Integration/Continuous Deployment (CI/CD) in automating the creation and configuration of virtual machines. By leveraging GitLab CI, Ansible playbooks, and a series of well-defined pipeline stages, the process of setting up a lab environment has been streamlined and made repeatable. Key takeaways include: 1. The importance of proper GitLab runner setup and configuration 2. The value of breaking down complex tasks into distinct pipeline stages 3. The critical role of artifacts in maintaining state between jobs 4. The need for error handling and troubleshooting in CI/CD pipelines This lab provides a solid foundation for further exploration of IaC and CI/CD concepts, preparing students for real-world scenarios where these technologies are increasingly essential.