# Start unprivileged Incus containers on top of Open vSwitch in a few steps
[toc]
---
https://inetdoc.net
The main address of this lab text is: https://md.inetdoc.net/s/zL2STyrLf6
> Copyright (c) 2026 Philippe Latu.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
> C-3PO, protocol droid by trade, network engineer by necessity, currently experiencing a fatal ‘missing coffee’ exception.
### Scenario
In this lab, you will build a small Open vSwitch‑backed network on a Debian Trixie host and attach a set of unprivileged Incus containers to it. This environment demonstrates how to separate host connectivity, virtual switching, and container networking while keeping full control over IPv4/IPv6 routing, firewalling, and address management.
The main stages of the scenario are as follows:
1. Install the necessary networking and container tools, including Netplan, Open vSwitch, nftables, dnsmasq and Incus, on a Debian Testing host or virtual machine. The latest version of Incus is installed from the Zabbly repository.
2. Create the Open vSwitch‑based topology by defining the c-3po switch, the vlan10 Switched Virtual Interface, enabling IPv4/IPv6 forwarding, and configuring dnsmasq plus an nftables masquerading ruleset for container egress.
3. Initialise Incus to use C-3PO instead of a local bridge and adapt the default profile so that the container NICs are bridged into VLAN 10. Then, launch the first set of Debian Trixie containers attached to this virtual network.
4. Validate the setup by checking addressing and routing from within the containers, testing East‑West connectivity, automating maintenance commands across running containers, and inspecting Open vSwitch state (ports, neighbours, and CAM table) on the host.
### Objectives
After completing this lab, you will be able to:
- Configure an Open vSwitch‑based Layer 2/3 fabric using Netplan, including a VLAN‑backed Switched Virtual Interface for container traffic.
- Deploy a basic routing, firewall, and DHCP/DNS stack with nftables and dnsmasq to provide IPv4/IPv6 connectivity for containers.
- Install and initialise Incus to consume the existing Open vSwitch bridge, and attach unprivileged containers to a tagged VLAN.
- Validate and troubleshoot the resulting setup by inspecting container connectivity, host routing, and Open vSwitch state (ports, neighbours, CAM entries).
---
## Part 1: Install Netplan and OvS and other tools as a prerequisite
The installation process begins with the base installation of Debian Testing on either a host system or a virtual machine.
:::info
Netplan is now included by default in the images provided at [cloud.debian.org](cloud.debian.org). Therefore, there is no need to install the `netplan.io` package as shown below.
:::
:::info
Having supported students who were struggling to incorporate Open vSwitch into cloud-init stages, I decided to pre-install the `openvswitch-switch` package in the qcow2 image files for the master virtual machines. This means that there is also no need to install the package manually after booting the virtual machine.
:::
```bash
sudo apt -y install netplan.io openvswitch-switch nftables dnsmasq
```
We can verify that the packages have been installed and display the versions of the tools used in this lab.
```bash
apt search ^netplan.io$
```
```bash=
netplan.io/testing,now 1.1.2-8 amd64 [installed]
Declarative network configuration for various backends at runtime
```
```bash
apt search ^openvswitch-switch$
```
```bash=
openvswitch-switch/testing,now 3.6.0-6+b1 amd64 [installed]
Open vSwitch switch implementations
```
```bash
apt search ^nftables$
```
```bash=
nftables/testing,now 1.1.6-1 amd64 [installed]
program for controlling Netfilter project packet filtering rules
```
```bash
apt search ^dnsmasq$
```
```bash=
dnsmasq/testing,now 2.92~rc3-1 all [installed]
small DNS cache proxy and DHCP/TFTP server – system daemon
```
---
## Part 2: Add a new switch and a Switched Virtual Interface (SVI)
Part 2 creates the **c-3po** Open vSwitch bridge and the **vlan10** Switched Virtual Interface, which together provide Layer 2 connectivity and a routed gateway for all attached containers.
Note that the example file uses automatic addressing for the enp0s1 host interface.
You may need to adjust these settings according to your requirements.
* **c-3po** is an access layer switch.
Quote from Star Wars:
> "Don't blame me. I'm an interpreter. I'm not supposed to know a power socket from a computer terminal."
* **vlan10** is a switched virtual interface that feeds the host routing table and acts as the default gateway for all containers
Here is a copy of the `/etc/netplan/10-enp0s1+ovs.yaml` file.
```yaml=
---
network:
version: 2
renderer: networkd
ethernets:
enp0s1:
dhcp4: true
dhcp6: false
accept-ra: true
bridges:
c-3po:
openvswitch: {}
vlans:
vlan10:
id: 10
link: "c-3po"
openvswitch: {}
addresses:
- "192.0.2.1/24"
- "fdc0:7a62:a::1/64"
- "fe80:a::1/64"
```
Run the following command to apply all the network parameters declared in the `/etc/netplan/10-enp0s1+ovs.yaml` file.
```bash
sudo netplan apply
```
Finally, check the status of all configured network interfaces.
```bash
sudo netplan status
```
```bash=
Online state: online
DNS Addresses: 127.0.0.53 (stub)
DNS Search: .
● 1: lo ethernet UNKNOWN/UP (unmanaged)
MAC Address: 00:00:00:00:00:00
Addresses: 127.0.0.1/8
::1/128
● 2: enp0s1 ethernet UP (networkd: enp0s1)
MAC Address: b8:ad:ca:fe:00:01 (Red Hat, Inc.)
Addresses: 198.18.53.1/22 (dynamic, dhcp)
2001:678:3fc:34:baad:caff:fefe:1/64 (dynamic, ra)
fe80::baad:caff:fefe:1/64 (link)
DNS Addresses: 172.16.0.2
2001:678:3fc:3::2
Routes: default via 198.18.52.1 from 198.18.53.1 metric 100 (dhcp)
172.16.0.2 via 198.18.52.1 from 198.18.53.1 metric 100 (dhcp)
198.18.52.0/22 from 198.18.53.1 metric 100 (link)
198.18.52.1 from 198.18.53.1 metric 100 (dhcp, link)
2001:678:3fc:34::/64 metric 100 (ra)
fe80::/64 metric 256
default via fe80::34:1 metric 100 (ra)
● 4: c-3po other UNKNOWN/UP (networkd: c-3po)
MAC Address: 46:2c:c3:2c:02:4c
Addresses: fe80::a807:89ff:fe34:a75b/64 (link)
Routes: fe80::/64 metric 256
● 5: vlan10 other UNKNOWN/UP (networkd: vlan10)
MAC Address: 46:2c:c3:2c:02:4c
Addresses: 192.0.2.1/24
fdc0:7a62:a:0:442c:c3ff:fe2c:24c/64 (ra)
fdc0:7a62:a::1/64 (ra)
fe80:a::1/64 (link)
fe80::442c:c3ff:fe2c:24c/64 (link)
DNS Addresses: 2620:fe::fe
2001:678:3fc:3::2
Routes: 192.0.2.0/24 from 192.0.2.1 (link)
fdc0:7a62:a::/64 metric 256
fdc0:7a62:a::/64 metric 1024 (ra)
fe80::/64 metric 256
fe80:a::/64 metric 256
```
## Part 3: Turn IPv(4|6) routing on at the kernel level
To enable the host to route traffic between the external network and the Open vSwitch VLAN, IPv4/IPv6 forwarding must be enabled in the kernel. Additionally, implementing a small set of sysctl configuration parameters also improves basic IPv4 filtering.
. Create a new file named `/etc/sysctl.d/70-routing.conf`.
```bash
cat << 'EOF' | sudo tee /etc/sysctl.d/70-routing.conf
# The 70-routing.conf parameters are applied last, after the files
# in the /usr/lib/sysctl.d directory.
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv4.conf.all.log_martians = 1
EOF
```
. Make it happen!
```bash
sudo sysctl --system | grep net
```
```bash=
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.c-3po.rp_filter = 2
net.ipv4.conf.enp0s1.rp_filter = 2
net.ipv4.conf.lo.rp_filter = 2
net.ipv4.conf.ovs-system.rp_filter = 2
net.ipv4.conf.vlan10.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.c-3po.accept_source_route = 0
net.ipv4.conf.enp0s1.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.ovs-system.accept_source_route = 0
net.ipv4.conf.vlan10.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.c-3po.promote_secondaries = 1
net.ipv4.conf.enp0s1.promote_secondaries = 1
net.ipv4.conf.lo.promote_secondaries = 1
net.ipv4.conf.ovs-system.promote_secondaries = 1
net.ipv4.conf.vlan10.promote_secondaries = 1
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
```
## Part 4: Configure dnsmasq for automatic addressing of containers
Part 4 configures **dnsmasq** on the container VLAN, enabling new Incus instances to automatically receive IPv4/IPv6 addresses, router advertisements and DNS settings. This turns the Open vSwitch segment into a self-contained, plug-and-play network for containers.
Edit the /etc/dnsmasq.conf file to set the required parameters for addressing containers and resolving names. Please note that running the following command will erase any previous dnsmasq configuration.
```bash
cat << 'EOF' | sudo tee /etc/dnsmasq.conf
# Specify Container VLAN interface
interface=vlan10
# Enable DHCPv4 on Container VLAN
dhcp-range=192.0.2.100,192.0.2.200,3h
# Enable IPv6 router advertisements
enable-ra
# Enable SLAAC
dhcp-range=::,constructor:vlan10,ra-names,slaac
# Optional: Specify DNS servers
dhcp-option=option:dns-server,172.16.0.2,9.9.9.9
dhcp-option=option6:dns-server,[2001:678:3fc:3::2],[2620:fe::fe]
# Avoid DNS listen port conflict between dnsmasq and systemd-resolved
port=0
EOF
```
Don't forget to restart service after editing the configuration file.
```bash
sudo systemctl restart dnsmasq.service
```
## Part 5: Masquerade traffic outgoing from the host interface
Part 5 introduces the minimum necessary **nftables** configuration to perform source NAT on all packets leaving the host via the enp0s1 interface. This allows containers on the Open vSwitch VLAN to reach external networks using the host’s IP address.
Create a new basic `/etc/nftables.conf` file that only configures source address translation for all outbound packets passing through the `enps0s1` interface.
```bash
cat << 'EOF' | sudo tee /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
table inet nat {
chain postrouting {
type nat hook postrouting priority 100;
oifname "enp0s1" masquerade
}
}
EOF
```
Don't forget to restart the `nftables` systemd service to enable the ruleset.
```bash
sudo systemctl enable --now nftables.service
```
```bash
sudo nft list ruleset
```
```bash=
table inet nat {
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
oifname "enp0s1" masquerade
}
}
```
---
## Part 6: Install Incus container manager
In Part 6, the Zabbly APT repository is added and the Incus container manager is installed. The unprivileged user is configured so that containers can be created and managed without root access.
We need to add a new package source.
* Add the new repository key
```bash
wget -O - https://pkgs.zabbly.com/key.asc | sudo tee /etc/apt/keyrings/zabbly.asc
```
* Add the new repository parameters
```bash
cat << EOF | sudo tee /etc/apt/sources.list.d/zabbly-incus-stable.sources
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: trixie
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF
```
* Add the Trixie stable version to Debian sources file
```bash
sudo sed -i 's/Suites:[[:space:]]\+testing /Suites: trixie testing /' /etc/apt/sources.list.d/debian.sources
```
* We are now ready to update package catalog and install Incus
```bash
sudo apt update
sudo apt -y install incus --no-install-recommends
```
* The normal user `etu` is our default unprivileged user and must belong to `incus-admin` and `incus` groups.
```bash
for grp in incus-admin incus
do
sudo adduser etu $grp
done
```
Log out and log back in to make it effective.
* After new login, group assignment is correct
```bash
groups
```
```bash=
etu adm sudo users incus-admin incus
```
---
## Part 7: Set the default container profile
Part 7 customises the Incus default profile. This means that, rather than using the local Linux bridge proposed by Incus, containers will connect directly to the existing C-3PO Open vSwitch bridge on VLAN 10.
This customization can be done using the `incus admin init` command, which offers a variety of options. The important thing is to prevent the creation of a local network bridge and instead use our Open vSwitch **c-3po** switch.
```bash
incus admin init
```
```bash
Would you like to use clustering? (yes/no) [default=no]:
Voulez-vous configurer un nouveau pool de stockage? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir, truenas) [default=dir]:
Where should this storage pool store its data? [default=/var/lib/incus/storage-pools/default]:
Would you like to create a new local network bridge? (yes/no) [default=yes]: no
Would you like to use an existing bridge or host interface? (yes/no) [default=no]: yes
Name of the existing bridge or host interface: c-3po
Would you like the server to be available over the network? (yes/no) [default=no]: yes
Addresses to associate with (without ports) [default=all]:
Port to bind to [default=8443]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]:
```
We have to change the `nictype:` from `macvlan` to `bridged` and set the `vlan:` number.
```bash
incus profile device set default eth0 nictype bridged
incus profile device set default eth0 vlan 10
```
```bash
incus profile show default
```
```bash=
config: {}
description: Default Incus profile
devices:
eth0:
name: eth0
nictype: bridged
parent: c-3po
type: nic
vlan: "10"
root:
path: /
pool: default
type: disk
name: default
used_by: []
project: default
```
If we want to make the initialisation of Incus reproducible, we can reuse the YAML dump of this setup. Here are the corresponding commands:
```bash
incus admin init --dump >incus-init.yaml
```
```bash
cat incus-init.yaml
```
```yaml=
---
config:
core.https_address: '[::]:8443'
networks: []
storage_pools:
- config:
source: /var/lib/incus/storage-pools/default
description: ""
name: default
driver: dir
storage_volumes: []
profiles:
- config: {}
description: Default Incus profile
devices:
eth0:
name: eth0
nictype: bridged
parent: c-3po
type: nic
vlan: "10"
root:
path: /
pool: default
type: disk
name: default
project: ""
projects:
- config:
features.images: "true"
features.networks: "true"
features.networks.zones: "true"
features.profiles: "true"
features.storage.buckets: "true"
features.storage.volumes: "true"
description: Default Incus project
name: default
certificates: []
cluster_groups: []
```
Here is the non-interactive Incus initialisation and customization command:
```bash
incus admin init --preseed < incus-init.yaml
```
We are done with the default profile.
---
## Part 8: Create a first set of Incus containers
In Part 8, a small set of Debian Trixie Incus containers is launched and connected to VLAN 10. It is then verified that each instance has booted correctly with working IPv4 and IPv6 connectivity.
```bash
for i in {0..2}
do
incus launch images:debian/trixie c${i}
done
```
```bash=
Launching c0
Launching c1
Launching c2
```
```bash
incus ls
```
```bash=
+------+---------+--------------------+------------------------------------------+-----------+-------------+
| NAME | STATE | IPv4 | IPv6 | TYPE | SNAPSHOTS |
+------+---------+--------------------+------------------------------------------+-----------+-------------+
| c0 | RUNNING | 192.0.2.183 (eth0) | fdc0:7a62:a:0:1266:6aff:fefb:bbae (eth0) | CONTAINER | 0 |
+------+---------+--------------------+------------------------------------------+-----------+-------------+
| c1 | RUNNING | 192.0.2.187 (eth0) | fdc0:7a62:a:0:1266:6aff:fe7e:cd6d (eth0) | CONTAINER | 0 |
+------+---------+--------------------+------------------------------------------+-----------+-------------+
| c2 | RUNNING | 192.0.2.161 (eth0) | fdc0:7a62:a:0:1266:6aff:fe31:7f36 (eth0) | CONTAINER | 0 |
+------+---------+--------------------+------------------------------------------+-----------+-------------+
```
---
## Part 9: Test the network configuration and connectivity of Incus containers.
Part 9 verifies that a newly created container with the correct IPv4/IPv6 addresses can access external package mirrors and communicate with its peers on VLAN 10.
```bash
incus exec c0 -- bash
```
```bash=
root@c0:~#
```
### 1. List network addresses
```bash
root@c0:~# ip addr ls
```
```bash=
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 10:66:6a:fb:bb:ae brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.0.2.183/24 metric 1024 brd 192.0.2.255 scope global dynamic eth0
valid_lft 10690sec preferred_lft 10690sec
inet6 fdc0:7a62:a:0:1266:6aff:fefb:bbae/64 scope global mngtmpaddr noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::1266:6aff:fefb:bbae/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
```
### 2. Test routing and name resolution at the same time
```bash
root@c0:~# apt update
```
```bash=
Hit:1 http://deb.debian.org/debian trixie InRelease
Get:2 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Get:3 http://deb.debian.org/debian-security trixie-security InRelease [43.4 kB]
Get:4 http://deb.debian.org/debian-security trixie-security/main amd64 Packages [85.2 kB]
Get:5 http://deb.debian.org/debian-security trixie-security/main Translation-en [53.6 kB]
Fetched 229 kB in 0s (685 kB/s)
All packages are up to date.
```
### 3. Ping other containers from c0
```bash
root@c0:~# for i in {1..2}; do ping -c2 c$i; done
```
```bash=
PING c1 (fdc0:7a62:a:0:1266:6aff:fe7e:cd6d) 56 data bytes
64 bytes from c1 (fdc0:7a62:a:0:1266:6aff:fe7e:cd6d): icmp_seq=1 ttl=64 time=0.341 ms
64 bytes from c1 (fdc0:7a62:a:0:1266:6aff:fe7e:cd6d): icmp_seq=2 ttl=64 time=0.078 ms
--- c1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.078/0.209/0.341/0.131 ms
PING c2 (fdc0:7a62:a:0:1266:6aff:fe31:7f36) 56 data bytes
64 bytes from c2 (fdc0:7a62:a:0:1266:6aff:fe31:7f36): icmp_seq=1 ttl=64 time=0.425 ms
64 bytes from c2 (fdc0:7a62:a:0:1266:6aff:fe31:7f36): icmp_seq=2 ttl=64 time=0.069 ms
--- c2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.069/0.247/0.425/0.178 ms
```
---
## Part 10: Automated container administration very first step
Part 10 introduces a small helper script that automates routine administration tasks such as upgrades and cleanup by running maintenance commands inside every running container.
To update packages in containers, let's create a shell script that runs commands in each running container.
```bash
cat << 'EOF' > run-in-c.sh
#!/usr/bin/env bash
set -euo pipefail
# All command *strings* as given on the CLI
cmds=("$@")
# Read container names into an array
mapfile -t containers < <(incus list --format csv --columns n status=Running)
for c in "${containers[@]}"; do
echo ">>>>>>>>>>>>>>>>> $c"
for cmd in "${cmds[@]}"; do
# Run via a single shell inside the container
incus exec "$c" -- bash -c "$cmd"
done
done
EOF
```
Then run the usual `apt` commands within containers.
```bash
bash run-in-c.sh \
"apt update" \
"apt -y full-upgrade" \
"apt clean" \
"apt -y autopurge"
```
---
## Part 11: Check OvS swicth ports and TCAM on the host
In Part 11, the Open vSwitch configuration and forwarding tables on the host are inspected to confirm that all container interfaces have been correctly tagged in VLAN 10 and that the MAC and neighbour entries match the expected topology.
Display OvS main switch configuration.
```bash
sudo ovs-vsctl show
```
Please note that all `veth` container port connections are tagged in VLAN 10.
```bash
0119a374-1812-4fe9-a391-8be6031e7d1c
Bridge c-3po
fail_mode: standalone
Port vethc68e3315
tag: 10
Interface vethc68e3315
Port vethdd774d01
tag: 10
Interface vethdd774d01
Port c-3po
Interface c-3po
type: internal
Port veth898bc1a1
tag: 10
Interface veth898bc1a1
Port vlan10
tag: 10
Interface vlan10
type: internal
ovs_version: "3.6.0"
```
Say hello to VLAN 10 neighborhood
```bash
ping -c2 ff02::1%vlan10
```
```bash=
PING ff02::1%vlan10 (ff02::1%vlan10) 56 data bytes
64 bytes from fe80:a::1%vlan10: icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from fe80::1266:6aff:fefb:bbae%vlan10: icmp_seq=1 ttl=64 time=0.796 ms
64 bytes from fe80::1266:6aff:fe7e:cd6d%vlan10: icmp_seq=1 ttl=64 time=0.844 ms
64 bytes from fe80::1266:6aff:fe31:7f36%vlan10: icmp_seq=1 ttl=64 time=0.859 ms
64 bytes from fe80:a::1%vlan10: icmp_seq=2 ttl=64 time=0.072 ms
--- ff02::1%vlan10 ping statistics ---
2 packets transmitted, 2 received, +3 duplicates, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.072/0.532/0.859/0.369 ms
```
As we already knew, we have three neighbors.
```bash
ip nei ls dev vlan10
```
```bash=
192.0.2.183 lladdr 10:66:6a:fb:bb:ae STALE
192.0.2.161 lladdr 10:66:6a:31:7f:36 STALE
192.0.2.187 lladdr 10:66:6a:7e:cd:6d STALE
fdc0:7a62:a:0:1266:6aff:fe31:7f36 lladdr 10:66:6a:31:7f:36 STALE
fdc0:7a62:a:0:1266:6aff:fefb:bbae lladdr 10:66:6a:fb:bb:ae STALE
fe80:a::1 lladdr 46:2c:c3:2c:02:4c router STALE
fdc0:7a62:a:0:1266:6aff:fe7e:cd6d lladdr 10:66:6a:7e:cd:6d STALE
fe80::1266:6aff:fe31:7f36 lladdr 10:66:6a:31:7f:36 REACHABLE
fe80::1266:6aff:fefb:bbae lladdr 10:66:6a:fb:bb:ae REACHABLE
fe80::1266:6aff:fe7e:cd6d lladdr 10:66:6a:7e:cd:6d REACHABLE
```
Display Open vSwitch Content Addressable Memory (CAM) entries.
```bash
sudo ovs-appctl fdb/show c-3po
```
```bash=
port VLAN MAC Age
LOCAL 0 46:2c:c3:2c:02:4c 294
3 10 10:66:6a:7e:cd:6d 75
4 10 10:66:6a:31:7f:36 75
1 10 46:2c:c3:2c:02:4c 75
2 10 10:66:6a:fb:bb:ae 75
```
## Part 12: Initialising the Debian VM using a Cloud Init YAML declaration
This part introduces the YAML declaration used to instantiate the **ovs-incus** virtual machine with [**cloud-init**](https://cloudinit.readthedocs.io/en/latest/index.html#), turning it into a reproducible template that provisions Debian, networking packages, and all required Open vSwitch, routing, and dnsmasq configuration on first boot.
Two YAML variants are provided, reflecting different levels of [**cloud-init**](https://cloudinit.readthedocs.io/en/latest/index.html#) features:
- a minimal configuration that focuses on package installation and basic files,
- an extended configuration that also manages APT policy, additional repositories, and post-configuration commands.
### 1. What is Cloud-init?
Cloud-init is an open-source initialisation tool that automatically configures a virtual machine at boot time using a YAML file. It handles tasks such as network setup, user creation, SSH keys, APT configuration, file injection, package installation and final orchestration commands, eliminating the need for manual intervention.
Cloud-init has a step-by-step process for starting up, and each step allows you to add your own user data to customise the instance. In this lab, the YAML declarations use these hooks (such as **bootcmd**, **write_files**, **apt**, **packages**, and **runcmd**) at exact points in that flow.
This means **there is a strict execution order** that must be followed when deciding where to place commands, file writes, and package operations. The way user data is written in YAML is closely linked to the boot stages. If a directive is misordered, it can lead to subtle race conditions. For example, this can mean running a command before its package is installed or configuring a service before its configuration files exist.
:::warning
Cloud-init is a powerful automation tool, but relying heavily on its execution order can quickly produce YAML declarations that are hard to read and debug. Subtle ordering dependencies between modules mean small changes in one hook can cause non-obvious side effects later in the boot flow.
This is why there are two different level YAML files in this part.
:::
### 2. What are the Cloud-init user-data hooks used here?
Below is an ordered list focused on the YAML keys we are using, in the order they are typically evaluated within the boot flow.
1. **bootcmd:**
Runs early in the boot process on **every boot**, allowing low-level commands such as preparing keyrings, creating directories, or tweaking the system before package and configuration modules run.
2. **write_files:**
Writes configuration files to disk so that services and package tools see the correct configuration when later modules (like apt and packages) execute.
3. **apt** (and apt: sources / apt: conf):
Configures APT options, repositories, and non-interactive dpkg behavior, ensuring that subsequent package installation uses the desired sources and policies.
4. **packages:**
Installs the requested Debian packages using the configured APT sources, typically after an automatic apt-get update triggered by the module.
5. **runcmd:**
Defines commands that are written to a script and executed near the end of the boot sequence, after packages are installed, commonly used for final orchestration like enabling services or running Incus initialisation.
### 3. First level YAML declaration proposal
The YAML declaration file below defines the virtual machine template that is used to create a dedicated `ovs-incus` host in the Teaching Private Cloud environment.
This example automatically provisions a Debian Testing image and pre-installs vital networking components, such as Nftables and Dnsmasq from the [shared lab scripts repository](https://gitlab.inetdoc.net/labs/startup-scripts).
This first-level YAML declaration deliberately uses a small subset of cloud-init user-data hooks. The plan is to keep the template easy to read and understand. The trade-off is that some tasks are left to be done later in the lab when the virtual machine has booted, rather than being fully automated at VM boot time.
Here is the list of hooks used here:
1. **packages:**
Installs the minimal set of networking packages (nftables and dnsmasq) required for the ovs-incus host.
2. **write_files:**
Drops the routing, dnsmasq, and nftables configuration files directly into /etc on first boot.
3. **runcmd:**
Applies the Netplan configuration, removes the default network file, and enables the nftables service once all files and packages are in place.
Here is a copy of the **`ovs+incus-level1.yaml`** file:
```yaml=
---
kvm:
vms:
- vm_name: ovs-incus
os: linux
master_image: debian-testing-amd64.qcow2
force_copy: false
memory: 2048
tapnum: 1
cloud_init:
force_seed: false
hostname: ovs-incus
packages:
- nftables
- dnsmasq
write_files:
- path: /etc/sysctl.d/70-routing.conf
content: |
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv4.conf.all.log_martians = 1
- path: /etc/dnsmasq.conf
content: |
# Specify Container VLAN interface
interface=vlan10
# Enable DHCPv4 on Container VLAN
dhcp-range=192.0.2.100,192.0.2.200,3h
# Enable IPv6 router advertisements
enable-ra
# Enable SLAAC
dhcp-range=::,constructor:vlan10,ra-names,slaac
# Optional: Specify DNS servers
dhcp-option=option:dns-server,172.16.0.2,9.9.9.9
dhcp-option=option6:dns-server,[2001:678:3fc:3::2],[2620:fe::fe]
# Avoid DNS listen port conflict between dnsmasq and systemd-resolved
port=0
- path: /etc/nftables.conf
content: |
#!/usr/sbin/nft -f
flush ruleset
table inet nat {
chain postrouting {
type nat hook postrouting priority 100;
oifname "enp0s1" masquerade
}
}
netplan:
network:
version: 2
renderer: networkd
ethernets:
enp0s1:
dhcp4: true
dhcp6: false
accept-ra: true
bridges:
c-3po:
openvswitch: {}
vlans:
vlan10:
id: 10
link: c-3po
addresses:
- 192.0.2.1/24
- fdc0:7a62:a::1/64
- fe80:a::1/64
runcmd:
- rm /etc/netplan/enp0s1.yaml
- netplan apply
- sysctl --system
- systemctl enable --now nftables.service
```
To launch a virtual machine using the YAML declaration file, all we have to do is:
```bash
lab-startup.py ovs+incus-level1.yaml
```
### 4. Second level YAML declaration proposal
Cloud-init user-data hooks in this second-level YAML declaration are used to create a VM with container management ready to use on first boot, but the configuration is much more complex. Here, the early bootcmd commands prepare APT keyrings and repositories. This makes sure that package installation and Incus-related configuration succeed when later modules run.
Here is the list of hooks used here:
1. **bootcmd:**
Prepares APT keyrings and repository files early so that package management is ready before other modules run.
2. **write_files:**
Installs configuration files such as dnsmasq, nftables, Netplan, and Incus preseed YAML into their final locations.
3. **apt** (and apt: conf / apt: sources):
Declares APT policy and custom repositories so that subsequent package operations use the correct sources and options.
4. **packages:**
Installs Incus and all required networking packages using the repositories prepared by bootcmd and apt.
5. **runcmd:**
Performs final orchestration steps, including network reloads and Incus initialisation, after all packages and configuration files are in place.
Here is a copy of the **`ovs+incus-level2.yaml`** file:
```yaml=
---
kvm:
vms:
- vm_name: ovs-incus
os: linux
master_image: debian-testing-amd64.qcow2
force_copy: false
memory: 2048
tapnum: 1
cloud_init:
force_seed: false
hostname: ovs-incus
bootcmd:
- mkdir -p /etc/apt/keyrings
- [cloud-init-per, once, zabbly-key, wget,
-O, /etc/apt/keyrings/zabbly.asc,
https://pkgs.zabbly.com/key.asc]
- >
sed -i 's/Suites:[[:space:]]\+testing /Suites: trixie testing /'
/etc/apt/sources.list.d/debian.sources
apt:
# Preserve dnsmasq.conf file during package install
# Add --no-install-recommends to incus install
conf: |
Dpkg::Options {
"--force-confdef";
"--force-confold";
}
APT::Install-Recommends "false";
APT::Install-Suggests "false";
packages:
- nftables
- dnsmasq
- incus
write_files:
- path: /etc/sysctl.d/70-routing.conf
content: |
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv4.conf.all.log_martians = 1
- path: /etc/apt/sources.list.d/zabbly-incus-stable.sources
content: |
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: trixie
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/zabbly.asc
- path: /etc/dnsmasq.conf
content: |
# Specify Container VLAN interface
interface=vlan10
# Enable DHCPv4 on Container VLAN
dhcp-range=192.0.2.100,192.0.2.200,3h
# Enable IPv6 router advertisements
enable-ra
# Enable SLAAC
dhcp-range=::,constructor:vlan10,ra-names,slaac
# Optional: Specify DNS servers
dhcp-option=option:dns-server,172.16.0.2,9.9.9.9
dhcp-option=option6:dns-server,[2001:678:3fc:3::2],[2620:fe::fe]
# Avoid DNS listen port conflict between
# dnsmasq and systemd-resolved
port=0
- path: /etc/nftables.conf
content: |
#!/usr/sbin/nft -f
flush ruleset
table inet nat {
chain postrouting {
type nat hook postrouting priority 100;
oifname "enp0s1" masquerade
}
}
- path: /home/etu/incus-init.yaml
content: |
config:
core.https_address: '[::]:8443'
networks: []
storage_pools:
- config:
source: /var/lib/incus/storage-pools/default
description: ""
name: default
driver: dir
storage_volumes: []
profiles:
- config: {}
description: Default Incus profile
devices:
eth0:
name: eth0
nictype: bridged
parent: c-3po
type: nic
vlan: "10"
root:
path: /
pool: default
type: disk
name: default
project: ""
projects:
- config:
features.images: "true"
features.networks: "true"
features.networks.zones: "true"
features.profiles: "true"
features.storage.buckets: "true"
features.storage.volumes: "true"
description: Default Incus project
name: default
certificates: []
cluster_groups: []
netplan:
network:
version: 2
renderer: networkd
ethernets:
enp0s1:
dhcp4: true
dhcp6: false
accept-ra: true
bridges:
c-3po:
openvswitch: {}
vlans:
vlan10:
id: 10
link: c-3po
addresses:
- 192.0.2.1/24
- fdc0:7a62:a::1/64
- fe80:a::1/64
runcmd:
- rm /etc/netplan/enp0s1.yaml
- netplan apply
- sysctl --system
- systemctl enable --now nftables.service
- adduser etu incus
- adduser etu incus-admin
- chown etu:etu /home/etu/incus-init.yaml
- >
runuser -u etu --
incus admin init --preseed <
/home/etu/incus-init.yaml
```
To launch a virtual machine using the YAML declaration file, all we have to do is:
```bash
lab-startup.py ovs+incus-level2.yaml
```
## To conclude...
Here is an image of the achieved logical topology.
This lab provides comprehensive instructions for setting up unprivileged Incus containers on Open vSwitch (OvS) within a Debian Trixie installation. It covers all the necessary steps, including installing the required tools, configuring a network switch and routing, setting up DHCP and DNS, installing Incus, and creating containers, and verifying the setup.
Cloud-init, combined with carefully designed YAML declarations, meets this lab’s initial objectives by turning a generic Debian image into a reproducible Incus host with Open vSwitch ready on first boot.
The two levels of user-data configuration illustrate how progressively richer use of hooks can move from a minimally configured VM to a fully operational container platform without manual post-install steps.
However, procedural automation has limits: tools like cloud-init excel at first-boot customization but do not naturally provide the strong, repeatable idempotency expected from full configuration management or infrastructure-as-code workflows. To reach that goal across the VM lifecycle and fleet, additional bricks such as Ansible for ongoing configuration and OpenTofu/Terraform for declarative infrastructure provisioning are typically combined with cloud-init, each tool covering a different layer of the automation stack.