Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| cloud-images |
New
|
Undecided
|
Unassigned | ||
| linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
| systemd (Ubuntu) |
Triaged
|
Medium
|
Nick Rosbrook | ||
Bug Description
Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images, there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning.
This is easiest to reproduce when emulating arm64 on amd64 host. The bug is a race condition, so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce.
The latest mantic minimized images from http://
This but is not present in the non minimized/base images @ http://
Reproducer
```
wget -O "launch-
chmod +x ./launch-
wget https:/
./launch-
```
You will then be able to log in with user `ubuntu` and password `passw0rd`.
You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned.
```
ubuntu@cloudimg:~$ ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s1: <BROADCAST,
link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
```
This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed.
You can verify this by running `sudo cloud-init clean && sudo cloud-init init`
You can then see a successfully configured network interface
```
ubuntu@cloudimg:~$ ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s1: <BROADCAST,
link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1
valid_lft 86391sec preferred_lft 86391sec
inet6 fec0::5054:
valid_lft 86393sec preferred_lft 14393sec
inet6 fe80::5054:
valid_lft forever preferred_lft forever
```
The bug is also reproducible with amd64 guest on adm64 host on older/slower hardware.
The suggested fixes while debugging this issue are:
* to include `virtio-net` as a built-in in the mantic generic kernel
* understand what needs to change in cloud-init so that it can react to late additions of network interfaces
I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware.
| Changed in linux (Ubuntu): | |
| milestone: | none → ubuntu-23.10 |
| description: | updated |
| description: | updated |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote Missing required logs. | #1 |
| Changed in linux (Ubuntu): | |
| status: | New → Incomplete |
| description: | updated |
| Philip Roche (philroche) wrote Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning | #2 |
| Dimitri John Ledkov (xnox) wrote | #3 |
| Changed in linux (Ubuntu): | |
| status: | Incomplete → Triaged |
| Changed in linux (Ubuntu): | |
| status: | Triaged → Fix Committed |
| Dimitri John Ledkov (xnox) wrote | #4 |
| tags: | added: rls-mm-incoming |
| summary: |
Mantic minimized/minimal cloud images do not receive IP address during - provisioning + provisioning; systemd regression with wait-online |
| tags: |
added: foundations-todo removed: rls-mm-incoming |
| tags: | added: rls-mm-incoming |
| Changed in systemd (Ubuntu): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| assignee: | nobody → Nick Rosbrook (enr0n) |
| Philip Roche (philroche) wrote | #5 |
| Philip Roche (philroche) wrote | #6 |
| Launchpad Janitor (janitor) wrote | #7 |
| Changed in linux (Ubuntu): | |
| status: | Fix Committed → Fix Released |
| tags: | removed: rls-mm-incoming |
| Changed in systemd (Ubuntu): | |
| importance: | High → Medium |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote | #8 |
| tags: | added: kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-oem-6.5 |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote | #9 |
| tags: | added: kernel-spammed-jammy-linux-azure-6.5-v2 verification-needed-jammy-linux-azure-6.5 |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote | #10 |
| tags: | added: kernel-spammed-jammy-linux-aws-6.5-v2 verification-needed-jammy-linux-aws-6.5 |
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 2036968
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.