Package ifupdown breaks network configuration by cloud-init

Bug #1867029 reported by AJ
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
High
Chad Smith
ifupdown (Ubuntu)
New
Undecided
Unassigned

Bug Description

It appears the `ifupdown` package breaks networking configuration that cloud-init would normally handle (?) on both providers. The result, if you image an instance with the `ifupdown` package installed, is that the image is not bootable by Azure/AWS. I'm not familiar with how Azure/AWS/etc use cloud-init for network start up however. I'm filing here for now, in case `cloud-init` needs to be more defensive against `ifupdown`.

To reproduce with Packer (I confirmed this method):

* Add necessary Azure subscription env vars from `ifupdown-bug.json`
* `packer build ifupdown-bug.json`
* Try to boot an instance with resulting image
* Instance will never boot -- more specifically, networking never comes up. See `boot.log` for an example boot

To reproduce manually (only a guess, I did not confirm):

* Boot 18.04 ubuntu instance
* Install ifupdown
* Image the instance
* Try to boot instance using new image
* Instance won't boot

I hit this with Packer, creating new images for booting instances in scaling groups -- `ifupdown` is brought in by `salt`. If this is reproducible via manual installation, there could be some backup images/snapshots that are unable to boot though.

This appears to be because cloud-init network start up operations are broken by whatever `ifupdown` sysv/systemd scripts perform on boot. With `ifupdown` installed on Azure, `eth0` does not get an IP address, the instance never fully boots according to Azure, and it will enter an error state. On AWS, the instance boots, but network operations like SSH fail.

The `boot.log` is from Azure. Eventually `cloud-init` gives an exception (see `exception.log`), but this seems like a secondary issue related to networking being down.

I did try to upgrade cloud-init using nightly PPAs (version 20.1 I believe), but no luck. This is on Ubuntu 18.04 instances from Azure marketplace and AWS Canonical AMI, cloud-init version `19.4-33-gbb4131a2-0ubuntu1~18.04.1`.

Related: https://github.com/Azure/WALinuxAgent/issues/1612

Tags: uec-images
Revision history for this message
AJ (agjohnson) wrote :
Revision history for this message
Ryan Harper (raharper) wrote :

Thanks for filing the issue.

I can reproduce this issue:

% lxc launch ubuntu-daily:bionic b7
% lxc exec b7 -- bash -c 'apt -qy install ifupdown; rm -f /etc/netplan/50-cloud-init.yaml'
% lxc restart b7

The issues at play are:

1) bionic Ubuntu images include a different /etc/network/interfaces file than Xenial

% lxc launch ubuntu-daily:xenial x7
% lxc exec x7 -- cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# Source interfaces
# Please check /etc/network/interfaces.d before changing this file
# as interfaces may have been defined in /etc/network/interfaces.d
# See LP: #1262951
source /etc/network/interfaces.d/*.cfg

vs. Bionic

% lxc exec b77 -- cat /etc/network/interfaces
# ifupdown has been replaced by netplan(5) on this system. See
# /etc/netplan for current configuration.
# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown

In particular, the Bionic images are missing this line:

source /etc/network/interfaces.d/*.cfg

2) cloud-init renders network-config to /etc/network/interfaces.d/50-cloud-init.cfg

3) networking.service runs ifup -a; however, it does not find a network config for the interfaces due to the missing source line in bionic ifupdown eni:

root@b77:~# cat /etc/network/interfaces
# ifupdown has been replaced by netplan(5) on this system. See
# /etc/netplan for current configuration.
# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown
root@b77:~# ifquery --list --exclude=log
lo
root@b77:~# vi /etc/network/interfaces
root@b77:~# cat /etc/network/interfaces
# ifupdown has been replaced by netplan(5) on this system. See
# /etc/netplan for current configuration.
# To re-enable ifupdown on this system, you can run:
# sudo apt install ifupdown
source /etc/network/interfaces.d/*.cfg
root@b77:~# ifquery --list --exclude=log
lo
eth0

As a workaround, you can add the missing line to /etc/network/interfaces

source /etc/network/interfaces.d/*.cfg

We'll need to sort out if ifupdown packaging or cloud-init will need to ensure that line is present in images.

Changed in cloud-init:
importance: Undecided → High
status: New → Confirmed
Ryan Harper (raharper)
tags: added: uec-images
Chad Smith (chad.smith)
Changed in cloud-init:
status: Confirmed → In Progress
assignee: nobody → Chad Smith (chad.smith)
Revision history for this message
Chad Smith (chad.smith) wrote :

An upstream commit is up for review with a fix for this issue:
https://github.com/canonical/cloud-init/pull/267

Revision history for this message
Chad Smith (chad.smith) wrote :

A separate bug has been filed against ifupdown package as well. Though I'm not sure it will be a bug that will be fixed because Ubuntu Bionic and later do not really support ifupdown, they support using netplan for network configuration.

https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1868329

Chad Smith (chad.smith)
Changed in cloud-init:
status: In Progress → Fix Committed
Revision history for this message
Chad Smith (chad.smith) wrote : Fixed in cloud-init version 20.2.

This bug is believed to be fixed in cloud-init in version 20.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.