Ubuntu
cloud-init package

cloud-init-nonet runs 'start networking' explicitly

Bug #1031065 reported by Scott Moser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
Medium
Scott Moser
Precise
Fix Released
Medium
Unassigned

Bug Description

== Begin SRU Information ==
[Impact]
Cloud-init in 12.04 has an upstart job named 'cloud-init-nonet' that
calls 'start networking' explicitly. This was done to fix a boot
deadlock (bug 800824), but it was not the proper fix.

A much more correct fix is now possible because of improvements
that have been made in mountall (bug 643289) and have been brought
back to 12.04.

calling 'start networking' from cloud-init-nonet could cause issues
because other upstart 'start on' conditions might not be met at this
point in boot.

The fix here is the same as is now applied in quantal and raring.
It more correctly addresses the root issue, that network-device-added
events were not being emitted in a container. There is now a job named
cloud-init-container that will emit 'network-device-added' events if
it is inside a container *and* its sanity checks show that the given
device has not already been brought up.

[Test Case]
You can demonstrate why this change was necessary, and that the
provided change fixes the issue by doing the following:

## create a 'source' (pristine) root ##
sudo lxc-create -t ubuntu-cloud --name source-precise-amd64 -- \
   --release precise --arch amd64 --stream daily

# set up 2 copies of the pristine root
# * 'nostart' just has cloud-init's call to 'start-networking' disabled
# * 'patched' contains the full upgraded cloud-init
sudo lxc-clone -o source-precise-amd64 -n nostart
f="/var/lib/lxc/nostart/etc/init/cloud-init-nonet.conf"
if [ ! -e "$f.dist" ]; then
   # disable 'start networking' in the cloud-init-nonet job
   sudo cp "$f" "$f.dist"
   sudo sed -i 's,^\([ ]\+start networking.*\),#\1,' "$f"
fi

sudo lxc-clone -o source-precise-amd64 -n patched
deb="cloud-init_0.6.3-0ubuntu1.5~ppa1_all.deb"
rpath=/var/lib/lxc/patched/rootfs
sudo cp $deb "/var/lib/lxc/patched/rootfs/tmp"
sudo LANG=C chroot "/var/lib/lxc/patched/rootfs" dpkg -i "/tmp/$deb"

## Now, start both. the 'nostart' root will hang on
## cloud-init waiting for networking to come up.
## the 'patched' will come all the way up quickly.
## You can stop them with 'sudo lxc-stop -n <name>'
sudo lxc-start -n patched -- /sbin/init --verbose
sudo lxc-start -n nostart -- /sbin/init --verbose

[Regression Potential]
Regressions would be likely to occur in one of 2 places:
a.) inside a container, where cloud-init-container caused a
    problem by emitting its network-device-added events

    Here, the problem would be very limited to lxc containers
    that have cloud-init inside them. This is likely very small.

b.) outside a container, where 'start networking' was previously
    "fixing" a boot deadlock that could have occurred.
    Here, the changes are basically making precise work like 12.10
    and 13.04, so hopefully issues would have been shaken out there.
== End SRU Information ==

In development of 'overlayroot' package, I was mounting / as rw in the initramfs.
This was causing a different order of execution of mounts, and as a result a different order of networking resolvconf and networking bringup.

It exposed a bug in resolvconf, which was being called on boot in this order:
==== Mon Jul 30 19:08:58 UTC 2012 /sbin/resolvconf -a lo.inet ====
==== Mon Jul 30 19:08:59 UTC 2012 /sbin/dhclient-script ====
==== Mon Jul 30 19:08:59 UTC 2012 /sbin/resolvconf -a eth0.dhclient ====
==== Mon Jul 30 19:08:59 UTC 2012 /sbin/resolvconf -a eth0.inet ====
==== Mon Jul 30 19:08:59 UTC 2012 /sbin/resolvconf --enable-updates ====
==== Mon Jul 30 19:08:59 UTC 2012 /etc/resolvconf/update.d/libc -u ====

The normal order is for resolvconf --enable-updates to be called (from /etc/init/resolvconf.conf) before anything else. As a result, I was seeing errors like:
  resolvconf: Error: /run/resolvconf/interface either does not exist or is not a directory

This may be exposing a more grave issue, in that I believe the reason for dhclient coming up before resolvconf.conf started was that it was being run as a result of /etc/init/network-interface.conf. I don't immediately see how that is guarnateed to have /run ounted at all.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: resolvconf 1.67ubuntu1
ProcVersionSignature: User Name 3.5.0-6.6-generic 3.5.0
Uname: Linux 3.5.0-6-generic x86_64
Architecture: amd64
Date: Mon Jul 30 20:07:04 2012
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: resolvconf
UpgradeStatus: No upgrade log present (probably fresh install)

Related Bugs:
 * bug 800824: cloud-init-nonet times out in lxc
 * bug 925122: container's udevadm trigger --add affects the host
 * bug 643289: idmapd does not starts to work after system reboot

See original description

Related branches

lp:~smoser/cloud-init/trunk.lp1031065 (Merged)
lp:~cloud-init-dev/cloud-init/trunk
Chad Smith (community): Needs Fixing
Hannes: Pending requested
Diff: 51583 lines (+11/-49380) (has conflicts)
424 files modified
.bzrignore (+0/-4)
ChangeLog (+0/-761)
HACKING.rst (+0/-48)
LICENSE (+0/-674)
MANIFEST.in (+0/-8)
Makefile (+0/-82)
README (+11/-0)
TODO.rst (+0/-43)
bin/cloud-init (+0/-673)
cloudinit/__init__.py (+0/-21)
cloudinit/cloud.py (+0/-109)
cloudinit/config/__init__.py (+0/-58)
cloudinit/config/cc_apt_configure.py (+0/-292)
cloudinit/config/cc_apt_pipelining.py (+0/-57)
cloudinit/config/cc_bootcmd.py (+0/-54)
cloudinit/config/cc_byobu.py (+0/-80)
cloudinit/config/cc_ca_certs.py (+0/-104)
cloudinit/config/cc_chef.py (+0/-342)
cloudinit/config/cc_debug.py (+0/-109)
cloudinit/config/cc_disable_ec2_metadata.py (+0/-36)
cloudinit/config/cc_emit_upstart.py (+0/-69)
cloudinit/config/cc_fan.py (+0/-101)
cloudinit/config/cc_final_message.py (+0/-73)
cloudinit/config/cc_foo.py (+0/-52)
cloudinit/config/cc_growpart.py (+0/-300)
cloudinit/config/cc_grub_dpkg.py (+0/-73)
cloudinit/config/cc_keys_to_console.py (+0/-62)
cloudinit/config/cc_landscape.py (+0/-99)
cloudinit/config/cc_locale.py (+0/-37)
cloudinit/config/cc_lxd.py (+0/-176)
cloudinit/config/cc_mcollective.py (+0/-88)
cloudinit/config/cc_migrator.py (+0/-85)
cloudinit/config/cc_mounts.py (+0/-405)
cloudinit/config/cc_package_update_upgrade_install.py (+0/-99)
cloudinit/config/cc_phone_home.py (+0/-122)
cloudinit/config/cc_power_state_change.py (+0/-223)
cloudinit/config/cc_puppet.py (+0/-118)
cloudinit/config/cc_resizefs.py (+0/-185)
cloudinit/config/cc_resolv_conf.py (+0/-116)
cloudinit/config/cc_rh_subscription.py (+0/-408)
cloudinit/config/cc_rightscale_userdata.py (+0/-102)
cloudinit/config/cc_rsyslog.py (+0/-366)
cloudinit/config/cc_runcmd.py (+0/-38)
cloudinit/config/cc_salt_minion.py (+0/-59)
cloudinit/config/cc_scripts_per_boot.py (+0/-41)
cloudinit/config/cc_scripts_per_instance.py (+0/-41)
cloudinit/config/cc_scripts_per_once.py (+0/-41)
cloudinit/config/cc_scripts_user.py (+0/-42)
cloudinit/config/cc_scripts_vendor.py (+0/-43)
cloudinit/config/cc_seed_random.py (+0/-94)
cloudinit/config/cc_set_hostname.py (+0/-37)
cloudinit/config/cc_set_passwords.py (+0/-167)
cloudinit/config/cc_snappy.py (+0/-304)
cloudinit/config/cc_ssh.py (+0/-142)
cloudinit/config/cc_ssh_authkey_fingerprints.py (+0/-105)
cloudinit/config/cc_ssh_import_id.py (+0/-99)
cloudinit/config/cc_timezone.py (+0/-39)
cloudinit/config/cc_ubuntu_init_switch.py (+0/-162)
cloudinit/config/cc_update_etc_hosts.py (+0/-60)
cloudinit/config/cc_update_hostname.py (+0/-43)
cloudinit/config/cc_users_groups.py (+0/-34)
cloudinit/config/cc_write_files.py (+0/-103)
cloudinit/config/cc_yum_add_repo.py (+0/-107)
cloudinit/cs_utils.py (+0/-105)
cloudinit/distros/__init__.py (+0/-950)
cloudinit/distros/arch.py (+0/-201)
cloudinit/distros/debian.py (+0/-227)
cloudinit/distros/fedora.py (+0/-31)
cloudinit/distros/freebsd.py (+0/-417)
cloudinit/distros/gentoo.py (+0/-160)
cloudinit/distros/net_util.py (+0/-182)
cloudinit/distros/parsers/__init__.py (+0/-28)
cloudinit/distros/parsers/hostname.py (+0/-88)
cloudinit/distros/parsers/hosts.py (+0/-92)
cloudinit/distros/parsers/resolv_conf.py (+0/-169)
cloudinit/distros/parsers/sys_conf.py (+0/-113)
cloudinit/distros/rhel.py (+0/-222)
cloudinit/distros/rhel_util.py (+0/-89)
cloudinit/distros/sles.py (+0/-179)
cloudinit/distros/ubuntu.py (+0/-31)
cloudinit/ec2_utils.py (+0/-201)
cloudinit/filters/__init__.py (+0/-21)
cloudinit/filters/launch_index.py (+0/-75)
cloudinit/handlers/__init__.py (+0/-274)
cloudinit/handlers/boot_hook.py (+0/-70)
cloudinit/handlers/cloud_config.py (+0/-163)
cloudinit/handlers/shell_script.py (+0/-55)
cloudinit/handlers/upstart_job.py (+0/-119)
cloudinit/helpers.py (+0/-458)
cloudinit/importer.py (+0/-58)
cloudinit/log.py (+0/-155)
cloudinit/mergers/__init__.py (+0/-166)
cloudinit/mergers/m_dict.py (+0/-88)
cloudinit/mergers/m_list.py (+0/-89)
cloudinit/mergers/m_str.py (+0/-46)
cloudinit/net/__init__.py (+0/-771)
cloudinit/net/network_state.py (+0/-447)
cloudinit/net/udev.py (+0/-54)
cloudinit/netinfo.py (+0/-249)
cloudinit/patcher.py (+0/-58)
cloudinit/registry.py (+0/-37)
cloudinit/reporting/__init__.py (+0/-42)
cloudinit/reporting/events.py (+0/-248)
cloudinit/reporting/handlers.py (+0/-91)
cloudinit/safeyaml.py (+0/-32)
cloudinit/settings.py (+0/-68)
cloudinit/signal_handler.py (+0/-71)
cloudinit/sources/DataSourceAltCloud.py (+0/-293)
cloudinit/sources/DataSourceAzure.py (+0/-651)
cloudinit/sources/DataSourceBigstep.py (+0/-57)
cloudinit/sources/DataSourceCloudSigma.py (+0/-141)
cloudinit/sources/DataSourceCloudStack.py (+0/-253)
cloudinit/sources/DataSourceConfigDrive.py (+0/-424)
cloudinit/sources/DataSourceDigitalOcean.py (+0/-110)
cloudinit/sources/DataSourceEc2.py (+0/-211)
cloudinit/sources/DataSourceGCE.py (+0/-167)
cloudinit/sources/DataSourceMAAS.py (+0/-353)
cloudinit/sources/DataSourceNoCloud.py (+0/-336)
cloudinit/sources/DataSourceNone.py (+0/-57)
cloudinit/sources/DataSourceOVF.py (+0/-429)
cloudinit/sources/DataSourceOpenNebula.py (+0/-455)
cloudinit/sources/DataSourceOpenStack.py (+0/-173)
cloudinit/sources/DataSourceSmartOS.py (+0/-536)
cloudinit/sources/__init__.py (+0/-337)
cloudinit/sources/helpers/__init__.py (+0/-13)
cloudinit/sources/helpers/azure.py (+0/-279)
cloudinit/sources/helpers/openstack.py (+0/-499)
cloudinit/sources/helpers/vmware/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/boot_proto.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config.py (+0/-95)
cloudinit/sources/helpers/vmware/imc/config_file.py (+0/-129)
cloudinit/sources/helpers/vmware/imc/config_namespace.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+0/-247)
cloudinit/sources/helpers/vmware/imc/config_source.py (+0/-23)
cloudinit/sources/helpers/vmware/imc/guestcust_error.py (+0/-24)
cloudinit/sources/helpers/vmware/imc/guestcust_event.py (+0/-27)
cloudinit/sources/helpers/vmware/imc/guestcust_state.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/guestcust_util.py (+0/-128)
cloudinit/sources/helpers/vmware/imc/ipv4_mode.py (+0/-45)
cloudinit/sources/helpers/vmware/imc/nic.py (+0/-147)
cloudinit/sources/helpers/vmware/imc/nic_base.py (+0/-154)
cloudinit/ssh_util.py (+0/-314)
cloudinit/stages.py (+0/-844)
cloudinit/templater.py (+0/-149)
cloudinit/type_utils.py (+0/-52)
cloudinit/url_helper.py (+0/-509)
cloudinit/user_data.py (+0/-352)
cloudinit/util.py (+0/-2226)
cloudinit/version.py (+0/-27)
config/cloud.cfg (+0/-115)
config/cloud.cfg-freebsd (+0/-88)
config/cloud.cfg.d/05_logging.cfg (+0/-66)
config/cloud.cfg.d/README (+0/-3)
doc/README (+0/-4)
doc/examples/cloud-config-add-apt-repos.txt (+0/-34)
doc/examples/cloud-config-archive-launch-index.txt (+0/-30)
doc/examples/cloud-config-archive.txt (+0/-16)
doc/examples/cloud-config-boot-cmds.txt (+0/-15)
doc/examples/cloud-config-ca-certs.txt (+0/-31)
doc/examples/cloud-config-chef-oneiric.txt (+0/-90)
doc/examples/cloud-config-chef.txt (+0/-95)
doc/examples/cloud-config-datasources.txt (+0/-73)
doc/examples/cloud-config-disk-setup.txt (+0/-251)
doc/examples/cloud-config-final-message.txt (+0/-7)
doc/examples/cloud-config-gluster.txt (+0/-18)
doc/examples/cloud-config-growpart.txt (+0/-31)
doc/examples/cloud-config-install-packages.txt (+0/-15)
doc/examples/cloud-config-landscape.txt (+0/-22)
doc/examples/cloud-config-launch-index.txt (+0/-23)
doc/examples/cloud-config-lxd.txt (+0/-55)
doc/examples/cloud-config-mcollective.txt (+0/-49)
doc/examples/cloud-config-mount-points.txt (+0/-46)
doc/examples/cloud-config-phone-home.txt (+0/-14)
doc/examples/cloud-config-power-state.txt (+0/-40)
doc/examples/cloud-config-puppet.txt (+0/-51)
doc/examples/cloud-config-reporting.txt (+0/-17)
doc/examples/cloud-config-resolv-conf.txt (+0/-20)
doc/examples/cloud-config-rh_subscription.txt (+0/-49)
doc/examples/cloud-config-rsyslog.txt (+0/-46)
doc/examples/cloud-config-run-cmds.txt (+0/-22)
doc/examples/cloud-config-salt-minion.txt (+0/-53)
doc/examples/cloud-config-seed-random.txt (+0/-32)
doc/examples/cloud-config-ssh-keys.txt (+0/-46)
doc/examples/cloud-config-update-apt.txt (+0/-7)
doc/examples/cloud-config-update-packages.txt (+0/-8)
doc/examples/cloud-config-user-groups.txt (+0/-109)
doc/examples/cloud-config-vendor-data.txt (+0/-16)
doc/examples/cloud-config-write-files.txt (+0/-33)
doc/examples/cloud-config-yum-repo.txt (+0/-20)
doc/examples/cloud-config.txt (+0/-645)
doc/examples/include-once.txt (+0/-7)
doc/examples/include.txt (+0/-5)
doc/examples/kernel-cmdline.txt (+0/-18)
doc/examples/part-handler-v2.txt (+0/-38)
doc/examples/part-handler.txt (+0/-23)
doc/examples/plain-ignored.txt (+0/-2)
doc/examples/seed/README (+0/-22)
doc/examples/seed/meta-data (+0/-30)
doc/examples/seed/user-data (+0/-3)
doc/examples/upstart-cloud-config.txt (+0/-12)
doc/examples/upstart-rclocal.txt (+0/-12)
doc/examples/user-script.txt (+0/-8)
doc/merging.rst (+0/-188)
doc/rtd/conf.py (+0/-77)
doc/rtd/index.rst (+0/-31)
doc/rtd/static/logo.svg (+0/-89)
doc/rtd/topics/availability.rst (+0/-20)
doc/rtd/topics/capabilities.rst (+0/-24)
doc/rtd/topics/datasources.rst (+0/-200)
doc/rtd/topics/dir_layout.rst (+0/-81)
doc/rtd/topics/examples.rst (+0/-133)
doc/rtd/topics/format.rst (+0/-159)
doc/rtd/topics/hacking.rst (+0/-1)
doc/rtd/topics/merging.rst (+0/-5)
doc/rtd/topics/modules.rst (+0/-342)
doc/rtd/topics/moreinfo.rst (+0/-12)
doc/sources/altcloud/README.rst (+0/-87)
doc/sources/azure/README.rst (+0/-134)
doc/sources/cloudsigma/README.rst (+0/-38)
doc/sources/cloudstack/README.rst (+0/-29)
doc/sources/configdrive/README.rst (+0/-123)
doc/sources/digitalocean/README.rst (+0/-21)
doc/sources/kernel-cmdline.txt (+0/-48)
doc/sources/nocloud/README.rst (+0/-71)
doc/sources/opennebula/README.rst (+0/-142)
doc/sources/openstack/README.rst (+0/-24)
doc/sources/ovf/README (+0/-83)
doc/sources/ovf/example/ovf-env.xml (+0/-46)
doc/sources/ovf/example/ubuntu-server.ovf (+0/-130)
doc/sources/ovf/make-iso (+0/-156)
doc/sources/ovf/ovf-env.xml.tmpl (+0/-28)
doc/sources/ovf/ovfdemo.pem (+0/-27)
doc/sources/ovf/user-data (+0/-7)
doc/sources/smartos/README.rst (+0/-149)
doc/status.txt (+0/-53)
doc/userdata.txt (+0/-79)
doc/var-lib-cloud.txt (+0/-63)
doc/vendordata.txt (+0/-53)
packages/bddeb (+0/-260)
packages/brpm (+0/-277)
packages/debian/changelog.in (+0/-6)
packages/debian/cloud-init.postinst (+0/-16)
packages/debian/cloud-init.preinst (+0/-20)
packages/debian/compat (+0/-1)
packages/debian/control.in (+0/-29)
packages/debian/copyright (+0/-29)
packages/debian/dirs (+0/-6)
packages/debian/rules.in (+0/-23)
packages/debian/watch (+0/-2)
packages/redhat/cloud-init.spec.in (+0/-200)
packages/suse/cloud-init.spec.in (+0/-163)
requirements.txt (+0/-36)
setup.py (+0/-215)
systemd/cloud-config.service (+0/-16)
systemd/cloud-config.target (+0/-11)
systemd/cloud-final.service (+0/-17)
systemd/cloud-init-generator (+0/-133)
systemd/cloud-init-local.service (+0/-22)
systemd/cloud-init.service (+0/-18)
systemd/cloud-init.target (+0/-6)
sysvinit/debian/cloud-config (+0/-64)
sysvinit/debian/cloud-final (+0/-66)
sysvinit/debian/cloud-init (+0/-64)
sysvinit/debian/cloud-init-local (+0/-63)
sysvinit/freebsd/cloudconfig (+0/-35)
sysvinit/freebsd/cloudfinal (+0/-35)
sysvinit/freebsd/cloudinit (+0/-35)
sysvinit/freebsd/cloudinitlocal (+0/-35)
sysvinit/gentoo/cloud-config (+0/-13)
sysvinit/gentoo/cloud-final (+0/-11)
sysvinit/gentoo/cloud-init (+0/-12)
sysvinit/gentoo/cloud-init-local (+0/-13)
sysvinit/redhat/cloud-config (+0/-121)
sysvinit/redhat/cloud-final (+0/-121)
sysvinit/redhat/cloud-init (+0/-121)
sysvinit/redhat/cloud-init-local (+0/-124)
templates/chef_client.rb.tmpl (+0/-58)
templates/hosts.debian.tmpl (+0/-26)
templates/hosts.freebsd.tmpl (+0/-24)
templates/hosts.redhat.tmpl (+0/-24)
templates/hosts.suse.tmpl (+0/-26)
templates/resolv.conf.tmpl (+0/-30)
templates/sources.list.debian.tmpl (+0/-32)
templates/sources.list.ubuntu.tmpl (+0/-58)
test-requirements.txt (+0/-17)
tests/configs/sample1.yaml (+0/-52)
tests/data/filter_cloud_multipart.yaml (+0/-30)
tests/data/filter_cloud_multipart_1.email (+0/-11)
tests/data/filter_cloud_multipart_2.email (+0/-39)
tests/data/filter_cloud_multipart_header.email (+0/-11)
tests/data/merge_sources/expected1.yaml (+0/-1)
tests/data/merge_sources/expected10.yaml (+0/-7)
tests/data/merge_sources/expected11.yaml (+0/-5)
tests/data/merge_sources/expected12.yaml (+0/-5)
tests/data/merge_sources/expected2.yaml (+0/-3)
tests/data/merge_sources/expected3.yaml (+0/-1)
tests/data/merge_sources/expected4.yaml (+0/-2)
tests/data/merge_sources/expected5.yaml (+0/-7)
tests/data/merge_sources/expected6.yaml (+0/-9)
tests/data/merge_sources/expected7.yaml (+0/-38)
tests/data/merge_sources/expected8.yaml (+0/-7)
tests/data/merge_sources/expected9.yaml (+0/-5)
tests/data/merge_sources/source1-1.yaml (+0/-3)
tests/data/merge_sources/source1-2.yaml (+0/-5)
tests/data/merge_sources/source10-1.yaml (+0/-6)
tests/data/merge_sources/source10-2.yaml (+0/-6)
tests/data/merge_sources/source11-1.yaml (+0/-5)
tests/data/merge_sources/source11-2.yaml (+0/-3)
tests/data/merge_sources/source11-3.yaml (+0/-3)
tests/data/merge_sources/source12-1.yaml (+0/-8)
tests/data/merge_sources/source12-2.yaml (+0/-5)
tests/data/merge_sources/source2-1.yaml (+0/-6)
tests/data/merge_sources/source2-2.yaml (+0/-5)
tests/data/merge_sources/source3-1.yaml (+0/-4)
tests/data/merge_sources/source3-2.yaml (+0/-4)
tests/data/merge_sources/source4-1.yaml (+0/-3)
tests/data/merge_sources/source4-2.yaml (+0/-6)
tests/data/merge_sources/source5-1.yaml (+0/-6)
tests/data/merge_sources/source5-2.yaml (+0/-8)
tests/data/merge_sources/source6-1.yaml (+0/-5)
tests/data/merge_sources/source6-2.yaml (+0/-8)
tests/data/merge_sources/source7-1.yaml (+0/-27)
tests/data/merge_sources/source7-2.yaml (+0/-17)
tests/data/merge_sources/source8-1.yaml (+0/-7)
tests/data/merge_sources/source8-2.yaml (+0/-6)
tests/data/merge_sources/source9-1.yaml (+0/-5)
tests/data/merge_sources/source9-2.yaml (+0/-6)
tests/data/mountinfo_precise_ext4.txt (+0/-24)
tests/data/mountinfo_raring_btrfs.txt (+0/-13)
tests/data/roots/simple_ubuntu/etc/networks/interfaces (+0/-3)
tests/data/user_data.1.txt (+0/-15)
tests/data/vmware/cust-dhcp-2nic.cfg (+0/-34)
tests/data/vmware/cust-static-2nic.cfg (+0/-39)
tests/unittests/helpers.py (+0/-346)
tests/unittests/test__init__.py (+0/-220)
tests/unittests/test_builtin_handlers.py (+0/-73)
tests/unittests/test_cli.py (+0/-54)
tests/unittests/test_cs_util.py (+0/-84)
tests/unittests/test_data.py (+0/-559)
tests/unittests/test_datasource/test_altcloud.py (+0/-452)
tests/unittests/test_datasource/test_azure.py (+0/-646)
tests/unittests/test_datasource/test_azure_helper.py (+0/-421)
tests/unittests/test_datasource/test_cloudsigma.py (+0/-99)
tests/unittests/test_datasource/test_cloudstack.py (+0/-87)
tests/unittests/test_datasource/test_configdrive.py (+0/-407)
tests/unittests/test_datasource/test_digitalocean.py (+0/-127)
tests/unittests/test_datasource/test_gce.py (+0/-166)
tests/unittests/test_datasource/test_maas.py (+0/-163)
tests/unittests/test_datasource/test_nocloud.py (+0/-187)
tests/unittests/test_datasource/test_opennebula.py (+0/-300)
tests/unittests/test_datasource/test_openstack.py (+0/-347)
tests/unittests/test_datasource/test_smartos.py (+0/-585)
tests/unittests/test_distros/test_generic.py (+0/-233)
tests/unittests/test_distros/test_hostname.py (+0/-38)
tests/unittests/test_distros/test_hosts.py (+0/-41)
tests/unittests/test_distros/test_netconfig.py (+0/-321)
tests/unittests/test_distros/test_resolv.py (+0/-67)
tests/unittests/test_distros/test_sysconfig.py (+0/-82)
tests/unittests/test_distros/test_user_data_normalize.py (+0/-297)
tests/unittests/test_ec2_util.py (+0/-139)
tests/unittests/test_filters/test_launch_index.py (+0/-132)
tests/unittests/test_handler/test_handler_apt_configure.py (+0/-109)
tests/unittests/test_handler/test_handler_ca_certs.py (+0/-271)
tests/unittests/test_handler/test_handler_chef.py (+0/-192)
tests/unittests/test_handler/test_handler_debug.py (+0/-81)
tests/unittests/test_handler/test_handler_disk_setup.py (+0/-30)
tests/unittests/test_handler/test_handler_growpart.py (+0/-220)
tests/unittests/test_handler/test_handler_locale.py (+0/-67)
tests/unittests/test_handler/test_handler_lxd.py (+0/-134)
tests/unittests/test_handler/test_handler_mounts.py (+0/-133)
tests/unittests/test_handler/test_handler_power_state.py (+0/-127)
tests/unittests/test_handler/test_handler_rsyslog.py (+0/-174)
tests/unittests/test_handler/test_handler_seed_random.py (+0/-227)
tests/unittests/test_handler/test_handler_set_hostname.py (+0/-72)
tests/unittests/test_handler/test_handler_snappy.py (+0/-306)
tests/unittests/test_handler/test_handler_timezone.py (+0/-76)
tests/unittests/test_handler/test_handler_write_files.py (+0/-112)
tests/unittests/test_handler/test_handler_yum_add_repo.py (+0/-68)
tests/unittests/test_helpers.py (+0/-33)
tests/unittests/test_merging.py (+0/-257)
tests/unittests/test_net.py (+0/-130)
tests/unittests/test_pathprefix2dict.py (+0/-44)
tests/unittests/test_registry.py (+0/-28)
tests/unittests/test_reporting.py (+0/-369)
tests/unittests/test_rh_subscription.py (+0/-214)
tests/unittests/test_runs/test_merge_run.py (+0/-54)
tests/unittests/test_runs/test_simple_run.py (+0/-81)
tests/unittests/test_sshutil.py (+0/-171)
tests/unittests/test_templating.py (+0/-119)
tests/unittests/test_util.py (+0/-489)
tests/unittests/test_vmware_config_file.py (+0/-103)
tools/21-cloudinit.conf (+0/-6)
tools/Z99-cloud-locale-test.sh (+0/-98)
tools/build-on-freebsd (+0/-66)
tools/ccfg-merge-debug (+0/-90)
tools/cloud-init-per (+0/-60)
tools/hacking.py (+0/-170)
tools/make-dist-tarball (+0/-21)
tools/make-mime.py (+0/-60)
tools/make-tarball (+0/-39)
tools/mock-meta.py (+0/-454)
tools/motd-hook (+0/-35)
tools/read-dependencies (+0/-29)
tools/read-version (+0/-26)
tools/run-pep8 (+0/-22)
tools/run-pyflakes (+0/-18)
tools/run-pyflakes3 (+0/-2)
tools/tox-venv (+0/-42)
tools/uncloud-init (+0/-141)
tools/validate-yaml.py (+0/-25)
tools/write-ssh-key-fingerprints (+0/-38)
tox.ini (+0/-35)
udev/66-azure-ephemeral.rules (+0/-18)
udev/79-cloud-init-net-wait.rules (+0/-10)
udev/cloud-init-wait (+0/-70)
upstart/cloud-config.conf (+0/-9)
upstart/cloud-final.conf (+0/-10)
upstart/cloud-init-blocknet.conf (+0/-83)
upstart/cloud-init-container.conf (+0/-57)
upstart/cloud-init-local.conf (+0/-16)
upstart/cloud-init-nonet.conf (+0/-66)
upstart/cloud-init.conf (+0/-9)
upstart/cloud-log-shutdown.conf (+0/-19)
JJ Asghar: Pending requested
Diff: 0 lines
Scott Moser: Pending requested
Diff: 3061 lines (+2239/-175)
39 files modified
ChangeLog (+39/-9)
cloud-init.py (+31/-1)
cloudinit/CloudConfig/__init__.py (+1/-1)
cloudinit/CloudConfig/cc_apt_pipelining.py (+53/-0)
cloudinit/CloudConfig/cc_ca_certs.py (+3/-1)
cloudinit/CloudConfig/cc_chef.py (+5/-5)
cloudinit/CloudConfig/cc_landscape.py (+5/-0)
cloudinit/CloudConfig/cc_resizefs.py (+31/-12)
cloudinit/CloudConfig/cc_salt_minion.py (+2/-1)
cloudinit/CloudConfig/cc_update_etc_hosts.py (+1/-1)
cloudinit/DataSource.py (+4/-1)
cloudinit/DataSourceCloudStack.py (+92/-0)
cloudinit/DataSourceConfigDrive.py (+231/-0)
cloudinit/DataSourceEc2.py (+2/-84)
cloudinit/DataSourceMAAS.py (+345/-0)
cloudinit/DataSourceNoCloud.py (+75/-2)
cloudinit/DataSourceOVF.py (+1/-1)
cloudinit/SshUtil.py (+2/-0)
cloudinit/UserDataHandler.py (+2/-2)
cloudinit/__init__.py (+44/-9)
cloudinit/netinfo.py (+5/-5)
cloudinit/util.py (+253/-30)
config/cloud.cfg (+3/-1)
debian.trunk/control (+1/-0)
doc/configdrive/README (+118/-0)
doc/examples/cloud-config-chef-oneiric.txt (+90/-0)
doc/examples/cloud-config-chef.txt (+48/-6)
doc/examples/cloud-config-datasources.txt (+18/-0)
doc/examples/cloud-config.txt (+11/-0)
doc/kernel-cmdline.txt (+48/-0)
doc/nocloud/README (+55/-0)
setup.py (+1/-0)
tests/unittests/test__init__.py (+242/-0)
tests/unittests/test_datasource/test_maas.py (+153/-0)
tests/unittests/test_handler/test_handler_ca_certs.py (+4/-0)
tests/unittests/test_userdata.py (+107/-0)
tests/unittests/test_util.py (+18/-2)
tools/Z99-cloud-locale-test.sh (+92/-0)
tools/run-pylint (+3/-1)
lp:ubuntu/quantal/cloud-init
lp:~smoser/ubuntu/precise/cloud-init/sru
lp:ubuntu/precise-proposed/cloud-init
lp:ubuntu/precise-updates/cloud-init
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Stéphane Graber (stgraber) wrote :

As suggested on IRC, I'm wondering if we couldn't fix the race by doing the following changes:

 - change network-interface to: "start on net-device-added and mounted MOUNTPOINT=/run"
 - change resolvconf to: "start on mounted MOUNTPOINT=/run or starting network-interface"

networking.conf should be fine as it's "local-filesystems and (stopped udevtrigger or container))" which should be triggered quite a bit after virtual-filesystems, so resolvconf should have run already.

Revision history for this message
Steve Langasek (vorlon) wrote :

I don't think there's been sufficient analysis of the root cause here to start changing start conditions on any of the standard jobs.

/etc/init/resolvconf.conf is:

  start on mounted MOUNTPOINT=/run

and /etc/init/udev.conf is:

  start on virtual-filesystems

The 'mounted' event is a hook, not a signal; mountall will not continue processing mounts, or emit any other signals, until all jobs depending on 'mounted' have finished.

This means that the resolvconf job is *guaranteed* to create the /run/resolvconf/interface directory before udev is started. And until udev is started, there should be no 'net-device-added' events, so no instances of /etc/init/network-interface.conf.

So something else is going on here. I don't see any bugs in resolvconf.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Problem was tracked down to cloud-init calling "start networking" directly from one of its upstart jobs, which bypasses the start condition of the networking job and triggers the bug.

21:16 < stgraber> [ 4.598897] networking
21:16 < stgraber> [ 5.016864] init: cloud-init-nonet main process (241) killed by TERM signal
21:16 < stgraber> [ 5.632677] resolvconf
21:16 < stgraber> [ 5.955406] virtual
21:16 < stgraber> [ 5.989643] local
21:17 < stgraber> ubuntu@ip-10-202-249-151:~$ grep -r 'start networking' /etc/init/
21:17 < stgraber> /etc/init/cloud-init-nonet.conf: start networking >/dev/null
21:17 < stgraber> smoser: ^
21:18 < stgraber> "start on mounted MOUNTPOINT=/ and stopped cloud-init-local" with clout-init-local being "start on mounted MOUNTPOINT=/"
21:18 < stgraber> so basically as soon as / is mounted, "start networking" is directly triggered by cloud-init-nonet bypassing the start condition

http://paste.ubuntu.com/1120257/ shows the process tree at the time.

affects: resolvconf (Ubuntu) → cloud-init (Ubuntu)
Revision history for this message
Scott Moser (smoser) wrote :

Some conversation in ubuntu-devel yesterday [1]. This seems to have been caused by the fix for bug 800824. That change [2] was done to address failure of networking to appear as started a container. Per Steve, "that looks like a wrong workaround for bug #925122".

Hopefully what that means is we can remove this code and lxc containers still boot correctly even in precise (in which case we should probably SRU that).

--
[1] http://irclogs.ubuntu.com/2012/07/31/%23ubuntu-devel.html#t01:41
[2] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/405

summary: - /sbin/resolvconf -a depends on /run/resolvconf/interface but it may not
- exist
+ cloud-init-nonet runs 'start networking' explicitly
Scott Moser (smoser)
description: updated
description: updated
Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Scott Moser (smoser) wrote :

I was able to recreate the original problem of bug 800824 by doing the following on a precise instance:
 - setup the instance, get source roots of the ubuntu-cloud lxc root:
   for r in oneiric precise quantal; do \
     sudo lxc-create -t ubuntu-cloud -n source-$r-amd64 -- \
        --release $r --arch amd64 --stream daily; done
 - RELEASE="oneiric"
 - sudo apt-get install lxc
 - sudo lxc-clone -o source-$RELEASE-amd64 $RELEASE-amd64
 - edit /var/lib/lxc/$RELEASE-amd64/rootfs/etc/init/cloud-init-nonet.conf
   commenting out the 'start networking' line.
 - sudo lxc-start --name=$RELEASE-amd64
 - Here, you will see the cloud-init nonet timeout, and
   wait for 120 seconds before seeing a tty prompt.
 - sudo lxc-stop -n $RELEASE-amd64
 - sudo lxc-destroy --name=$RELEASE-amd64

If you do *not* comment the line out in cloud-init-nonet.conf, you will
see the cloud-init output and boot quickly.

Repeating the above process for RELEASE=precise will show both cases working
 RELEASE=oneiric with 'start networking' : boots quickly
 RELEASE=oneiric without 'start-networking': timeout
 RELEASE=precise with 'start networking' : boots quickly
 RELEASE=precise without 'start-networking': timeout
 RELEASE=quantal with 'start networking' : boots quickly
 RELEASE=quantal without 'start-networking': timeout

On quantal, the cloud-init output in the timeout case looks like:
  Cloud-init v. 0.7 running 'init-local' at Wed, 22 Aug 2012 20:09:07 +0000. Up 5131.22 seconds.
  cloud-init-nonet waiting 120 seconds for a network device.

What is the correct fix here?

Changed in cloud-init (Ubuntu):
status: Triaged → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

So, after much discussion [1] we came to the realization of why 'start networking' was here.
The issue was that the normal case for cloud-images (and ubuntu as a whole) is that / is mounted "ro" when /sbin/init starts. When this is the case, /run will be mounted before /. However, if / is mounted "rw" then "mountall first scans for all filesystems that don't require further processing and reports them mounted", so "mounted MOUNTPOINT=/" occurs before virtual-filesystems.

if mounted MOUNTPOINT=/ occurs before virtual-filesystems, then udev will not start because it depends on virtual-filesystems. That means means networking wont get started either, as ifup occurs via udev events for discovered network adapters. As a result, we hang in cloud-init-nonet waiting for networking devices to come up that never will.

Note, slangasek pointed out that while /run is guaranteed to be mounted before / (except withotu 'ro'), virtual-filesystems is *not* guaranteed. So, even with 'ro' root, a race condition could occur here. It would also, then imply that cloud-init cannot actually use some filesystems in cloud-init-local (such has /proc or /sys).

slangasek suggested that a fix for bug 643289 would improve the situation here as the virtual-filesystems could all move forward even while the 'mounted MOUNTPOINT=/' were processing (currently it blocks *everything*). That would mean we wouldn't deadlock here.

--
[1] http://irclogs.ubuntu.com/2012/09/05/%23ubuntu-devel.html#t18:16

description: updated
Revision history for this message
Steve Langasek (vorlon) wrote :

After reviewing the patches on bug #643289, I see that some further work will be needed because those patches only adjust the 'mounting' event, not the 'mounted' event. IMHO we should do both.

Steve Langasek (vorlon)
Changed in cloud-init (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Scott Moser (smoser) wrote :

Finding a better solution to the 'start networking' that we have here has lead me to open bug 1047712.
The case that I was debugging was actually not even improved by the 'start networking'. The issue that lead us here was that the ephemeral images for maas that we were trying to refresh have had issues coming up and woudl hit this timeout.

slangasek helped me to come up with a solution that has gotten us closer.
I've replaced 'start networking' with "virtual-filesystems-emitter" (see attached) in the cloud-init-nonet job.
Essentially that just gets virtual-filesystems event emitted if it hasnt already been emitted, which jump starts the networking code even while mounted MOUNTPOINT=/ is being processed.

Revision history for this message
Steve Langasek (vorlon) wrote :

Scott, can you please try a test package of mountall built from lp:ubuntu/mountall? I think I have both the mounting and mounted events handled correctly there (and in my tests, seemingly faster).

Testing in scenarios other than the cloud boot case would also certainly be helpful.

Revision history for this message
Scott Moser (smoser) wrote :

I built a deb from lp:ubuntu/mountall revno 415.
installed it into a ephemeral image for quantal
commented out 'start networking' from /etc/init/cloud-init-nonet.conf, and got the /var/log/kern.log at http://paste.ubuntu.com/1198891/ .

it seems something is blockign virutal-filesystems event from occuring still.

Revision history for this message
Scott Moser (smoser) wrote :

Steve,
I've also now tried your mountall in an lxc chroot, and it did not resolve my issue there either.
I've set up a lxc container as described above, commented out the 'start networking' and then:
$ sudo ln -sf console /path/to/rootfs/dev/kmsg
$ sudo lxc-start --name=$REL-amd64 --console console.log -- /sbin/init --verbose

The console.log is at http://paste.ubuntu.com/1199472/

Revision history for this message
Steve Langasek (vorlon) wrote :

Attaching plaintext console log.

Revision history for this message
Steve Langasek (vorlon) wrote :

The log for the container seems to show reasonable ordering for the events. It also shows:

cloud-init-nonet waiting 120 seconds for a network device.
cloud-init-nonet gave up waiting for a network device.
<6>init: cloud-init-nonet main process (31) exited normally
<6>init: cloud-init-nonet goal changed from start to stop
<6>init: cloud-init-nonet state changed from running to stopping
<6>init: Handling stopping event

However, this is a container so AFAICS there *isn't* a network interface to be brought up. Isn't that a different problem?

Revision history for this message
Steve Langasek (vorlon) wrote :

network-interface (lo) is the only interface job shown, so either something is going wrong when bringing up this interface that prevents /etc/network/if-up.d/upstart from emitting the static-network-up event, or there are other network interfaces configured in /etc/network/interfaces that have no corresponding udev events, but do come up successfully in response to ifup -a. Can you attach /etc/network/interfaces from this container, for reference?

Revision history for this message
Scott Moser (smoser) wrote :

There is an 'eth0' defined in /etc/network/interfaces.
And, there is a network device eth0

$ sed -e "/^$/d" -e '/^#/d' etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp

$ ls -l /run/network
total 4
-rw-r--r-- 1 root root 16 Sep 12 18:33 ifstate
-rw-r--r-- 1 root root 0 Sep 12 18:33 ifup.eth0
-rw-r--r-- 1 root root 0 Sep 12 18:31 ifup.lo
drwxr-xr-x 2 root root 40 Sep 12 18:33 static-network-up-emitted

Note, that eth0 came up 2 minutes (120 seconds) after lo.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.0~bzr659-0ubuntu1

---------------
cloud-init (0.7.0~bzr659-0ubuntu1) quantal; urgency=low

  * New upstream snapshot.
    * add signal handlers to more cleanly exit
    * add logging fallback in case logging fails
    * documentation fix for landscape config (LP: #1042764)
    * do not write a comment in /etc/hostname (LP: #1052664)
    * fix incorrect ubuntu mirrors for 'ports' arches [Robbie Basak]
    * fix generation of hostname based on ip address in datasource
      (LP: #1050962) [Andy Grimm]
    * remove 'start networking' from cloud-init-nonet and replace it
      with container specific fixes (LP: #1031065)
    * fix landscape configuration so client will run (LP: #1042758)
    * enable all available datasources (adding AltCloud and None)
    * fix bug where user data scripts re-ran on upgrade from 10.04 versions
      (LP: #1049146)
 -- Scott Moser <email address hidden> Wed, 19 Sep 2012 22:08:51 -0400

Changed in cloud-init (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

Just a note, an SRU of this depends on SRU of mountall 643289.

Changed in cloud-init (Ubuntu Precise):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

I built a precise cloud-init with the 'start networking' removed and tried starting an lxxc instace with that from current precise daily (with mountall 2.36.3). This issue persisted.

Steve, did you tihnk this *should* be fixed in precise?

Revision history for this message
Steve Langasek (vorlon) wrote :

With the current mountall in precise, 2.36.3, yes - there shouldn't be any further need to call 'start networking' directly. Are the symptoms exactly the same as before the mountall fix?

Scott Moser (smoser)
description: updated
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Scott Moser (smoser) wrote :

Verified. See below.

$ sudo apt-get update && sudo apt-get install -y lxc
$ lsb_release -c
Codename: precise
$ dpkg-query --show lxc
lxc 0.7.5-3ubuntu66

$ sudo lxc-clone -o source-precise-amd64 -n patched
$ cat /var/lib/lxc/source-precise-amd64/rootfs/etc/cloud/build.info
build_name: server
serial: 20130218

$ sudo lxc-clone -o source-precise-amd64 -n patched
$ url="https://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.5/+build/4312778/+files/cloud-init_0.6.3-0ubuntu1.5_all.deb"
$ deb="cloud-init_0.6.3-0ubuntu1.5_all.deb"

$ wget -O "$deb" "$url"
$ sudo cp $deb "/var/lib/lxc/patched/rootfs/tmp"
$ sudo LANG=C chroot "/var/lib/lxc/patched/rootfs" dpkg -i "/tmp/$deb"
$ sudo lxc-start -n patched -- /sbin/init --verbose

Verify can log in as ubuntu/ubuntu.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.5

---------------
cloud-init (0.6.3-0ubuntu1.5) precise-proposed; urgency=low

  * debian/update-grub-legacy-ec2: consider kernels bootable on ec2
    that are named -generic, in addition to -virtual. This fixes a problem
    where the kernels installed by linux-lts-quantal were not added to
    /boot/grub/menu.lst (LP: #1005551)
  * debian/patches/lp-1077020-fix-ca-certificates-blanklines.patch: fix
    adding of empty lines in ca-certificates file (LP: #1077020)
  * debian/patches/lp-1031065-nonet-not-start-networking.patch: do not 'start
    networking' in cloud-init-nonet upstart job. Doing so can cause networking
    to be started earlier than it should be. Instead, add a
    cloud-init-container job that runs only in a container and emits
    net-device-added (LP: #1031065).
  * debian/patches/lp-1037567-add-config-drive-v2-support.conf:
    backport support for config-drive-v2 which is part of Openstack Nova in
    Folsom and later. (LP: #1037567) (LP: #1100545)
 -- Scott Moser <email address hidden> Wed, 16 Jan 2013 19:37:57 -0500

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.