Breaks machine without IPv4: "Route info failed"

Addendum: the cloud-config URL was based on a hostname (which resolved in both IPv4 and IPv6), not an IPv6 address. I'll try again but without IPv4 resolution for that hostname.

/var/log/cloud-init.log would be useful.

I'm sure it would, but see the part where I can't log in...

I got a very similar response by deploying a regular dual-stack node, commenting out it network interface's DHCPv4 config in /etc/network/interfaces, and rebooting.

Something very strange happened: the network interface comes up with IPv6 only, which in itself is what you'd expect. But it gets only a link-local address! The cloud-init log was full of failed attempts to read the instance-id from the (correct-looking) MAAS metadata URL: name or service unknown.

Log from a node that was initially deployed dual-stack, then was rebooted IPv4 disabled on its eth0, and finally (to get the logs across) rebooted again with IPv4 enabled again.

So this is not the original problem case, but it looks very similar.

Output log, also from the node that was deployed dual-stack, then rebooted as pure IPv6, then rebooted dual-stack again.

The long pause after the route_info error seems to be a DNS wait. Once I add a dns-nameservers entry for the IPv6 stanza that gets written into the node's /etc/network/interfaces during install, the wait disappears. The route_info error is still there though, and networking is still broken.

In other news, nodes deployed in this way do get to the Deployed state! That means that cloud-init does manage to contact the region controller API, and the only way I can see that happening is through IPv6. And the region controller hostname resolves only to a global address, not a link-local one.

So it looks as if everything works just fine for a while and then it all gets shut down somehow during the boot process.

Log from a node that was deployed as dual-stack, then got rebooted as IPv6-only (where it failed to configure its network interface).

The reconfiguration as IPv6-only involved:
 * Removing the ‘iface eth0 inet dhcp’ line from /etc/network/interfaces.
 * Removing the IPv4 nameserver line from /etc/resolv.conf.

Output log matching the preceding cloud-init.log: node is first deployed as dual-stack, then reconfigured to be IPv6-only, and rebooted.

And, this time I can confirm that during the deployment boot, the node successfully made its metadata requests through IPv6. *However* it doesn't seem to use the statically configured IPv6 address!

When I deploy a node with IPv4 disabled, on a network with router advertisements, the node seems to use an autoconfigured IPv6 address at one point in the middle of the procedure. First, during installation when IPv4 still works, cloud-init retrieves its preseed and curtin retrieves various metadata items, all through IPv4. Then, about 35 seconds later, ‘wget’ on the node does a POST to the metadata API from the autoconfigured IPv6 address. Then, some 42 seconds after that, the node retrieves more metadata items from its link-local IPv6 address.

So I see the node using IPv6, but I never get to see it using its configured IPv6 address.

I reproduced this locally, all on a 'virbr0' network.
essentially, it boils down to this /etc/network/interfaces works:
 | auto lo
 | iface lo inet loopback
 | auto eth0
 | iface eth0 inet static
 | netmask 255.255.255.255
 | address 10.254.254.253
 | iface eth0 inet6 static
 | netmask 64
 | address 2001:db8::1:3
 | dad-attempts 0

But removing the 'iface eth0 inet static' stanza (or even actually moving it to below the 'inet6 static' stanza) ends up configuring only the link local link for ipv6, and cloud-init fails trying to reach the maas metadata service.

After the system boots, I can log in and run 'ifdown eth0; ifup eth0' and the link comes up correctly. However before doing so I see only:

| 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
| inet 127.0.0.1/8 scope host lo
| valid_lft forever preferred_lft forever
| inet6 ::1/128 scope host
| valid_lft forever preferred_lft forever
| 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
| link/ether 52:54:00:03:c3:9b brd ff:ff:ff:ff:ff:ff
| inet6 fe80::5054:ff:fe03:c39b/64 scope link
| valid_lft forever preferred_lft forever
| 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
| link/ether 52:54:00:0e:3a:3b brd ff:ff:ff:ff:ff:ff

This would seem quite possibly like a duplicate of bug 1352255 .
it seems like the work around above (just writing a ipv4 stanza) might be acceptable for the time being.

It doesn't look like bug 1352255 for me. When I install 14.04 from USB stick and give it the exact same /etc/network/interfaces, it works as expected. But when the same machine boots a MAAS-installed 14.04 image with that configuration, it's broken.

I found a simpler workaround, which unfortunately only seems to work on 14.04: when IPv4 is disabled, configure each network interface with an IPv4 address of 0.0.0.0 (and arbitrary netmask, I guess).

It's not pleasant that it doesn't work for 12.04 or 13.10, but hopefully for something as radically modern as disabling IPv4 that's probably not a big obstacle.

My other worry is that it might cause trouble for machines with multiple network cards. I'm going out to get myself some more network cards for testing.

Jeroen,
  The 'route info' bug is a bug in cloud-init, but isnt fatal. its just cloud-init failing to parse 'route' output as you dont have ipv4 routes that it is expecting.
  The fact that ipv6 networking does not come up automatically on a device if ipv4 config is not provided is not provided is not cloud-init specific. It really feels to me that this is somehow racey as I've now reproduced that if I just *move* the 'iface eth0 inet static' to below the inet6 , that it will fail, but above it makes it pass.

Ie, this is broken:
auto eth0
iface eth0 inet6 static
  netmask 64
  address 2001:db8::1:3
  dad-attempts 0
iface eth0 inet static
  netmask 255.255.255.255
  address 127.0.254.1

but moving that inet static above works fine.

I've stripped the kernel timestamps out of the 2 logs above. The difference in configuration is only as described in comment 16. That points squarely to 'startpar-bridge' being the racey piece.

Do you think there might be some difference between the “regular” 14.04 install image and the ones MAAS uses that would cause the race to work out differently between them? The outcomes have seemed pretty consistent for me: a regular install works without IPv4 configuration, a MAAS install does not.

see attachment
# after patching /etc/init/network-interface.conf with '--verbose'
# on the 'ifup --allow auto' line, we get this output in
# /var/log/upstart/network-interface-eth0.conf
#
# Things to note:
# * it seems potentially wrong is that 'upstart'
# is called via if-up.d after the *first* address is brough up
# rather than after both addresses are up. This could cause race
# conditions or unexpected behavior as that is
# what runs 'initctl emit --no-wait static-network-up'
# * modprobe -q net-pf-10 doesn't do anything on an ubuntu kernel (built-in)
# * cloud image *does* have 'ethtool', which means ifup.d/ethtool
# does some things
# * I was unable to recreate failure with having no network configured
# and running the ipv6 things from a script after system all the way up
# * order in /etc/network/interfaces most certainly does matter

ok. so more info here.
 * cloud-init does not cause the problem.
   I've disabled cloud-init's upstart jobs and the behavior does not change.
 * ethtool is not involved. i've chmod -x /sbin/ethtool.
 * dad (detect address duplication) seems to not be involved.
   I've added 'pre-up echo 0 > /proc/sys/net/ipv6/conf/eth0/accept_dad' and
   'dad-attempts 0' to the stanza with no change in behavior.
 * in a 'ipv6 only' setup, which ever address is first is silently dropped
   Ie, if you want ipv6 only, then just do this:
      auto eth0
      iface eth0 inet6 static
        netmask 128
        address ::2
      iface eth0 inet6 static
        netmask 64
        address 2001:db8::1:3
   The localhost address will just be dropped.
   In my logging, I can also tell that it is gone by the time the second
   address is added (probably sub-seconds later).

I've wrapped /sbin/ip to see whats going on.
when /etc/network/interfaces has:
 auto eth0
 iface eth0 inet6 static
   netmask 64
   address 2001:db8::1:3

With that wrapping in place (and the retry as seen in the attachment), I see the following in /run/ip.log

 /sbin/ip link set dev eth0 up
 /sbin/ip -6 addr add 2001:db8::1:3/64 dev eth0
 /sbin/ip addr show dev eth0
 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
     inet6 2001:db8::1:3/64 scope global tentative
        valid_lft forever preferred_lft forever
     inet6 fe80::5054:ff:fe12:3456/64 scope link tentative
        valid_lft forever preferred_lft forever
 re-running: -6 addr add 2001:db8::1:3/64 dev eth0
 RTNETLINK answers: File exists
 2
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 tentative
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 tentative
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 tentative
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 tentative
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 tentative
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 tentative
 /sbin/ip -o -6 address list dev eth0 to 2001:db8::1:3/64 dadfailed

we most certainly *did* bring up the interface and assign it the ip address. The 'tentative' flag goes away if you disable dad via /proc. And the address was there a very short time after. The group of 'tentative' and 'dadfailed' are /lib/ifupdown/settle-dad.sh trying to wait until dad has finished. they're ignorable, as it exits 0, and 'dad-attmempts 0' does not change behavior.

Something is removing address from the interface, and doing so without using the 'ip' command.

ok. so, i now know what happens, and why adding anything before the ipv6 address "fixes" the problem.
boot happens like this:
 * system boot
 * net-device-added fires network-interface.conf
 * net-device-added runs ifup --allow auto
 * ifup --allow auto reads /etc/network/interfaces *in order*
 * after the *first* stanza is brought up, /etc/network/if-up.d/upstart is invoked
 * /etc/network/if-up.d/upstart finds all interfaces to be up and emits static-network-up
 * /etc/init/failsafe.conf is stopped, freeing procps.conf to start
 * procps.conf runs sysctl on files in /etc/sysctl.d/*.conf
    which includes /etc/sysctl.d/10-ipv6-privacy.conf

ipv6-privacy.conf ends up killing the existing addresses.

Note, cloud-init's route info is busted for ipv6. That should be fixed by fixing bug 925145.

Thanks for digging this up! Are you recommending that to disable IPv4 we generate double IPv6 stanzas, with the first one having a bogus address? I filed bug 1379639 to remind us to check up on this later.

(BTW I guess ::2 isn't really a localhost address in IPv6; there's only ::1, not a big loopback network like in IPv4.)