cloud-init misconfigure the network on SLES

OK, thanks for the logs. Could you re-attach those running via sudo (or as root)? The default user on SLES does not have permissions to read the journal.

What I see so far looks like networking did not come up after cloud-init-local.service completes and writes out a network config.

2019-09-11 18:00:15,242 - stages.py[INFO]: Applying network configuration from ds bringup=False: {'ethernets': {'eth0': {'set-name': 'eth0', 'match': {'macaddress': u'00:0d:3a:6e:6f:8f'}, 'dhcp4': True}}, 'version': 2}

This results in the following files being written:

% cat test_azure_sles/etc/sysconfig/network/ifcfg-eth0
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:0d:3a:6e:6f:8f
NM_CONTROLLED=no
ONBOOT=yes
STARTMODE=auto
TYPE=Ethernet
USERCTL=no

Upstream cloud-init on SLES does not generate/update /etc/resolv.conf but in the logs the cloud-init in does:

2019-09-11 18:00:15,246 - util.py[DEBUG]: Writing to /etc/sysconfig/network/ifcfg-eth0 - wb: [644] 191 bytes
2019-09-11 18:00:15,247 - util.py[DEBUG]: Reading from /etc/resolv.conf (quiet=False)
2019-09-11 18:00:15,247 - util.py[DEBUG]: Read 795 bytes from /etc/resolv.conf
2019-09-11 18:00:15,247 - util.py[DEBUG]: Writing to /etc/resolv.conf - wb: [644] 866 bytes

At first, I thought maybe it was missing this commit:

% git show b74ebca563a21332b29482c8029e7908f60225a4
commit b74ebca563a21332b29482c8029e7908f60225a4
Author: Robert Schweikert <email address hidden>
Date: Wed Jan 23 22:35:32 2019 +0000

    net/sysconfig: do not write a resolv.conf file with only the header.

    Writing the file with no dns information may prevent distro tools
    from writing a resolv.conf file with dns information obtained from
    a dhcp server.

diff --git a/cloudinit/net/sysconfig.py b/cloudinit/net/sysconfig.py
index ae41f7b..fd8e501 100644
--- a/cloudinit/net/sysconfig.py
+++ b/cloudinit/net/sysconfig.py
@@ -557,6 +557,8 @@ class Renderer(renderer.Renderer):
             content.add_nameserver(nameserver)
         for searchdomain in network_state.dns_searchdomains:
             content.add_search_domain(searchdomain)
+ if not str(content):
+ return None
         header = _make_header(';')
         content_str = str(content)
         if not content_str.startswith(header):
@@ -666,7 +668,8 @@ class Renderer(renderer.Renderer):
             dns_path = util.target_path(target, self.dns_path)
             resolv_content = self._render_dns(network_state,
                                               existing_dns_path=dns_path)
- util.write_file(dns_path, resolv_content, file_mode)
+ if resolv_content:
+ util.write_file(dns_path, resolv_content, file_mode)
         if self.networkmanager_conf_path:
             nm_conf_path = util.target_path(target,
                                             self.networkmanager_conf_path)
diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
index d679e92..5313d2d 100644
--- a/tests/unittests/test_net.py
+++ b/tests/unittests/test_net.py
@@ -2098,6 +2098,10 @@ TYPE=Ethernet
 USERCTL=no
 """
         self.assertEqual(expected, found[nspath + 'ifcf...

The original description certainly indicates that cloud-init still writes an "empty" resolv.conf file. However this issues has been addressed for a while thus something does not fit together. It makes no sense that this would only show up in Azure and in the bug that triggered investigation and ultimately led to a fix it has been confirmed as fixed.

Attached the cloud-init collected logs by a root user.
Here is the contents of the /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit.
;
### /etc/resolv.conf file autogenerated by netconfig!
#
# Before you change this file manually, consider to define the
# static DNS configuration using the following variables in the
# /etc/sysconfig/network/config file:
# NETCONFIG_DNS_STATIC_SEARCHLIST
# NETCONFIG_DNS_STATIC_SERVERS
# NETCONFIG_DNS_FORWARDER
# or disable DNS configuration updates via netconfig by setting:
# NETCONFIG_DNS_POLICY=''
#
# See also the netconfig(8) manual page and other documentation.
#
# Note: Manual change of this file disables netconfig too, but
# may get lost when this file contains comments or empty lines
# only, the netconfig settings are same with settings in this
# file and in case of a "netconfig update -f" call.
#
### Please remove (at least) this line when you modify the file!

And the contents of /etc/sysconfig/network/ifcfg-eth0
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:0d:3a:6e:6f:8f
NM_CONTROLLED=no
ONBOOT=yes
STARTMODE=auto
TYPE=Ethernet
USERCTL=no

Also please note that I am running "ifup eth0" to be able to ssh into the machine, but this is just a local change.

Cloud-init is writing a /etc/resolv.conf that only has commented lines and the cloud-init header, When i debugged the following code in sysconfig.py

        content = resolv_conf.ResolvConf("")
        if existing_dns_path and os.path.isfile(existing_dns_path):
            content = resolv_conf.ResolvConf(util.load_file(existing_dns_path))
        for nameserver in network_state.dns_nameservers:
            content.add_nameserver(nameserver)
        for searchdomain in network_state.dns_searchdomains:
            content.add_search_domain(searchdomain)
        if not str(content):
            return None

the content is copied from the old /etc/resolv.conf but there is no name servers or search domains that are added to the content from the network_state.

Who is responsible for adding the name server and the search domain to the network_state ?

When I manually added explicit name server and search domain from DataSourceAzure.py in the parse_network_config function as following:

            netconfig = {'version': 2, 'ethernets': {}}
            LOG.debug('Azure: generating network configuration from IMDS')
            netconfig['nameservers'] = {}
            netconfig['nameservers']['addresses'] = '168.63.129.16'
            netconfig['nameservers']['search'] = 'xkf00b0rtzgejkug4xc2pcinre.xx.internal.cloudapp.net'
            network_metadata = imds_metadata['network']

And then changed the _V2_common in network_state.py to build the name_cmd with the "address" key instead of the "addresses" key. Because the "address" key is required in the handle_nameserver function which is called from _v2_common as following:

    def _v2_common(self, cfg):
        LOG.debug('v2_common: handling config:\n%s', cfg)
        if 'nameservers' in cfg:
            search = cfg.get('nameservers').get('search', [])
            dns = cfg.get('nameservers'...

Thank you for the logs.

> Who is responsible for adding the name server and the search domain to the network_state ?

For Azure, there will not be any nameserver/search values added to network state as Azure's network-config is to DHCP only on eth0 and The DHCP _client_ from the OS runs will update /etc/resolv.conf

Looking at the log, it appears that Wickedd does run DHCP, but cloud-init.service started too early.

# here wicked starts
Sep 11 17:59:55.140087 localhost systemd[1]: Started System Logging Service.
Sep 11 17:59:55.796842 localhost systemd[1]: Started wicked DHCPv4 supplicant service.
Sep 11 17:59:55.805347 localhost systemd[1]: Started wicked DHCPv6 supplicant service.
Sep 11 17:59:55.812068 localhost systemd[1]: Started wicked AutoIPv4 supplicant service.
Sep 11 17:59:55.819849 localhost systemd[1]: Starting wicked network management service daemon...
Sep 11 17:59:55.933440 localhost systemd[1]: Started wicked network management service daemon.
Sep 11 17:59:55.940875 localhost systemd[1]: Starting wicked network nanny service...
Sep 11 17:59:56.022347 localhost systemd[1]: Started wicked network nanny service.

# then cloud-init-local.service
Sep 11 18:00:07.462643 localhost cloud-init[769]: Cloud-init v. 19.1 running 'init-local' at Wed, 11 Sep 2019 18:00:07 +0000. Up 37.63 seconds.

# then cloud-init-local runs dhclient to bring the network up to crawl Azure IMDS and write out
# network config file (/etc/sysconfig/network/ifcfg-eth0
Sep 11 18:00:09.304202 localhost dhclient[834]: Internet Systems Consortium DHCP Client 4.3.3
Sep 11 18:00:09.304631 localhost dhclient[834]: Copyright 2004-2015 Internet Systems Consortium.
Sep 11 18:00:09.304827 localhost dhclient[834]: All rights reserved.
Sep 11 18:00:09.305018 localhost dhclient[834]: For info, please visit https://www.isc.org/software/dhcp/
Sep 11 18:00:09.305262 localhost dhclient[834]:
Sep 11 18:00:09.415443 localhost dhclient[834]: Listening on LPF/eth0/00:0d:3a:6e:6f:8f
Sep 11 18:00:09.415458 localhost dhclient[834]: Sending on LPF/eth0/00:0d:3a:6e:6f:8f
Sep 11 18:00:09.415938 localhost kernel: NET: Registered protocol family 17
Sep 11 18:00:09.415496 localhost dhclient[834]: Sending on Socket/fallback
Sep 11 18:00:09.415533 localhost dhclient[834]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4 (xid=0x1d3935b9)
Sep 11 18:00:09.431456 localhost dhclient[834]: DHCPOFFER from 168.63.129.16
Sep 11 18:00:14.432053 localhost dhclient[834]: DHCPREQUEST on eth0 to 255.255.255.255 port 67 (xid=0x1d3935b9)
Sep 11 18:00:14.449380 localhost dhclient[834]: DHCPACK from 168.63.129.16 (xid=0x1d3935b9)
Sep 11 18:00:14.460401 localhost dhclient[834]: bound to 10.0.0.17 -- renewal in 4294967295 seconds.

# Now cloud-init.service runs and *expects* networking to be up, but it isn't
Sep 11 18:00:15.621036 momousta-vm-945207600 cloud-init[861]: Cloud-init v. 19.1 running 'init' at Wed, 11 Sep 2019 18:00:15 +0000. Up 45.81 seconds.
Sep 11 18:00:15.636134 momousta-vm-945207600 cloud-init[861]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
Sep 11 18:00:15.648761 momousta-vm-945207600 cloud-init[861]: ci-info: +--------+-------+-----------+--...

On Thu, Sep 12, 2019 at 18:01:39 -0000, Ryan Harper wrote:
> I think we need to instead skip writing resolv.conf unless we have
> content to add
>
> % git diff cloudinit/net/sysconfig.py
> diff --git a/cloudinit/net/sysconfig.py b/cloudinit/net/sysconfig.py
> index be5dede..12c6e36 100644
> --- a/cloudinit/net/sysconfig.py
> +++ b/cloudinit/net/sysconfig.py
> @@ -585,7 +585,11 @@ class Renderer(renderer.Renderer):
> content.add_nameserver(nameserver)
> for searchdomain in network_state.dns_searchdomains:
> content.add_search_domain(searchdomain)
> - if not str(content):
> + # content may be non-empty if an existing were present so also
> + # skip writing a resolv conf if we don't have any nameservers or
> + # search paths
> + if not any([len(network_state.dns_nameservers),
> + len(network_state.dns_searchdomains)]):
> return None
> header = _make_header(';')
> content_str = str(content)

The net effect would be the same in end, but perhaps the intent of the
code would be clearer if this check were at the top of the function?

(If we don't have any content to add, there's no need to even read in
the existing resolv.conf file... let alone pass over both of the no-op
"for" loops.)

       Nathan

Based on the content in the description of the bug (the
previously-existing /etc/resolv.conf), cloud-init should *never* be
writing that file when 'netconf' is involved. It seems rather that
resolv.conf is generated based on contents of
/etc/sysconfig/network/config .

Maybe in the case that there was no dns information, cloud-init should nto
write /etc/sysconfig/network/config . It is not obvious to me that there
is a clean solution here, but if we take the policy in cloud-init of "do
not ever render empty dns settings", then that means a system that had old
dns settings but got no dns settings on this new instance id will continue
on with the old settings. DHCP muddies those waters for sure. Static
networking with explicitly empty dns settings probably is an edge case.

With regard to having to bring up the interface manually:
> I had to bring it up by executing "ifup eth0" from Azure.py code after
> the network config is applied. This way I was able to ssh into the VM.

I think that Ryan identified some incorrect ordering with cloud-init and
wicked that nees to be fixed.

@Nathan, definitely an improvement, I'll update my merge proposal.

@Scott

For statically configured DNS settings, writting to /etc/resolv.conf is functional but not optimal on SUSE; doing so disables the netconfig automatic management. As an improvement, cloud-init should look to write these values as NETCONFIG_ variables in /etc/sysconfig/network/config

https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha.network.html#sec.network.manconf.files.resolv

In the case for DHCP, our current policy allows static DNS (we've also called it "global" dns settings, per type:nameserver in v1 config) to apply to any interface that has static ip configuration but no explicit nameserver/search settings. Interfaces with DHCP do not get the "global" dns settings applied.

We've room for improvement in netplan (v2) where there are explicit DNS overrides for DHCP:

https://github.com/CanonicalLtd/netplan/blob/master/doc/netplan.md#dhcp-overrides

Actually cloud-init.service has "Before=wicked.service" when I changed it to "After=wicked.service", the eth0 interface was brought up without me explicitly bringing it up. I'm attaching the logs for that VM.
When I checked the "cloud-init.service" file in cloud-init 18.2, It had "After=wicked.service" but in 18.5 and 19.1 it has "Before=wicked.service". I brought up this point before in an offline discussion with Robert and he indicated that in 18.5 a dependency of cloud-init on dhcp-client was added to support the process flow intended by upstream and support dynamic network configuration via cloud-init.
@Robert, can you please elaborate more on that?

That's this commit.

commit e0084a599d5259e507298175dad8a1e719017895
Author: Robert Schweikert <email address hidden>
Date: Wed Oct 24 14:34:50 2018 +0000

    systemd: On SUSE ensure cloud-init.service runs before wicked

    With db50bc0d9 the sysconfig renderer was enabled for
    openSUSE and SUSE Linux Enterprise. This requires that
    cloud-init.service starts before wicked.

    LP: #1799709

diff --git a/systemd/cloud-init.service.tmpl b/systemd/cloud-init.service.tmpl
index b92e8ab..5cb0037 100644
--- a/systemd/cloud-init.service.tmpl
+++ b/systemd/cloud-init.service.tmpl
@@ -14,8 +14,7 @@ After=networking.service
 After=network.service
 {% endif %}
 {% if variant in ["suse"] %}
-Requires=wicked.service
-After=wicked.service
+Before=wicked.service
 # setting hostname via hostnamectl depends on dbus, which otherwise
 # would not be guaranteed at this point.
 After=dbus.service

How cloud-init wants things to work is:

1) cloud-init-local.service runs before networking is up and then will use dhclient (dhcp-client) to temporarily bring up networking to crawl the IMDS to obtain the platform specific network config and write this out to the system

2) os networking starts/comes up

3) cloud-init.service waits for networking to be up runs right after the service, but before network-online.target

Cloud-init invokes dhclient directly in (1) and explicitly does not leave any state from running dhcp around in case that IMDS were to provide a conflicting network configuration; this also avoids starting OS networking services (like wicked) until cloud-init-local has written out the expected network config.

I'll let Robert respond here when he can, but I think we'd want to revert the commit above. The referenced bug number doesn't have any further details about what was or was not working without it.

I do not remember the details of the change in cloud-init.service. It could be a misunderstanding of the way things were/are supposed to be ordered, then again having "After=" for the other distros is pretty darn obvious, or it could be that a loop is created. Although the creation of a loop between "After=dbus.service" and "After=wicked.service" is hard to imagine, but it will have to be tested.

The creation and wording of lp#1799709 does suggest confusion :(

In any event this is not really related to the problem that we have no DNS information. WHich this bug is really about. cloud-init should not write a resolv.conf file that is "empty" i.e. only has the comments. That's the first problem that needs to be solved.

The patch that is in the 19.1 package that is currently distributed in openSUSE and SLE to avoid writing an "empty" config file is as follows:

--- cloudinit/net/sysconfig.py.orig
+++ cloudinit/net/sysconfig.py
@@ -545,6 +545,8 @@ class Renderer(renderer.Renderer):
             content.add_nameserver(nameserver)
         for searchdomain in network_state.dns_searchdomains:
             content.add_search_domain(searchdomain)
+ if not str(content):
+ return str(content)
         header = _make_header(';')
         content_str = str(content)
         if not content_str.startswith(header):
@@ -658,7 +660,8 @@ class Renderer(renderer.Renderer):
             dns_path = util.target_path(target, self.dns_path)
             resolv_content = self._render_dns(network_state,
                                               existing_dns_path=dns_path)
- util.write_file(dns_path, resolv_content, file_mode)
+ if resolv_content:
+ util.write_file(dns_path, resolv_content, file_mode)
         if self.networkmanager_conf_path:
             nm_conf_path = util.target_path(target,
                                             self.networkmanager_conf_path)

This is basically the same as in in "master". In "master" instead of returning an empty string, content is initialized to resolv_conf.ResolvConf(""), None is returned. This means in master "_render_dns()" has different return types, which is a style choice and that's a different discussion.

From a code perspective the package delivers the equivalent to what is in master. Next step for me is to do some testing in Azure.

Suitably modified cloudinit/net/sysconfig.py on an instance produces

2019-09-23 12:27:57,994 - sysconfig.py[INFO]: RJS no content for resolv

and renders no resolv.conf file after running 'systemctl start cloud-init-local.service'. Will need access to an instance that shows the behavior per the initial description.

And as a side note comment #13 was incorrect, w.r.t. writing resolv.conf there is no special patch, I was looking at an older package version. Meaning the 19.1 package distributed for openSUSE and SLE runs the upstream code and returns "None" when there is no dns content in the network config.

The issue is that the content string is reinitialized with the contents of the already existing "resolv.conf" file as following. But the file only contains commented line.

        if existing_dns_path and os.path.isfile(existing_dns_path):
            content = resolv_conf.ResolvConf(util.load_file(existing_dns_path))

So even though there is no name servers nor search domains added to the content, It's still not empty. And the check (not str(content)) will return false. The header will be added to the content and the file will be modified. So netconfig is not touching it.

        if not str(content):
            return None

summary: ensuring cloud-init.service runs after wicked and avoid updating resolv.conf without dns values in network_config will fix this issue.

I'm confident that if cloud-init.service runs after wicked.service and the proposed fixed attached to this bug will resolve this issue. I just wanted to make sure we weren't missing a scenario in reasoning behind wicked running before.

> or it could be that a loop is created. Although the creation of a loop between
> "After=dbus.service" and "After=wicked.service" is hard to imagine, but it will have
> to be tested.

That could be an issue; if so, you likely can replace dbus.service with dbus.socket; as this allows the services to initiate connections to dbus. This is usually sufficient for a dependent service to get going.

As Moustafa pointed out, we need a better patch (which I've proposed and linked to this bug above) which checks if the network config has any dns information before attempting to update resolv.conf; it appears that in some SLES images they contain a resolv.conf template which is used by 'netconfig' and the cloud-init comment added to the file causes netconfig to not update resolv.conf, meaning no DNS at all.

The attached patch LGTM, I have added it to our current package in OBS [1]. I am currently testing the ordering change.

[1] https://build.opensuse.org/package/show/Cloud:Tools/cloud-init

Patch tested and order change tested. Both work, found new independent issue in Azure data source.

AFAIK, the attached patch should be landed and we can close this bug.

Order change for wicked: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/373166

This bug is fixed with commit f80f7f0d to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=f80f7f0d

This bug is believed to be fixed in cloud-init in version 19.2-50. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3448