cloud-init selects sysconfig netconfig renderer if network-manager is installed on Ubuntu

Looks like it might be an issue either in curtin or MAAS based on the network configuration.

Once the machine fails to deploy can you provide the output of:

maas {profile} machine get-curtin-config {system_id}

FYI, I've added a cert task for this. I don't know for sure this is curtin, it looks like something may have changed in one of the hundreds of dependency packages that checkbox pulls in causing curtin to fail.

Rod is investigating it on our side.

We have a bug for this as well, 1189973 but duping for that kills the MAAS (possibly curtin) task. So I un-duped it for now

We've traced the problem to the network-manager package, which gets pulled in by a dependency in canonical-certification-server. Apparently, curtin or cloud-init (I'm not sure which) is now skipping netplan configuration when the network-manager package is installed.

Neither curtin, nor cloud-init will *skip* generating networking. However, if there exists some additional netplan config in the target system that cloud-init is not aware (maybe provided in the NetworkManager package (or something else)) then there may be a conflict in the configuration that prevents netplan apply from bringing up the network.

If possible, getting the systemd journal and what's in /etc/netplan and /run/systemd/{netif,network} and /var/log/cloud-init.log could help see what's going on.

I've attached the /var/log/cloud-init.log file from a node that failed deployment. (This is a different node from the one that generated the earlier logs.) The /etc/netplan directory is empty, and neither there is no /run/systemd directory on this node that failed to deploy.

2019-03-14 17:32:34,606 - __init__.py[DEBUG]: Selected renderer 'sysconfig' from priority list: None

This is a cloud-init bug. The sysconfig renderer has NetworkManager support, this triggered cloud-init to render sysconfig instead of netplan.

You can workaround this issue by including the following curtin config when deploying.

write_files:
  policy:
    path: /etc/cloud/cloud.cfg.d/01_network_renderer_policy.cfg
    content: |
      #cloud-config
      system_info:
        network:
          renderers: ['eni', 'netplan']

Thanks for the quick fix, Ryan! I've confirmed that your curtin config workaround in comment #15 works. Do you have an estimate for how long it'll be before a fix goes live? (I ask so we can plan whether we should push your workaround through one of the certification packages.)

On Fri, Mar 15, 2019 at 2:50 PM Rod Smith <email address hidden> wrote:

> Thanks for the quick fix, Ryan! I've confirmed that your curtin config
> workaround in comment #15 works. Do you have an estimate for how long
> it'll be before a fix goes live? (I ask so we can plan whether we should
> push your workaround through one of the certification packages.)
>

Depends on where you need it. It can likely land upstream either today
or on Monday; and would be available via the cloud-init-dev daily PPA;
however, an SRU will take at least another week after next; We're almost
done with an existing cloud-init SRU; so we'd likely not start another SRU
until the current one is in -updates.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1819994
>
> Title:
> cloud-init selects sysconfig netconfig renderer if network-manager is
> installed on Ubuntu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1819994/+subscriptions
>

hi Jeff and all,

After upgrade online, it is MAAS 0.4.0 show under version tale, but still 2.4.2 under the log. in the same time, the deploy fails again. please double check the log and let me know if there is any comments.

Best Regards,
Amy

Hi Amy,

first, which machine failed? I see a bunch of machines in the /var/log/maas/rsyslog/ directory, and I'm not sure exactly which one to look at.

Secondly, the version you posted in the screen shot looks correct, can you show me the output of:

ls -l /etc/maas/preseed/curtin_userdata*

Amy: Also, could you send me a tarball containing /etc/maas/preseeds ??

hi Jeff,

it is SR590 Cascadelake deploy failed with the new MAAS 0.4.0. the attahmen above is collected from The environment with SR590 Cascadelake.
Besides, the same issue also occurs on SR650 cascadelake.

best Regards,
Amy

Amy, I think you're confusing the MAAS version (which is 2.4.2 on one of our installations) and the maas-cert-server package version (the latest of which is 0.4.0). The maas-cert-server 0.3.9 package includes a workaround (but NOT A FIX) for this bug, and 0.4.0 provides some unrelated improvements, so the installation SHOULD succeed after you've upgraded maas-cert-server to version 0.3.9 or 0.4.0. If it's still failing, then it could be you'll need to apply the workaround described by Ryan Harper in comment #15, which is different from the workaround in maas-cert-server 0.3.9 and 0.4.0. (Post back if you need help applying Ryan's workaround.) It could also be that you're looking at a completely different problem.

hi Rod,

Thanks for your update, we will use the workaround to execute the current certification test on Purley Cascadelake.
As to the Deploy failure on MAAS 0.4.0, do you advise we raise the other defect to track?

Best Regards,
Amy

This bug is fixed with commit 5de83fc5 to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=5de83fc5

This bug is believed to be fixed in cloud-init in version 19.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Sorry for the later reply, the issue does not occur with current Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~18.04.1. please move on and close it. thanks a lot.

Best Regards,
Amy

Hi Amy, it's likely that you're still using our patched tooling that includes a workaround. cloud-init 18.5 should not work.

Just a heads up, the fix is now in -updates, I've tested this locally on a couple deployments and it seems to resolve the issue we had before. Asking my team to verify on a couple more deployments for due diligence.

I've tested this on three nodes on two MAAS servers (my own home MAAS server and maastiff, our MAAS server in the certification lab), using both 18.04 and 19.04. It looks good to me.

thanks for your kindly update, i will do the double check with the latest one.

Hi Amy et al,

I'm going to mark this Fix Released, as 19.1 has made its way in to Ubuntu. Please let us know if you don't think this is fixed!

Dan

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3354