ubuntu-fan autopkgtests are broken against systemd-resolved

Attached is the partial patch, which should also provide insight into how to fix the docker test.

Ok, I get the drift, but frankly when systemd-resolvd/systemd-networkd change the startup timing that drastically I guess fan tests failing might be the least of our problems...

I could not see the docker failure when running a local ADT test. So I added the checking with some additional output to see what is going on. The latest test failure (at least now it is detected quickly) http://autopkgtest.ubuntu.com/packages/u/ubuntu-fan/artful/amd64 shows that the host side is ok (using systemd-resolve --status on the first default route found returns a sensible forwarder). Also the fact that the image can be downloaded hints that the host side is good. And the LXD test is using the same setup (also the same underlay of 10.220.0.0/16) and works.

The difference is that LXD is using the dnsmasq based resolver which the fan tools set up, while docker is using its own (127.0.0.11) service. Not sure what the exact environment setup in ADT testing is, but could docker be affected by the same multiple default route issue which we had with the LXD tests? I don't think fanatic can do much as dns does not seem to be something one can influence with network create.

This bug was fixed in the package ubuntu-fan - 0.12.6

---------------
ubuntu-fan (0.12.6) artful; urgency=medium

  * fanatic: Add short success delay to nc_send (LP: #1721352)
  * fanatic: Catch test preparation steps failing (LP: #1718548)
  * fanatic: Add DNS checks to local-test preparation (LP: #1718548)

ubuntu-fan (0.12.5) artful; urgency=medium

  * DEP8: Fix LXD default interface detection (LP: #1718548)
  * fanctl: return error on fail_up (LP: #1719644)

 -- Stefan Bader <email address hidden> Fri, 06 Oct 2017 12:15:38 +0200

Hello Steve, or anyone else affected,

Accepted ubuntu-fan into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-fan/0.12.7~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Steve, or anyone else affected,

Accepted ubuntu-fan into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-fan/0.12.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I don't understand why this was SRUed to xenial, where we do not use systemd-networkd / systemd-resolved; and there is no SRU test case listed in the bug. Łukasz, why was this accepted and what do you expect for testing?

The britney hint was bumped for ubuntu-fan to override the failing autopkgtests. Which means this bug was automatically marked as closed by the migration of a version of the package that still had failing autopkgtests. Reopening.

@Steve, we try to get the code base of Fan synced across releases (one part of the reasons is that docker and lxd may backport things at any time, the other part is to avoid having to maintain subtly different code bases). The changes I attributed to this bug are an additional exit in an awk script which is verified by the tests still working. The other part is adding tests and print statements for DNS configuration (host and container side) which can be verified by looking at the test logs.

As for the docker tests failing, we probably should move that to its own bug. Right now this only happens in the CI environment and I cannot see much that the Fan testing could do:
- the hosting VM (artful/bionic) is using netplan and from print statements
  has systemd-resolv ready before launching the container
- the container (docker) uses its own DNS resolver, we pull latest lts images so
  those are xenial based atm.
- the image is pulled by dockerd, so that has at least DNS resolution for the
  proxy
- however inside the container (and unfortunately that is rather limited in the
  supplied commands) DNS lookups via the docker resolver fail.

I can pull the current cloud-images used by adt locally and get all testing passed without the slightest issue.

On Wed, Nov 15, 2017 at 09:32:18AM -0000, Stefan Bader wrote:
> @Steve, we try to get the code base of Fan synced across releases (one
> part of the reasons is that docker and lxd may backport things at any
> time, the other part is to avoid having to maintain subtly different
> code bases). The changes I attributed to this bug are an additional exit
> in an awk script which is verified by the tests still working. The other
> part is adding tests and print statements for DNS configuration (host
> and container side) which can be verified by looking at the test logs.

> As for the docker tests failing, we probably should move that to its own bug. Right now this only happens in the CI environment and I cannot see much that the Fan testing could do:

The bottom line here is that an SRU with failing autopkgtests is not
releasable. The CI infrastructure is the *primary target* for autopkgtests.
If you are not going to support these autopkgtests in the CI infrastructure,
they should be disabled, not left failing.

From my perspective the failing docker.io tests aren't a different bug, they
started failing at the same time according to
<http://autopkgtest.ubuntu.com/packages/u/ubuntu-fan/artful/amd64>.

Event though both test failures where triggered by the change from ifupdown to systemd-networkd/resolvd the reasons for the problems are completely different. For lxd the problem is in the dep8 test script. For docker it is turns out to be a problem of docker itself to automatically guess which nameserver to use.

For this reason I created three individual bug reports:
- bug #1732739 -> lxd breakage
- bug #1732747 -> additional DNS checks and logging (pre-req
                  for docker fix)
- bug #1732717 -> docker breakage

I will duplicate this bug against the first new one because the bug description was talking mostly about this one. And then use the new bug reports for driving the delivery of fixes.