transient systemd ordering issue when using overlayroot

==== Begin SRU Template ====
[Impact]
Overlayfs allows a user to mount the root filesystem read-only
and put an overlay over it.
It handles mounting of the root filesystem in the initramfs, and updates
/etc/fstab to account for non-root filesystems and generally make
/etc/fstab "correct".

systemd can get confused by a /etc/fstab that looks like this:

   LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
   /media/root-ro/ / overlay lowerdir=...

It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
This would lead to boot ordering cycles. The fix that was ultimately
applied was just to comment out the line for /media/root-ro.

That way systemd wouldnt try to ensure that /media/root-ro got mounted.

The fix that was put in place was just to comment out the '/' (/media/root-ro)
line in /etc/fstab. So essentially:
 - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
 + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0

Since that happens in the initramfs before systemd ever generates mount
options, it has the desired affect. And /media/root-ro is already mounted
before systemd starts.

[Test Case]
## setup system to build maas images
$ sudo apt-get install -qy --no-install-recommends bzr
$ bzr branch lp:maas-images
$ cd maas-images
$ ./system-setup

## download the reference squashfs images.
$ arch=amd64
$ burl=http://cloud-images.ubuntu.com/
# download reference squashfs images
$ for rel in xenial zesty artful; do
     file=../$rel-server-cloudimg-$arch.squashfs
     [ -f "$file" ] && continue
     wget "$burl/$rel/current/$file" -O "$file.tmp" && mv "$file.tmp" "$file";
  done

## build maas image output. This creates a squashfs image and kernel/initramfs
## files that have the updated packages inside.
## It does take a while to run, and lots of cpu and IO.
## You can reduce the kernels built by commenting out 'lowlatency' and 'edge'
## kernels:
## sed -i -e '/^[ ].*\(lowlatency\|edge\)/s/^/#/' conf/meph-v3.yaml
# either set 'proposed' or add the repo via M2E_ADD_REPOS
$ proposed="--proposed"
$ export M2E_ADD_REPOS=ppa:smoser/cloud-initramfs-tools
$ for rel in xenial zesty artful; do
  img=../$rel-server-cloudimg-$arch.squashfs
  PATH=$PWD/bin:$PATH ./bin/meph2-build $proposed --config=conf/meph-v3.yaml \
        $arch $rel $(date +"%Y%m%d") $img ../$rel.d/ 2>&1 |
        tee ../$rel.log
  done

## After running, you will have files like:
$ find xenial.d/xenial -type f | sort
xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd
xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel
xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-initrd
xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-kernel
xenial.d/xenial/amd64/20171221/squashfs
xenial.d/xenial/amd64/20171221/squashfs.manifest

## You can verify in the $rel.log files you have 'Setting up cloud-initramfs-'
$ grep '^Setting up overlayroot' xenial.log
Setting up overlayroot (0.27ubuntu1.5~ppa16.04.1) ...

## Now download the test-boot script, provide it a directory and
## the squashfs, kernel, initramfs

$ ./test-boot test-x xenial.d/xenial/amd64/20171221/squashfs \
    xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel \
    xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd

$ r=zesty; ./test-boot work.d $r.d/*/amd64/*/squashfs $r.d/*/amd64/*/ga-*/generic/boot-kernel $r.d/*/amd64/*/ga-*/generic/boot-initrd
Now, do:
  cd work.d
  ./boot

## Running ./boot will serve the squashfs image over http, and then
## boot the kernel and initrd with a kernel command line of
## root=http://<url>/squashfs
## and
## ds=nocloud-net;seedfrom=http://10.0.2.2:9999/
## which will the read the 'user-data' and 'meta-data' files in that dir.
##
## The result is that you can log in with 'ubuntu' and 'passw0rd'

## Then collect some information
# you'll see the 'LABEL=' entry is commented out.
$ cat /etc/fstab
# local-fs.target should be fine
$ systemctl status local-fs.target --no-pager
# just for record
$ cat /proc/cmdline

[Regression Potential]
Not to omany things would depend on the /etc/fstab that exists in
an system booted with overlayroot. Something that was reading /etc/fstab
could be confused by this change.

[Other Info]
This change is believed to fix bug 1680197 as well, but due to the
sporadic nature, we are not certain.

The fix applied here is a backport what is currently in bionic to
each release. in xenial, that means a new binary package
(cloud-initramfs-updateroot).

==== End SRU Template ====

I had reported this once before in bug 1666573.
I thought it was fixed there, but apparently not.

The failure shows itself as (controls removed):
  [ 80.089654] systemd[1]: media-root\x2dro.mount: Found ordering cycle on media-root\x2dro.mount/start
  [ 80.099474] systemd[1]: media-root\x2dro.mount: Found dependency on -.mount/start
  [ 80.108348] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
  [ 80.117231] systemd[1]: Unable to break cycle
  [ 80.132286] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided
  [ 80.180707] systemd[1]: Unnecessary job for VIRTUAL-DISK cloudimg-rootfs was removed.
...
 [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
 [DEPEND] Dependency failed for /boot/efi.
 [DEPEND] Dependency failed for Local File Systems.

The /etc/fstab as read from / once we've left the initramfs
looks like:
  #
  # This fstab is in an overlay. The real one can be found at
  # /media/root-ro/etc/fstab
  # The original entry for '/' and other mounts have been updated to be placed
  # under /media/root-ro.
  # To permanently modify this (or any other file), you should change-root into
  # a writable view of the underlying filesystem using:
  # sudo overlayroot-chroot
  #
  LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
  LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported

Notes:
 * I do not believe this is strictly tied to open-iscsi, but the
   test that we see it failing in is the open-iscsi test.

Related bugs:
 * bug 1666573: transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root
 * bug 1680197: Zesty deployments failing sporadically
 * bug 1723183: [systemd] transient systemd ordering cycle in boot with overlayroot