subiquity service fails to start

It Works For Me(tm)

mwhudson@aeglos:~/src$ sha1sum artful-live-amd64.iso
00c003ea19d775589a0b29ff8f2452d177f535d2 artful-live-amd64.iso

Are there incriminating messages in syslog/the journal?

Also, how are you starting your VM? I'm starting it with kvm -m 1024 -boot d -cdrom artful-live-amd64.iso -hda ~/src/target.img where target.img is a raw disk image.

journalctl:
Oct 05 06:15:43 ubuntu-server systemd[1284]: subiquity.service: Failed at step EXEC spawning /snap/bin/subiquity.subiquity-loadkeys: No such file or directory

So this looks to be some manner of race condition. The file is of course there when I finally get a shell.

I see the job declares 'Requires=snapd.service', but doesn't say anything about /ordering/ (i.e. there is no snapd unit listed in 'After').

There is a systemd unit for snap-subiquity-98.mount; unfortunately there is no unversioned unit nor an unversioned alias, so we can't depend on that.

This VM is set up using virt-manager, using some semblance of 'defaults' from zesty. Attached is the libvirt xml for this VM.

I do see snapd using 100% CPU at the time I log in; so that probably contributes to the race. I don't know what it's doing with the CPU - key generation?

Oh gooooooooooooood....

I can think of a (terrible? I can't decide) workaround for this: have subiquity declare a one-shot do-nothing service, subiquity.started or something, which snapd will arrange to order after the relevant mount job, and order the main subiquity.service After= this subiquity.started.service.

https://code.launchpad.net/~mwhudson/livecd-rootfs/subiquity-ordering/+merge/331844 and https://github.com/CanonicalLtd/subiquity/pull/252 implement this workaround and seem to work for me (of course I don't experience the bug either, but at least they don't break anything)

Thanks, that workaround looks good and I've acked both MPs.

The other question, of course, is why is snapd spinning forever in my VM if it isn't in yours?

This is fixed (or at the very least should be) in today's daily.

This is not in fact fixed. The current guess at the problem is that although subiquity.service is ordered after snap.subiquity.started.service, the latter is not known to systemd early enough and so it starts subiquity.serivce anyway.

My next idea for a hack is to put "ExecStartPre=/usr/bin/snap version" or something like that in subquity.service on the theory that thanks to the magic of socket activation this will block until snapd is actually serving api requests, and that it doesn't serve api requests until seeding is done and the mounts are set up. I haven't actually checked this last bit though.

Perhaps we should engage the snapd team (i.e. post in the forum) about this. It's pretty thorny.

https://code.launchpad.net/~mwhudson/livecd-rootfs/subiquity-snapd-created-service/+merge/337405 and https://github.com/CanonicalLtd/subiquity/pull/280 together hopefully fix this.

I think this is really actually fixed now.