snap-preseed hangs indefinitely with firefox snap

Here's a full stack trace from the hanging command: https://paste.ubuntu.com/p/TbNmDBGHSr/

Removing `--snap firefox` from the `sudo snap prepare` command prevents `snap-preseed` from hanging.

Here's some more clues maybe. While the preseed is hanging you "chroot chroot snap changes" outputs

ID Status Spawn Ready Summary
1 Doing today at 22:25 UTC - Initialize system state

and "chroot chroot snap tasks 1" outputs https://paste.ubuntu.com/p/ZpyMPPsv3R/. The stuck task appears to be:

Doing today at 22:24 UTC - Automatically connect eligible plugs and slots of snap "gtk-common-themes"

and I presume the output at the bottom of

2022-09-26T22:28:08Z INFO Waiting for conflicting change in progress: conflicting plug snap firefox, task "connect" every 5 seconds is relevant?

Here's the state.json from the preseed attempt: https://paste.ubuntu.com/p/r8nFv7sknd/ which is almost certainly relevant. On a trivial look-see there is no cycle in the task

Oh running with SNAPD_DEBUG=1 gives some more output: https://paste.ubuntu.com/p/7zq5Nqq9qd/ (not really telling us anything new though).

I'll look into this. It could be a case with how we deal with slots/plugs between dependencies - but I don't know anything for sure yet.

This bug was fixed in the package livecd-rootfs - 2.785

---------------
livecd-rootfs (2.785) kinetic; urgency=medium

  * Disable the snap-preseed calls in the interest of getting images built for
    the 22.10 beta. (LP: #1990884)

 -- Brian Murray <email address hidden> Mon, 26 Sep 2022 15:55:26 -0700

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/1990884

Fix will be released with snapd 2.57.4

> Disable the snap-preseed calls in the interest of getting images built for the 22.10 beta. (LP: #1990884)

This is great fix for ISOs but what about cloud-images? As part of our testing for all cloud-images, CPC makes sure that the snap-preseed was run to (significantly) reduce the first boot time of Ubuntu images on the clouds. This test is now failing across all the clouds.
I believe we should reduce this fix to only the "ubuntu" project as "ubuntu-server" and "ubuntu-cpc" are not likely to be affected by the original issue.

WSL rootfs images built since Sep 27th lack the hardlinks inside `/var/lib/snapd/snaps/` that should point to the files existing in `/var/lib/snapd/seed/snaps/`. Could that be a consequence of disabling snapd-preseed calls?

On Wed, 5 Oct 2022 at 06:15, Carlos Nihelton <email address hidden>
wrote:

> WSL rootfs images built since Sep 27th lack the hardlinks inside
> `/var/lib/snapd/snaps/` that should point to the files existing in
> `/var/lib/snapd/seed/snaps/`. Could that be a consequence of disabling
> snapd-preseed calls?
>

Yes.

small poke -- will this be rectified soon, to ensure Kinetic releases are pre-seeded?

If not, we'll need to put a "skip" in the livecd-rootfs code to ensure CPC projects are being preseeded (we weren't hit by the specific bug for firefox,and thus we are having downstream issues with all our tests failing due to pre-seed not happening and boot times being too slow)

Hi John,

We committed a fix for this with snapd 2.57.4 (if you can use the edge channel). But for the release schedule for when 2.57 is officially available you'll have to hear Michael Vogt (mvo) about. I believe it should be early next week, but I am not quite sure.

Hi John and Gauthier!

Did snapd 2.57.4 work out for CPC? I still see WSL kinetic images not being preseeded.

CPC isn't driving this. We can't use edge channels on the launchpad builders for pre-seeding, so we have to wait for a release. And CPC builds weren't originally affected.

if whomever was originally affected can create a build env with snapd from edge for testing, they can verify the change. Otherwise we must wait for the release this week. I see it moved to candidate today.

This is only marked fix committed, but 2.57.4 with the necessary fixes is in kinetic and images are building with the livecd-rootfs workaround disabled.