casper fails to add any users because GID 999 is already taken

Lubuntu lunar ISOs booted on

- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
an old BIOS only box

- sony vaio ultrabook svp11216cgb (i5-9400u, 4gb, intel haswell-ULT)
more modern uEFI box

and the LIVE session is unusable.

This issue was noted first by KGIII & reported on #lubuntu-devel, today is not the first ISO where this issue was noted.

<kgiii> I was trying to do a quick test and bumped into this. If it's still an issue on Monday, I'll go ahead and try on real hardware. I want to eliminate me as the problem, but it's multiple versions of VirtualBox on two separate bits of hardware. So, that mostly eliminates me - mostly...

<kgiii> Effected builds 20230127 and 20230128. And, I figured this one was important enough to report to chat. I couldn't find a way by the black screen.

Status changed to 'Confirmed' because the bug affects multiple users.

Status changed to 'Confirmed' because the bug affects multiple users.

Affects me too. (I tested Lubuntu in Dell Precision M4800.)

I edited the bug report to reflect systemd-journal's use of 999, and am self-assigning.

Subscribing the sponsor and author for the systemd merge. I'm guessing this isn't something that could have been easily prevented unless you were looking for it, that being said, Foundations should be aware of this change, in case there are side-effects.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2004092

Confirmed here.

Of course here is where it began - but I'm posting to ensure I'm subscribed to additional comments.

I made a persistent live Lubuntu system with mkusb:dus-iso2usb in a USB-SSD from the daily iso dated 2023-01-25 and made it up to date by

sudo apt update && sudo apt full-upgrade

This system works as it should (persistent live) after shut down and boot.

When there is a [correctly configured] system, the current set of program packages works, which matches the conclusion by Simon, that configurations during [the first] boot fail in the current daily iso file (casper fails to add any users).

In this persistent live system the UID of lubuntu is still 999 and the UID of systemd-journal is 101.

> systemd-journald now uses GID 999

Ummm where is this coming from? Debian Policy states:

9.2.2. UID and GID classes
--------------------------
[...]

100-999:
   Dynamically allocated system users and groups. Packages which need
   a user or group, but can have this user or group allocated
   dynamically and differently on each system, should use "adduser
   --system" to create the group and/or user. "adduser" will check for
   the existence of the user or group, and if necessary choose an
   unused id based on the ranges specified in "adduser.conf".

I see no addgroup calls in the systemd maintainer scripts. And in a lunar container created before systemd 252 was uploaded, I see systemd-journald created as group 101. This appears to be happening via /usr/lib/sysusers.d/systemd-journal.conf.

I think it is a bug in systemd to be bypassing the process specified in Debian Policy.

> Rather than going through the painstaking task of updating systemd
> to use a *different* user, let's just update casper to use 1000,
> which is the first dynamically-allocated user account per Debian
> Policy[1].

This is the GID expected to be used for the first user created on the target system. It's not clear to me that changing the casper user to use this uid won't badly confuse the installer when it comes time to create the actual user account, or at least cause gid 1000 to be skipped.

@vorlon I just attempted an installation of Ubuntu Desktop 23.04 using the latest (broken) ISO. I am able to get to the installer since a setup wizard has me create a user account to get into the live ISO. I saw multiple "System program problem detected" warnings appear during the installation, however the installation appears to have succeeded, and the UID of the user created during the installation is 1000. Firefox launches without issue on the installed system.

I don't believe this proves that having Casper make a user with UID 1000 is entirely safe, but it at least appears to work on the surface.

This bug was fixed in the package casper - 1.479

---------------
casper (1.479) lunar; urgency=medium

  * Fix adduser, which fails due to systemd taking 999 (LP: #2004092).

 -- Simon Quigley <email address hidden> Sat, 28 Jan 2023 21:09:53 -0600

> It's not clear to me that changing the casper user to use this uid won't badly confuse the installer when it comes time to create the actual user account, or at least cause gid 1000 to be skipped.

It's not clear to me that it will. Casper doesn't touch the squashfs that ends up on the installed system, it simply creates a user in the overlay filesystem. If this does end up changing the installer behavior, that's an installer bug; it should only care about what is actually in the squashfs.

(If there's another use case I'm failing to consider, let me know.)

I cloned Lubuntu Lunar from today's (2023-01-29) daily iso file to a USB-SSD and booted it in a Dell Precision M4800.

It seems to me that it works correctly to run a live session with the userID number 1000 for 'lubuntu' :-)

I made a persistent live Lubuntu system with mkusb:dus-iso2usb in a USB-SSD from today's daily iso (2023-01-29).

This system works as it should (persistent live) also after shut down and boot (in a Dell Precision M4800).

-o-

lubuntu@lubuntu:~$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.2G 1.8M 3.2G 1% /run
/dev/loop0 2.8G 2.8G 0 100% /cdrom
/cow 50G 60M 47G 1% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 16K 5.0M 1% /run/lock
tmpfs 16G 0 16G 0% /tmp
tmpfs 3.2G 76K 3.2G 1% /run/user/1000
/dev/sdb5 50G 60M 47G 1% /media/lubuntu/writable
/dev/sdb3 4.9G 2.8G 1.8G 61% /media/lubuntu/isodevice

lubuntu@lubuntu:~$ grep -e '^lubuntu' -e '^systemd' /etc/group
systemd-journal:x:999:
systemd-network:x:998:
systemd-timesync:x:997:
systemd-resolve:x:996:
lubuntu:x:1000:
lubuntu@lubuntu:~$

> It's not clear to me that changing the casper user to use this uid won't badly confuse the installer when it comes time to create the actual user account, or at least cause gid 1000 to be skipped.

After some testing with the newly-uploaded casper, I can confirm this is not the case. I tested with the following setups:
 - Ubuntu Budgie, with Ubiquity
 - Lubuntu, with Calamares
 - Ubuntu, with the ubuntu-desktop-installer snap

The live user has UID/GID 1000 and so does the installed user, in all three cases.

I would test OEM, except it doesn't make sense. There is already an initial OEM user created, which will have 1000 either way. That's expected behavior.

On the casper side, I think everything is working as intended.

I should also mention, that I have created a second user 'tester' in previous versions of persistent live Lubuntu for example to test bug #2000787 "Live system cannot leave 'Lock Screen' when selected from menu".

This user had userID number 1000, and it never caused any problem. If I remember correctly, I also ran the installer in such a system, and the userID number 1000 did not cause any problem.

> I think it is a bug in systemd to be bypassing the process
> specified in Debian Policy.

Note that systemd specifies its own UID/GID "spec" here: https://systemd.io/UIDS-GIDS/.

I think assigning 999 to systemd-journald conforms with both systemd's and Debian's policy, no? The systemd-journal group is a system group, and the GID is dynamically allocated (by systemd-sysusers, not adduser). See [1][2] for systemd-sysusers documentation.

It seems that fixing casper is correct, because statically/explicitly allocating a UID/GID in that range *is* in violation of both Debian's and systemd's policies.

[1] https://www.freedesktop.org/software/systemd/man/systemd-sysusers.html
[2] https://www.freedesktop.org/software/systemd/man/sysusers.d.html

Just to catch the bug report up with the IRC discussion:

<vorlon> we got away with it in casper because we knew there weren't going to be 899 IDs allocated in this range in the installer context, and the casper user goes away post-install
<vorlon> systemd doesn't statically set them, it allocates them using "first available" ID at boot time; however, it seems to start checking "first available" from the other direction
<vorlon> from a policy perspective, I think the biggest bug is: policy says to use adduser; adduser has a config file that lets the admin change the allowed range (to subset it, perhaps, for compatibility with "legacy" IDs shared across an authentication domain); systemd doesn't honor adduser.conf but instead has its limit hard-coded at build time
<enr0n> vorlon: I'll look into the adduser.conf piece specifically. But note that we as a distro can configure systemd's boundaries at build time
<vorlon> enr0n: yes, but the disconnect is with it not honoring changes specified by the admin at runtime

I noticed that the live Lunar Ubuntu Desktop (standard Ubuntu) arrives at a page that prompts to enter user name and password, in other words, there is not longer a standard user with a blank password.

This should be considered when we decide how to deal with assigning a user ID for the live session of Lubuntu.

Tagged incoming as while there is a modified casper uploaded, the matching merge proposal was disapproved (https://code.launchpad.net/~tsimonq2/casper/+git/casper/+merge/436490), so the next casper upload will regress the current version of the fix. A longer term solution is needed.

> the next casper upload will regress the current version of the fix

I'm all in favor of a better longer-term fix replacing this one being tracked in systemd, but an unrelated casper upload regressing this change prior to a longer-term solution being in place will cause issues with all amd64 live ISOs.

As noted above, this change does not present Ubiquity, ubuntu-desktop-installer, or Calamares from assigning a UID and GID of 1000 to the installed user, which is the same as it was before.

What is the rationale for reverting this change?

>> the next casper upload will regress the current version of the fix
> What is the rationale for reverting this change?

As of yet I think that has't happened. Right now we have 2 caspers
* git casper at https://git.launchpad.net/casper, version 1.478, without this proposed fix since the MP was rejected
* archive casper, version 1.480, which per https://launchpadlibrarian.net/649860924/casper_1.479_1.480.diff.gz does seem to still have this change

I'm only trying to point out that the next time someone uploads directly from what is in "git casper", it will drop this change, so we need a long-term fix.