[FFE] Move to logind for session tracking

Note that this also blocks on the MIR bug 1152187, but I would like to discuss this FFE in parallel.

I've been trying to understand what might be required to do this. I don't want to make any decision on my own, so here's my probably incomplete outline. Most of the information was gleaned from reading the bugs/BP and discussing a bit with pitti. Errors are no doubt mine.

  - Update systemd to 195
    + Brings with it a udev SONAME bump

  - libudev1 transition. This can be done as a "soft" transition - the ABI hardly changed between the two (the on-disk format also didn't change) and so it's safe to have libudev0 (src:udev) and libudev1 (src:systemd) around simultaneously. There might be tweaks required to adjust for e.g. removed symbols (udev_get_{dev,sys,run}_path for example are gone and should be hardcoded).

  - Seed libpam-systemd and unseed libpam-xdg-support

  - Port packages as we can to using logind instead of consolekit. Both can coexist so this is another soft transition.

  - An outstanding question is what to do with the cgroups. Some software uses [ -e /sys/fs/cgroup/system/ ] or similar to mean either "Is systemd init?" or "Can I use logind?".

Why was this not landed before feature freeze? One attribute of the development schedule this cycle is that we move feature freeze later in the cycle so that people could get work done before hand with the understanding that we would not generally approve things that didn't make it.

Has the implications of this for non-Unity desktops been explored at all?

The task list for consolekit is rather prodigious:

Task: ubuntu-desktop, ubuntu-usb, print-server, cloud-image, server, kubuntu-desktop, kubuntu-active-desktop, kubuntu-active, edubuntu-desktop, edubuntu-usb, xubuntu-desktop, mythbuntu-frontend, mythbuntu-frontend, mythbuntu-desktop, mythbuntu-backend-slave, mythbuntu-backend-slave, mythbuntu-backend-master, mythbuntu-backend-master, lubuntu-core, ubuntustudio-desktop

Also "the last UDS' is not the UDS at which raring was planned, so I think it's rather irrelevant to the discussion.

libsolid4 (the KDE hardware interface layer) in KDE 4.10 (the Kubuntu KDE
version for raring) does not support logind, but there are patches available
from upstream which upstream has said they will help us integrate. It would
also affect the lightdm KDE greeter. That work is not done yet.

FWIW, if the release time says this is better kept for post-raring, I'm okay with that. I just got asked from two different directions to try and ask for a FFE here.

BTW, the intention is to not break any existing software with this. If any software suddenly stops working because logind is running, that would certainly be a reason to nack this, as it would then mean that our analysis so far is flawed.

  - Update systemd to 195
    + Brings with it a udev SONAME bump

For the record, this is optional. The only applicable CVE of systemd (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-1174) is fixed in that version, and this doesn't yet ship a newer udev/libudev. However, I figure the libudev soname bump is the least risky part here.

While this doesn't have to be done in lockstep, I think ending up shipping both of consolekit and logind on a single image in 13.04 is a bad outcome; we're trying to drive our footprint down, this would have the opposite effect. If we're not confident to have the transition done before 13.04 comes out, I don't think we should start transitioning anything to depend on logind until after the release.

> - Update systemd to 195
> + Brings with it a udev SONAME bump

> For the record, this is optional.

Yes, and I don't see any reason to couple this to the transition. The version of systemd in Debian is 44 - there may be packages of 195 somewhere, but they're not in the archive or (last I knew) the git repo. I don't think we want to take this on right now - upstream udev behavior changes have recently been a source of regressions that took quite a while to fix, I don't think we should risk that right now.

It seemed to me like the discussion on IRC today went back and forth about if
the switch to logind and the udev transition needed to be tied together and if
not, which was more important to do first. Given that this we just thought up
in mid-cycle, rather than planned, I'm starting to think that waiting makes a
lot more sense.

On Mon, Mar 11, 2013 at 09:55:48PM -0000, Scott Kitterman wrote:
> It seemed to me like the discussion on IRC today went back and forth about
> if the switch to logind and the udev transition needed to be tied together
> and if not, which was more important to do first.

This was mostly a matter of me playing catch-up with pitti, who had a much
better understanding of the interdependencies than I did and has been
aggressively pursuing this in a ppa over the past few days.

The plan we came up with was the following:

1) update systemd to a more recent upstream version (19x), as this is a
prerequisite for obsoleting ConsoleKit from the seeds. (requires an FFe for
the new upstream version)
2) rebuild the archive against libudev1, now built from systemd.
3) in parallel, fix the udev package to not build libudev0, and to drop the
hard-coded udev->libudev0 dep.
4) seed logind in place of consolekit+libpam-xdg-support, and transition
reverse-dependencies to logind. (Requires a second FFe - and we may stop
here for 13.04, depending on available time.)
5) post-13.04, revisit the question of getting a newer udev with libkmod
support.

> Given that this we just thought up in mid-cycle, rather than planned,

For the record, this was blueprinted a while ago (back at UDS-Q); the latest
session concerned the finer details of how to migrate to logind now that
it's available, the fact that we've had a further detailed discussion should
not be taken as an indication that this is a new thought.

Okay. What happens if we do step one and stop? Is it reasonable to approve that and see how that goes before deciding on the udev stuff?

On Mon, Mar 11, 2013 at 11:46:27PM -0000, Scott Kitterman wrote:
> Okay. What happens if we do step one and stop? Is it reasonable to
> approve that and see how that goes before deciding on the udev stuff?

That doesn't seem a very sensible stopping point to me. The interface
surface of the systemd package as it's currently used in the archive is so
small (a pair of trivial dbus services) that there's no real risk of
hard-to-wrangle bugs here; and I don't think it's particularly useful to
update systemd to 19x right now if we're *not* also going to update libudev
et al.

OK. As long as someone is on the hook to fix regressions from the libudev1 migration, that makes sense.

Some comments:

 * The libudev0 → libudev1 migration is the least risky here. I can prepare it today in the PPA for the handful of source packages which need (trivial) adaptions for the dropped symbols, the other 20 or so reverse dependencies are mere rebuilds.

 * In theory we can also update systemd to 198 without switching to logind, i. e. just grant the first FFE. That would be really risk-free, as systemd itself has only been in raring for a couple of days, and nothing uses it yet. In that case I'd disable all udev library bits from the source package. It still makes some sense as one would then have current libraries available for experimentation, for the Ubuntu GNOME PPA, etc., but of course for Ubuntu itself there would be no recognizable change.

 * I can commit to dealing with fallout from the logind migration, I have time for this. This part is really the most risky one as it involves the most work, and may have regressions which are hard to spot at first sight (e. g. yesterday I noticed my camera wasn't working in Google+, I'll track that down today).

 * "ScottK> What happens if we do step one and stop?" As I said, we can run CK and logind in parallel, so the impact would be to ship the two packages (both are rather small), and having the additional (small) runtime impact of two D-BUS services.

Someone added a reference to http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-1174 . This has been fixed upstream in http://cgit.freedesktop.org/systemd/systemd/commit/?id=5ebff53 and backported to Debian in 44-1 in April 3. Thus this is fixed in raring as well.

libudev1 migration: I searched for all packages which use symbols that got dropped in libudev1. Detailled notes are in the spec, but summary is that only one package (lvm2) needs a sourceful upload. I cherry-picked the upstream fix for that into the PPA. For udev we only need to drop the libudev0/-dev binary package and drop udev's unnecessary dependency to libudev0.

OK. Since no one else jumped in ....

FFe approved for systemd update and the Udev transition. Then let's see where we are. If you end up having to upload anything that's on an ISO for a flavor participating in Beta 1, please double check that it's in stgraber's transition block so we don't get it in the release pocket before we're done with Beta 1 (his block is in part based on work I did that I know is incomplete).

> FFe approved for systemd update and the Udev transition.

Thanks. I uploaded 198 to raring-proposed now, and will upload the 39 reverse libudev0 depends for the transition. Iain blocked both systemd and udev from migration, which should hold back all reverse dependencies of libudev1 as well. So we can stage this up in -proposed until after beta1.

Resetting the status for the main FFE issue of moving to logind.

libudev1 migration is complete in raring-proposed. The blocks of udev and systemd will be kept until after beta-1 release, though.

Any thoughts on whether we should go logind for raring or not? I wonder how much energy I should put into maintaining the logind PPA, and playing catch-up with normal raring uploads. If we postpone that to raring+1, then this is rather pointless.

Also, if we don't do logind, but want the D-BUS services, we need to either drop the logind upstart job or split out logind again (I merged that into systemd-services as Debian didn't like the over-splitting), as the FFE for timezoned and friends is approved already.

Could you recap where we are, what the next steps are and what's been done to
mitigate any associated risks with moving forward?

Where we are:
 - current systemd v198 is in raring, with broken out systemd-services package which provides logind, hostnamed, etc.
 - libudev1 transition is mostly complete and in raring; the only remaining reverse dep to libudev0 is chromium on armhf as it currently FTBFSes there. (This problem is now independent of this FFE)
 - The package split is quite intrusive, but has been discussed with Debian and they want to adopt this. Debian also wants to migrate logind without enforcing systemd everywhere just yet (as the latter is politically much harder)
 - I fixed udev to work with logind's dynamic ACLs by backporting the "uaccess" builtin. udev rules are set up to use udev-acl with ConsoleKit and uaccess with logind. This is in raring.
 - Running logind separately causes some API confusion about sd_booted() for programs, as they now need to differentiate between "systemd init" and "logind". This has been discussed with upstream, and a fixed sd_booted() is in systemd upstream. We have a recipe how to fix packages that call this, and we know about all affected packages (12) from an archive grep. Details are in the blueprint.
 - I and other people (I know of Iain Lane and Stephane Graber) have run logind for quite a while now. I have run it exclusively without ConsoleKit for 1.5 weeks now, and did not get any crash of logind or the PAM module
- Running with logind exclusively uncovered some race condition in Unity's indicator/shutdown (bug 1155021)
- Consumers of CK have been identified with an archive grep (undeclared dependencies), and for many of them I added an analysis what needs to happen (details in blueprint)
- Login managers (lightdm etc.) generally are not supposed to directly talk to lightdm for creating sessions. That's supposed to be done by libpam-systemd and works fine. Of course they need to be adjusted for the shutdown/reboot API if they currently talk to ConsoleKit.
- I fixed the systemd upstream tests to run during package build, and wrote autopkgtests for all D-BUS services and logind.

Next steps regardless of this FFE:
- Fix upstream software that calls sd_booted() but wants to talk to logind to call access("/run/systemd/seats/", F_OK) instead.
- Add logind support to upstream lightdm (Robert Ancell has a WI) and Unity (I'll coordinate with Didier, or do it myself)

Next steps if FFE is rejected:
- If bug 1153633 (systemd-shim MIR) is accepted and we want to go through with bug 1153567 for raring, we need to either disable logind from systemd-services, or split it out again.

Next steps if FFE is accepted:
- Adjust seeds to move from libpam-xdg-support to libpam-systemd, which will pull in systemd-services. (It conflicts/replaces/provides libpam-xdg-support)
- Upload https://launchpad.net/~ubuntu-core-dev/+archive/logind and https://launchpad.net/~bkerensa/+archive/logind PPAs which have packages transitioned to talk to logind instead of CK. These are the most important ones.
- For the other reverse CK dependencies, either enable logind support for them (if present), or port it ourselves in easy cases (e. g. the shutdown/reboot API is exactly the same structure, just different D-BUS names), or add an explicit consolekit de...

I'll be almost entirely offline the next three days. I'd appreciate it if another release team member would follow up on this. No matter which option we pick, we should do it soon.

I think the amount of remaining churn we're looking at here speaks against an FFe for switching to logind in 13.04. Since this feature is not critical path for anything else, I'm going to nack here.

I don't know if the desktop team is still planning to switch from ubuntu-system-service to systemd-services in 13.04. My understanding is that the systemd packages are currently in a state that would allow them to do this - though the merger of logind back into systemd-services means that we'll have logind installed and running on the CD whether or not we follow through on using it (with PAM stack, etc. changes). Martin, is there a list of upstream software affected by the wrong sd_booted() check?

(Switch to systemd-services - ok, I see this is covered by bug #1153567 and bug #1153633.)

If the list of upstream packages needing fixed for sd_booted() is substantial, I think we should split logind back out. The package split here should be driven by Ubuntu's needs, not by the Debian systemd maintainers' preferences.

BTW, expanding on the "too much churn" comment, I see bug #1155021 in Unity is still marked "new", and this (and similar bugs) would be a prerequisite for the switch.

> Martin, is there a list of upstream software affected by the wrong sd_booted() check?

Yes, it's linked from the spec: http://people.canonical.com/~pitti/tmp/cgroup-sd_booted.txt

OK, so I'll go forward with disabling logind from the raring packages, and keep it enabled in the PPA only. I'll also have a go at Unity today (that's mostly monkey-patching), and filing upstream bugs for sd_booted(), so that after raring's release there are more things in place for the switch.

> we need to either disable logind from systemd-services, or split it out again.

I thought about this a bit more: Given that we still want to do the transition in a month, and not do intrusive packaging and conffile changes, I'd rather just keep logind in systemd-services, but make it inert until we enable it by default. This is now being tracked in bug 1156074, I'll fix it today.