No further respawns after a "telinit u"

10:23 < lool> Keybuk: I think my init just became somewhat broken on armel
              after today's upgrade; it's not respawning gettys anymore
10:23 < lool> The only thing in daemon.log is:
10:23 < lool> Mar 25 09:00:52 babbage init: Re-executing /sbin/init
10:24 < lool> In syslog I also see immediately thereafter:
10:24 < lool> Mar 25 09:06:49 babbage kernel: [159660.980000] udev: starting
              version 140
10:26 < lool> PID 1 is in: select(7, [3 5 6], [], [], NULL
10:41 < lool> It seems it's the libc6 upgrade triggering the restart of
              init/upstart
10:47 < lool> Keybuk: This happened on my two armel installs
10:48 < seb128> lool: today's upgrade should be pretty non-update since we are
                frozen for beta no?
10:48 < lool> Keybuk: I did a "touch /etc/event.d/ttyS0", this made upstart go
              out of the select, scan the file again, but it didn't respawn the
              getty
10:49 < lool> seb128: I think the bug was there before beta, but any libc
              update which runs "init u" will have triggered it, and there was
              a libc update just before beta freeze (last friday)
10:49 < lool> I'm only upgrading now
10:50 < lool> seb128: But right, it's the upgrade which I ran today, not from
              the updates the archive got today, thanks for clarifying
10:50 < lool> running "init u" manually has a larger effect, but still doesn't
              respawn; /me reboots

(Confirmed after a reboot that "init u" still breaks)

Not specific to armel; same thing on my amd64 jaunty desktop.

So what's needed is for upstart to save it's state across re-exec()s; this is a large task.

The reason the kill is currently needed is bug #188925. Thus it can't be reverted/changed into a log.

We should recommend a reboot after libc6 upgrades for now.

The Upstart task is covered by bug #348455

(I can't seem to mark only a task as a dup - and don't want to pollute the upstream upstart bug with a discussion about Ubuntu's glibc and reboot policies)

How about saving to an unlinked tmp file that is not marked for
close-on-exec, where the state can be serialized/unserialized across the
exec?

On Thu, 2009-03-26 at 19:01 +0000, Kees Cook wrote:

> How about saving to an unlinked tmp file that is not marked for
> close-on-exec, where the state can be serialized/unserialized across the
> exec?
>
There's lots of interesting ways to do it.

That's not the problem ;)

The problem is that once you define that API, it has to be stable
*forever* to support upgrades.

I'm not willing to define that API just yet.

Scott
--
Scott James Remnant
<email address hidden>

Upgrading should only be necessary between nearby versions, so it doesn't seem a problem to change/break the dump format now and again. But I agree that it's significant work to set up such a dump/reload mechanism if it doesn't exist yet.

When upgrading to a version of upstart which is too sidtant, it seems reasonable still to require a reboot since this will not be the common case. Would it perhaps be better not to attempt to re-exec init at all after an upgrade of this type, and simply to wait for the next reboot, having alerted the user in some way? This is not ideal, but it seems better than breaking terminal logins until the next reboot.

"sidtant" = "distant" (oops)

Dave, if we don't re-exec upstart it keeps using the old libc and prevents remounting of the / read-only; this puts users at risk of data loss so is more critical.

I think it should always be possible to remount / read-only unless init or other required processes hold some files open for write access.
I can't see why re-exec'ing init or not would make a difference to this; it only affects when the inodes of the old libc are harvested; that's down to filesystem internals and shouldn't affect the remount if I understand correctly.

Bug #188925 is where this was decided

Ah, right.

I had a prod, and I understand what's going on a bit better now--- you're right, upgrading libc (or indeed any libs) will prevent read-only remount of the affected filesystems until the processes using the old libraries die or re-exec.

I couldn't work out why this happened, but is looks like having a _deleted_ file open or mmapped (even if read-only) prevents the filesystem from being remounted readonly. This seems counterintuitive, but I guess the backend reason is to avoid the inode being orphaned in the filesystem. In principle, this could be avoided without keeping the filesystem writable, but it's really a mainline kernel and filesystem issue and not Ubuntu specific.

We could work around the issue in the libc upgrade by moving the old libc elsewhere in / instead of deleting it (it must be a real move, not a copy-unlink; I've confirmed that it works). This could possibly be done in the libc6 maintainer scripts. The file can be cleaned up after the next boot.

However, it may not be too worthwhile to spend effort on setting this up if there is a longer term plan to make upstart support re-exec'ing itself better anyway... ?