libvirt-bin upstart job considers service "started" before it is ready

Root cause analysis:

libvirtd creates and starts listening on the socket after forking to daemonize. However, it also ensures that the original process only exits after the daemon process has started to listen on the socket (using a pipe to signal readiness internally).

In the Sys V init case, this means that the init.d script would not exit until the daemon is ready.

In the Upstart case, it seems to me that Upstart's documented "expect daemon" behaviour means that Upstart treats the daemon as started as soon as the second (daemon) fork occurs, and there is no provision to wait() on the original process.

Here is the patch I'm using to reproduce the race.

cjwatson suggested that I consider "expect stop" like this openssh patch: http://anonscm.debian.org/loggerhead/pkg-ssh/openssh/trunk/revision/3578

However, I also wonder if it would be a problem for upstart to wait() on the original process anyway in the cases of "expect daemon" and "expect fork". Is there any reason why this would break anything? It would seem closer to Sys V's behaviour to me.

So, I think there are three things we can do:

1) Make uvtool's postinst wait for libvirtd's unix socket to exist and be connectable. I think I should do this anyway as a workaround, as it's not particularly intrusive, will get us by for now and will help with the no-change backport to Precise I'd like to be possible without having to worry about a libvirt SRU. But it doesn't seem like a clean solution to me. I have a debdiff ready for this that uses socat, and another that uses a Python script to avoid a universe dependency in case we want to MIR uvtool in the future.

2) Modify libvirt's packaging to use "expect stop". I have an experimental debdiff that appears to work fine with this option and solves the original issue with no modification to uvtool's postinst needed. But it seems a bit ugly to me, since libvirt's behaviour in not existing the original called process before it is ready seems perfectly reasonable to me.

3) Make upstart wait on the original called process after it detects the fork with "expect daemon". Presumably this applies to "expect fork" as well. Is there any use case for which this isn't reasonable or would create a regression? Are there any other problems with this approach?

Patch to make uvtool-libvirt.postinst wait for libvirtd's socket to be ready with socat.

Patch to make uvtool-libvirt.postinst wait for libvirtd's socket to be ready with Python, to avoid a universe dependency.

Experimental patch on libvirt. This introduces a sleep to help with demonstrating and testing the race, converts the packaging to use "expect stop" and patches libvirt to support sending itself a SIGSTOP when it is ready.

Further discussion here: https://lists.ubuntu.com/archives/upstart-devel/2013-October/002772.html

Following that thread, I've come to the following conclusions:

libvirtd is perfectly acceptable in its current behaviour. upstart plans to support this edge case in the future. The only impact of this bug at the moment is in uvtool's postinst. A simple workaround can be used in uvtool's postinst. Although we can patch libvirtd to support "expect stop", it seems pretty invasive for something that we might want to drop in the future, and a the workaround in uvtool's postinst seems like the easier option.

So I'll mark this as Won't Fix in libvirt for now, to reflect the existence of this problem but my current intention to leave it as is. And I'll work around the problem in uvtool.

This bug was fixed in the package uvtool - 0~bzr36-0ubuntu1

---------------
uvtool (0~bzr36-0ubuntu1) saucy; urgency=low

  * New upstream snapshot:
    - Workaround to make sure that libvirtd is running and ready before
      attempting to create the volume pool (LP: #1228210).
      + d/uvtool-libvirt-postinst: wait logic.
      + d/control: add dependency on socat (used by the wait logic).
 -- Robie Basak <email address hidden> Wed, 02 Oct 2013 11:35:06 +0100

Fixed in bzr rev 36.