packages fail to install: Failed to connect to socket /com/ubuntu/upstart: Connection refused

Status changed to 'Confirmed' because the bug affects multiple users.

Terminal log contains:

----------
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
invoke-rc.d: initscript cups, action "start" failed.
dpkg: error processing cups (--configure):
 subprocess installed post-installation script returned error exit status 1
----------

Seems that the post-install script of CUPS is not able to start the CUPS daemon because Upstart is not running. Reassigning ...

This is piling up a number of duplicates, bumping importance.

Most of the duplicates are from saucy:

 $ lp-bug-dupe-properties --rtags -b 901038
LP: #901038 has 4 duplicates
  precise: 901035
  saucy: 1189175 1189243 1191237

xnox - could you have a look at this as james is on holiday for a bit? thanks!

I believe the problem here is that although upgrading the Upstart package itself and any of its dependent library packages will force a stateful re-exec, that operation severs all D-Bus connections (since D-Bus does not provide a way to serialise objects such as DBusConnection) and crucially that operation is async...

= Scenario causing this Bug =

I think the behaviour these bugs are caused by is something like:

1) dpkg upgrades Upstart (or one of its dependent libraries).
2) The relevant postinst maintainer script requests Upstart re-exec itself.
3) The re-exec starts (asynchronously).
4) dpkg moves on to installing/upgrading other packages.
5) invoke-rc.d, called from a maintainer script from another package decides it needs to call start(8) or initctl(8) for example.
6) The appropriate command is run (start/initctl), but fails since Upstart is still finishing re-exec'ing itself.

= Thoughts =

== Tweak invoke-rc.d ==

One possible fix would be to modify invoke-rc.d so that before attempting to run start etc, it first checks that Upstart is contactable by waiting for a "reasonable amount of time" for Upstart to respond (assuming it could be in the process of being re-exec'ed). However, that could be unreliable unless that check occurred before *every* call to initctl/start/stop/restart/reload-configuration. This is achievable but would clutter the code a little.

== Create a dpkg Upstart trigger ==

Another option might be to create a dpkg trigger to handle restarting upstart and have that trigger call 'telinit u' *and then wait* again for that "reasonable amount of time" until Upstart responds before the trigger ends. This sounds attractive since the logic for restarting and waiting is contained in one location. However, if my understanding of triggers is correct (?) this would actually cause another problem: since Upstart grew the ability to re-exec, the expected behaviour is that "as soon as Upstart has been upgraded, the running version of Upstart will match the installed version of Upstart". Using a trigger would break this understanding since the Upstart restart would only occur *after* all the other packages (that declare an interest in the upstart trigger) have been updated.

== Modify initctl ==

A third option could be to modify initctl to make it retry on failure to connect. This could in fact be the default since on a normally working system, there would be no behaviour change. Then, when invoke-rc.d calls initctl/start/etc, it should DTRT under all scenarios.

= Plan =

Short term, tweaking invoke-rc.d is probably the most pragmatic (albeit potentially racy) solution. Long term, we should modify initctl.

Thinking about it, a better solution to modifying initctl would probably be to modify telinit such that it runs fully synchronous when a restart is requested (ie handes the D-Bus disconnection, then polls to reconnect). The only problem there is deciding upon a reasonable "reasonable time to wait" since we cannot reasonably change the behaviour of telinit by adding extra options.

Yet another idea: add a new initctl command 're-exec' that essentially does 'telinit u' but provides options to either run async or sync.

Strike the telinit "reasonable time" comment in #8 - if we've requested Upstart restart, the only reasonable time to wait is forever.

Just ran into this on a 14.04 -> 14.04.1 upgrade, and it confused the heck out of me, since it was a non-trivial amount of time before initctl started responding again. Can we get this prioritized and fixed (preferably by making telinit block until it's actually done)?

Also, why the heck does this re-exec business take so long? It took well over a minute from 'telinit u' until 'initctl version' worked again on the five machines I just upgraded. There's no way it has so much to do here that it should take that long.

As discussed on #ubuntu-devel with Adam (who was upgrading a number of trusty servers to utopic), the problem he saw was a combination of this bug coupled with bug 1338637 (which explains the poor re-exec performance).

I was actually updating trusty to trusty. ie: just regular SRUs will pretty much explode because of this, which was why I was so alarmed.

This bug was fixed in the package upstart - 1.13.2-0ubuntu6

---------------
upstart (1.13.2-0ubuntu6) vivid; urgency=medium

  * debian/upstart-bin.upstart.cron.daily: Emit "rotate-logs" event direct
    into session init, by-passing system upstart & session
    event-bridge. This way session logs will be rotated, even upstart is
    not system init.
  * debian/control: make upstart-monitor & upstart-dconf-bridge be
    installable with just session init, upstart-bin.
  * extra/upstart-local-bridge: implement systemd pid1 logic.
  * extra/upstart-event-bridge: make it re-emit upstart-local-bridge
    events direct.
  * util/telinit: Revert to synchronous behaviour coupled with unavoidable
    poll to ensure telinit only returns once a re-exec has completed (LP:
    #901038).
  * move udev & local bridges to upstart-bin package.
 -- Dimitri John Ledkov <email address hidden> Sat, 17 Jan 2015 01:59:42 +0000

Status changed to 'Confirmed' because the bug affects multiple users.