Ubuntu
upstart package

init: main process can respawn even when the job is stopping

Bug #568288 reported by Tvrtko Ursulin on 2010-04-22

This bug affects 8 people

Affects		Status	Importance	Assigned to	Milestone
	upstart	Fix Released	Medium	Dimitri John Ledkov
	upstart (Ubuntu)	Fix Released	Undecided	Unassigned

Bug Description

Binary package hint: upstart

Version: 10.4 Beta, latest as of 21.04.2010.

Sometimes Upstart seems to get confused when "respawn" statement is used for a service and respawns when process exits normally on explicit stop event. Log then looks like this:

Apr 22 09:39:08 ubuntu1004alpha3crescendo init: Connection from private client
Apr 22 09:39:08 ubuntu1004alpha3crescendo init: sav-protect goal changed from start to stop
Apr 22 09:39:08 ubuntu1004alpha3crescendo init: sav-protect state changed from running to pre-stop
Apr 22 09:39:08 ubuntu1004alpha3crescendo init: sav-protect pre-stop process (3415)
Apr 22 09:39:09 ubuntu1004alpha3crescendo savd: savd.daemon: Sophos Anti-Virus daemon stopped.
Apr 22 09:39:09 ubuntu1004alpha3crescendo init: sav-protect main process (3379) exited normally
Apr 22 09:39:09 ubuntu1004alpha3crescendo init: sav-protect main process ended, respawning
Apr 22 09:39:09 ubuntu1004alpha3crescendo init: sav-protect goal changed from stop to respawn
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect pre-stop process (3415) exited normally
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect goal changed from respawn to start
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect state changed from pre-stop to stopping
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: Handling stopping event
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect state changed from stopping to killed
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect state changed from killed to post-stop
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect state changed from post-stop to starting
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: Handling starting event
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect state changed from starting to pre-start
Apr 22 09:39:10 ubuntu1004alpha3crescendo init: sav-protect pre-start process (3431)
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect pre-start process (3431) exited normally
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect state changed from pre-start to spawned
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect main process (3453)
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect state changed from spawned to post-start
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect post-start process (3454)
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect post-start process (3454) exited normally
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: sav-protect state changed from post-start to running
Apr 22 09:39:11 ubuntu1004alpha3crescendo init: Handling started event
Apr 22 09:39:15 ubuntu1004alpha3crescendo savd: savd.daemon: Sophos Anti-Virus daemon started.

According to my observations this happens most of the time but not always. Just this morning I got it to stop the service cleanly, then I started, killed the service abnormally and after it respawned stop from then on only respawns it. I though it may be related to abnormal killing before, but even after fresh boot stopping the service only respawns now.

Related branches

lp:~xnox/upstart/fix-568288

lp:upstart

Merged into lp:~daniel-sonck/upstart/cron-replacement at revision 1284

Daniël Sonck: Approve on 2011-10-14

Revision history for this message

Scott James Remnant (Canonical) (canonical-scott) wrote on 2010-04-22:

This looks to be the case if you have a pre-stop script; the state will not be "stopping" so the respawn code is allowed to run (it never checks the goal)

Obviously we should never respawn if the goal is "stop"

Changed in upstart (Ubuntu):
status:	New → Invalid
summary:	- Upstart respawn handling seems sometimes broken + init: main process can respawn even when the job is stopping
Changed in upstart:
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

Tvrtko Ursulin (tvrtko-ursulin) wrote on 2010-04-23:

Or should it even respawn at all if main process has exited normally? I guess it depends on whether you want service to ever be able to exit without respawnig if goal is not stop.

Revision history for this message

Scott James Remnant (Canonical) (canonical-scott) wrote on 2010-04-23: Re: [Bug 568288] Re: init: main process can respawn even when the job is stopping

On Fri, 2010-04-23 at 08:13 +0000, Tvrtko Ursulin wrote:

> Or should it even respawn at all if main process has exited normally? I
> guess it depends on whether you want service to ever be able to exit
> without respawnig if goal is not stop.
>
No, but "normal exit" already covers that - the problem here is that it
checks the state of the job rather than the goal to determine whether to
inhibit respawn

Scott
--
Have you ever, ever felt like this?
Had strange things happen? Are you going round the twist?

Revision history for this message

Pascal Hartig (passy) wrote on 2011-09-22:

Apparently, this is the desired behavior. There is a test for this exact case:

/* Check that we can handle the running process of a respawn job
  * exiting before the pre-stop process finishes. This should
  * mark the job to be respawned when the pre-stop script finishes
  * instead of making any state change.
  */
TEST_FEATURE ("with respawn of running while pre-stop process");

I don't understand how this is beneficial, though. Isn't pre-stop the right way to stop a process that does not correctly reacts to signals?

Revision history for this message

alexius ludeman (lexinator) wrote on 2012-10-02:

This bug is affecting our application.

We configure upstart to respawn so that in production it will automatically restart the application if it crashes. We also use pre-stop to request over http to ask the process to exit. The pre-stop procedure allows the process to finish serving current clients requests, and other clean up duties before it exits. However the combination of these two options then cause the application to be restarted upon running "stop <service>". This seems like an unusual behavior to us.

So now we are faced with either abandoning upstart or put the respawn logic somewhere else.

thanks

Revision history for this message

Marcus Sundberg (adamel) wrote on 2012-11-02:

Properly handle stopping a job from pre-stop. Edit (1.2 KiB, text/plain)

The attached patch makes stopping a job from pre-stop work properly.

Both stop and restart actions work as intended, and the test case in comment #4 still succeeds.

Revision history for this message

Scott James Remnant (scott) wrote on 2012-11-29:

upstart-1.2-respawn-pre-stop.patch Edit (7.9 KiB, text/plain)

Here's a far simpler patch - it just skips the respawn handling entirely (including all the "failed" stuff) if the job is being stopped anyway.

I think this is probably "right"

Dimitri John Ledkov (xnox) on 2013-07-13

Changed in upstart:
assignee:	nobody → Dmitrijs Ledkovs (xnox)

Revision history for this message

Matthew Hall (mhall-9) wrote on 2013-10-03:

This bug is 3 years old and has a patch. Can we patch it and get this over with?

Dimitri John Ledkov (xnox) on 2013-10-07

Changed in upstart:
status:	Triaged → Fix Committed

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2014-01-26:

Part of upstart 1.10 release.

Changed in upstart:
status:	Fix Committed → Fix Released
Changed in upstart (Ubuntu):
status:	Invalid → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Patches

Add patch

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuupstart package

init: main process can respawn even when the job is stopping

Bug Description

Related branches

Other bug subscribers

Patches

Remote bug watches

Ubuntu
upstart package