Ubuntu
puppetmaster package

[SRU] Service restart fails because process has not exited

Bug #1315021 reported by Matthew Wedgwood
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
puppetmaster (Ubuntu)
Fix Released
Undecided
Felipe Reyes
Trusty
Fix Released
Undecided
Felipe Reyes
Utopic
Fix Released
Undecided
Felipe Reyes

Bug Description

[Impact]

Users restarting puppet and puppetmaster may have problems restarting the
daemons, in systems under heavy load may be more frequent, because if the stop
operation takes more than 1 second the start operation fails due that the
daemon is still running.

If the daemon isn't started in a machine managed by puppet (or the
puppetmaster) will require manual intervention to bring up the system.

[Test]

- Create a small VM (512MB, 1 vcpu)
- Install puppet and puppetmaster packages
- Restart each daemon twice with a 10 seconds pause
  # service puppetmaster restart; sleep 10; service puppetmaster restart

Expected results:

- The service is restarted

Actual results:

- The service is stopped, but it isn't started again the console prints:

root@vivid-puppetmaster:~# service puppetmaster restart ; sleep 10 ; service puppetmaster restart
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...fail!

[Regression Potential]

- If after 10 seconds the daemon isn't exited the start-stop-daemon command
  will send the KILL signal and wait for 5 seconds, this is different from the
  current behavior that just informs the restart failure to the user and
  doesn't send the KILL signal.

[Other Info]

When restarting the puppetmaster process (via the service command, or calling
the initscript directly), the restart can fail if the original process does
not exit immediately. This is because the initscript is not written to wait
for the original process to exit.

In my testing under Trusty, adding "-R n" to the start-stop-daemon command in
the initscript fixed the problem. It's difficult to determine an appropriate
value for n, but it could be tunable via /etc/default/puppetmaster. It's
definitely a problem not to wait at all for the process to exit.

See original description
Matthew Wedgwood (mew)
description: updated
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in puppetmaster (Ubuntu):
status: New → Confirmed
Revision history for this message
Bryan Quigley (bryanquigley) wrote :

Reproducible simply by:
$ sudo /etc/init.d/puppetmaster restart; sleep 10; sudo /etc/init.d/puppetmaster restart
 * Restarting puppet master [ OK ]
 * Restarting puppet master [fail]

tags: added: cts
Revision history for this message
Bryan Quigley (bryanquigley) wrote :

Upstream has another fix:
https://github.com/puppetlabs/puppet/commit/5a23d8b95bf05584dd8a9cba5aab13dfc029707a

Revision history for this message
Felipe Reyes (freyes) wrote :

For the record, upstream was aware of this problem and fixed their init script[0], this fix wasn't picked by the deb package.

[0] https://github.com/puppetlabs/puppet/commit/5a23d8b95bf05584dd8a9cba5aab13dfc029707a

Revision history for this message
Felipe Reyes (freyes) wrote :

Hi,

I added a '--retry TERM/10/KILL/5'that gives enough time to the stop operation succeed. Here is the output of a vivid system without the patch:

root@vivid-puppetmaster:~# service puppetmaster restart ; sleep 10 ; service puppetmaster restart
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...fail!

And here it is with the patch applied:

root@vivid-puppetmaster:~# for i in {0..10}; do service puppetmaster restart ; sleep 10 ;done
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.
 * Restarting puppet master
   ...done.

Changed in puppetmaster (Ubuntu):
assignee: nobody → Felipe Reyes (freyes)
Revision history for this message
Felipe Reyes (freyes) wrote :

Here I'm uploading a new version of the patch that fixes the same problem for /etc/init.d/puppet and removes a unneeded 'sleep 1' between stop and start.

Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching backport of the patch for utopic.

description: updated
summary: - Service restart fails because process has not exited
+ [SRU] Service restart fails because process has not exited
Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching backport of the patch for trusty

Revision history for this message
Chris J Arges (arges) wrote :

Already uploaded and in vivid.

Changed in puppetmaster (Ubuntu Trusty):
assignee: nobody → Felipe Reyes (freyes)
Changed in puppetmaster (Ubuntu Utopic):
assignee: nobody → Felipe Reyes (freyes)
Changed in puppetmaster (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote :

Sponsored for Trusty/Utopic.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Matthew, or anyone else affected,

Accepted puppet into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/puppet/3.6.1-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Matthew, or anyone else affected,

Accepted puppet into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/puppet/3.4.3-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Felipe Reyes (freyes) wrote :

I installed puppet and puppetmaster from *-proposed inside a clean lxc container, I successfully restarted the services without failures in loop for more than 10 minutes, none of the restarts failed.

tags: added: verification-done-trusty verification-done-utopic
removed: verification-needed
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of the Stable Release Update for puppet has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Felipe Reyes (freyes)
Changed in puppetmaster (Ubuntu Trusty):
status: New → Fix Released
Changed in puppetmaster (Ubuntu Utopic):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.