Broken logrotate script in freeradius-2.1.12+dfsg-1.2ubuntu8 on Ubuntu 14.04 LTS

I just realized that there is a better solution for the logrotate script: By using "service freeradius reload", the script will work when Upstart is used but also when the traditional init script is used.

The attachment "Suggested patch for logrotate script" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Status changed to 'Confirmed' because the bug affects multiple users.

Wily is using the systemV init scripts and is not affected by this bug.

Trusty is using upstart which lacks a reload function currently.

Changing the logrotate postrotate script on trusty to call 'service freeradius reload' will not work without explicitly creating a reload function in the upstart job. The patch as included in comment #2 will kill the parent of freeradius (which is the upstart shell script) and will attempt to restart it but the original freeradius process is still running and no longer managed by upstart. When this occurs, upstart will show that the freeradius service is not running and will not be able to start it because the port is in use by the un-managed freeradius process.

Next action:
 - Investigate adding a reload handler for the upstart job. This will need the PID for the freeradius process that was started so that it can send a HUP signal as seen in the systemV init job. In the case that the PID file is missing then the reload should do nothing and exit cleanly as the lack of PID file would indicate that freeradius is not running; we would not want to restart the service as this would have the affect of starting a stopped service. Then change logrotate script in trusty to call 'service freeradius reload' and consider removing the redirect to /dev/null.

I have run out of time today but have left these notes so that we can pursue it further.

Here is a debdiff to show how we can manage the process properly in an upstart job (missing exec) so that a restart or reload works as we would expect. The patch also fixes the logrotate script to use the upstart controls rather than the systemv init script (which should not have been shipped because it isn't used on Trusty).

Prior to this patch you can't restart/reload the process because the process isn't properly managed by upstart and logrotate's reload of freeradius was a no-op. With this patch, logstart will properly call 'service freeradius reload', ignoring failure due to the service not running.

Package upgrade is a problem.... Is there a way to fix the management of a running freeradius process on a user's system when this package is updated? We would not want to perform an unexpected restart of the service and without this fix reloading the service won't work (nor will the restart because it kills the shell that start the freeradius process but not the freeradius process itself). However, having the upgraded package on the system won't make the current situation any worse and will resolve the issue on the first reboot (or manually killing freeradius and starting again).

Best I know that we can do is put the fix in place and it will be available after a restart; having the upgraded package on the system won't make the current situation any worse but it won't fix the issue prior to a restart or some manual intervention to clean up the unmanaged freeradius process.

To illustrate the current process problem:
 1 - Install freeradius on trusty
 2 - Run 'start freeradius'
 3 - Get the PID upstart is using for freeradius with 'status freeradius'
 4 - Check ps for that PID
    (e.g. if PID is 1023 run 'ps -ef | grep 1023 | grep -v grep'. Upstart job PID will map to /bin/sh, the parent of freeradius)
 5 - Run 'reload freeradius'.
      Expect success, exit 0 and freeradius is still running (SIGHUP was not delivered, check with a shim if you like)
 6 - Run 'status freeradius' .
      Results in "freeradius stop/waiting" because the parent shell script died when it received HUP but freeradius is still running.
 7 - Run 'ps -ef |grep freeradius'.
      Expect freeradius still running (same PID as step 4 but PPID is now init)
      This process isn't tracked by upstart now. You can't 'start freeradius' because the new process will exit upon failing to open the port already opened by the untracked freeradius.

Robie / Chris,

Can you review comment #6 for the proposed patch and upgrade issue/strategy?

The upstart config for freeradius is bad; reload(8) or restart(8) will lose track of the process. A reload won't send SIGHUP to the right process and a restart(8) will not stop the existing freeradius process; in either case upstart loses track of the process and thinks the service has stopped, further attempts to start the service will fail because of the existing process holding the port open. The patch fixes this for new, clean starts of the service but on a running system I don't think we have a way to make things better without manual intervention ('pkill freeradius' when the user can afford to take the process down) or a reboot.

I might be missing something clever, but I doubt there is a good way to fix this for the different states we might find things in during upgrade.

Scenarios:
1 - freeradius is not running. Upgrading the package is great.
2 - freeradius is runing and 'reload/restart' has been called previously (i.e. upstart can't manage the freeradius process). Upgrading will not regress the system because upstart could not manage the service prior to package upgrade.
3 - freeradius is running and 'reload/restart' has never been called (i.e. upstart can still stop the process properly). Upgrading will fix logrotate and the next rotation will call 'reload freeradius' causing upstart to lose track of the process. It can no longer be stopped by upstart.

The only issue is scenario #3 which regresses the user's ability to stop freeradius *if* they have not tried to reload or restart the service. I don't think that we can take the action to restart the service on package upgrade if upstart can still manage the service due to service interruption. The next logrotation attempt will ensure that the user can not manage freeradius from upstart.

Updated the patch. The post-install scripts will restart the daemon so that an upgrade will restart the daemon so that future reloads do work. This addresses the 3rd scenario in comment #7. There would be no regression for users on upgrade. If the service has been reloaded prior to the upgrade such that upstart can't manage the daemon then this won't fix things but won't regress the user either. In that case, the user will need to kill the running, unmanaged freeradius process and then start from upstart again.

Marking vivid as invalid as the this affects trusty (upstart) only not systemd based releases (vivid/wily)

Back to the prior patch to reduce scope by removing changes for 'init.d' -> 'service' for force-reload operations in postinst scripts if invoke-rc.d is not available. This is not changed as we can assume invoke-rc.d is installed and the SRU change needs to be minimal.

Robie, can you review the patch in comment #10 for this fix? Thanks.

Thanks! This looks good. I particularly appreciate the effort you took to figure through any upgrade path issues and the comprehensive coverage you have of that in the Regression Potential section. This gives me much more confidence in sponsoring this.

One very minor nitpick - I used version 2.1.12+dfsg-1.2ubuntu8.1 instead of 2.1.12+dfsg-1.2ubuntu9 to make it clear that this is an SRU. Technically 2.1.12+dfsg-1.2ubuntu9 would work in this case though and the SRU team have no specific policy against it. More information on picking a version number here (for version numbers, the same scheme works for both security and SRUs): https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging

Uploaded. Thanks again!

Hello Sebastian, or anyone else affected,

Accepted freeradius into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeradius/2.1.12+dfsg-1.2ubuntu8.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I verified that the bug is fixed in version 2.1.12+dfsg-1.2ubuntu8.1 and that the upgrade process (restart) works as intended.

Thank you very much for your efforts in fixing this bug.

This bug was fixed in the package freeradius - 2.1.12+dfsg-1.2ubuntu8.1

---------------
freeradius (2.1.12+dfsg-1.2ubuntu8.1) trusty; urgency=medium

  * Manage process in upstart properly and fix logrotate reload (LP: #1406105)

 -- Robert C Jennings <email address hidden> Tue, 04 Aug 2015 21:00:25 -0500

The verification of the Stable Release Update for freeradius has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.