cupsd crashes regularly (daily)

I reproduced the crash again today, this time with dbus debug symbols installed. The stack trace is attached. The stack is entirely avahi + dbus code, and the signature is very similar to other crashes reported in other applications. Looks like there might be a locking problem in the way cups uses avahi?

This thread describes a similar stack trace, and has some tips about using avahi in a threaded environment:
http://marc.info/?l=freedesktop-avahi&m=122021745832173

Comment #12 of bug 524566 shows a very similar stack (without avahi), and the remainder of the bug has additional analysis.

Bug 1055060 has a crash at the same location within dbus, though the bottom of the stack is much different.

Status changed to 'Confirmed' because the bug affects multiple users.

I reproduced again. There is a different stack trace, but it's still triggered from avahi calling into dbus_watch_handle.

I have done some tests to investigate the problem. After removing the patch for forward-porting the CUPS Broadcasting/Browsing and running the resulting CUPS for a week I did not get any crash any more, returning to the original version gave me one crash every day or every two days. Then I tried the attached patch to set the needed locks for using Avahi in a threaded environment, but instead of crashing CUPS gets stuck, still running but not responding any more. This is even worse as in the case of a crash Upstart restarts CUPS and so CUPS stays available.

So as the patch for CUPS Broadcasting/Browsing is planned to be removed again in Raring, I suggest for an SRU in Quantal only to suppress the Apport pop-ups of the crashes somehow, as a workaround. I talked with pitti on IRC about this possibility.

Setting to "Triaged" in Raring as the problem will go away with the planned removal of the patch.

> I suggest for an SRU in Quantal only to suppress the Apport pop-ups of the crashes somehow, as a workaround.

The simplest, but rather blunt, method is to add a file "/etc/apport/blacklist.d/cups" to a cups SRU with "/usr/sbin/cupsd". However, that would suppress *all* cups crashes, not only this one.

In order to filter out only this one in a way that completely avoids the popup (as opposed to saying "we already know about this crash"), the crash needs to be suppressed in the daemon. If you know the place where it crashes, you could temporarily install a SIGSEGV signal handler, then call the potentially crashing code path, and restore it. However, this is pretty hackish, and I wonder if there wouldn't be a more appropriate fix such as adding a NULL pointer check somewhere etc.

Unfortunately, the actual crash happens somewhere in Avahi or D-Bus and the stacktraces do not show through which part of CUPS the crashing piece of code was reached. Probably it is in a thread started by Avahi. So it is far from simple to find the place in CUPS where an additional NULL check is needed (this would be the real fix).

Therefore and keeping in mind that the CUPS broadcasting/browsing patch will go away in Raring, I suggest the general blocking of cupsd crash reports via /etc/apport/blacklist.d/cups. In addition, CUPS was very stable all the time, so the chance of other crashes to happen is rather low. So I think we can take the risk.

Uploaded a fixed package to quantal-proposed. As soon as the package gets approved it will be available for testing and a comment with instructions will be posted here. Please test it and give us fedback as your feedback is required to make the package an official update.

SRU Team: debdiff is attached.

Note thst the proposed package is applying a workaround which is specific to Quantal. The fix in Raring will be removing the patch for forward-porting CUPS broadcasting/browsing and replacing it by an extra daemon for Avahi browsing, making use of CUPS' Avahi broadcasts. The daemon will be part of the cups-filters package and I have nearly completed its development. So please process this SRU without the Raring task being "Fix Released".

For me the crash always happens when the logrotate is done around 8am in the morning (beginning of /ver/log/syslog), probably the logrotating of the CUPS log files (perhaps the reload of the CUPS daemon) triggers the crash.

Restarting CUPS as in /etc/logrotate.d/cups, via

sudo invoke-rc.d --quiet cups force-reload

does not cause a crash for me, so probably manipulating the log files of CUPS (with CUPS still running), right before restarting CUPS causes the crash.

I have now released cups-filters 1.0.26 upstream, with the new cups-browsed daemon added. cups-browsed browses the Bonjour broadcasts of shared remote CUPS queues and makes the queues available locally, eliminating the need of the forward-port patch for CUPS broadcasting/browsing. As soon as this package is available in Raring I will remove the patch from Raring's CUPS, eliminating the crashes.

This bug was fixed in the package cups - 1.6.1-0ubuntu13

---------------
cups (1.6.1-0ubuntu13) raring; urgency=low

  * debian/patches/forward-port-cups-1-5-x-cups-browsing.patch: Removed the
    forward-port of CUPS broadcasting/browsing as from cups-filters 1.0.27
    on we have cups-browsed which does Bonjour browsing for us making the
    patch unneeded. This eliminates also a crash bug in cupsd (LP: #1086019,
    LP: #1061063, LP: #1061069).
  * debian/rules: Removed "--with-remote_protocols='CUPS dnssd'" from the
    ./configure command line and removed the "CUPS" from
    "--with-local_protocols='CUPS dnssd'". These settings are not supported
    any more in stock CUPS 1.6.x (without forward-port of CUPS broadcasting/
    browsing).
  * debian/cups.postinst: Again clean /etc/cups/cupsd.conf from all keywords
    and settings which got obsolete with the dropping CUPS Broadcasting/
    Browsing in CUPS 1.6.x: BrowsePoll, BrowseAllow, BrowseDeny, BrowseOrder,
    and BrowseRemoteProtocols lines get removed and the "cups" argument gets
    removed from the BrowseLocalProtocols line.
  * debian/patches/airprint-support.patch,
    debian/patches/cupsd-no-crash-on-avahi-threaded-poll-shutdown.patch:
    Refreshed with quilt.
  * debian/patches/CVE-2012-5519.patch: Manually updated.
 -- Till Kamppeter <email address hidden> Fri, 28 Dec 2012 22:27:30 +0100

This bug was fixed in the package cups-filters - 1.0.28-0ubuntu1

---------------
cups-filters (1.0.28-0ubuntu1) raring; urgency=low

  * New upstream release
     - cups-browsed: Added daemon to browse the Bonjour broadcasts of
       shared remote CUPS printers and automatically add local raw queues
       pointing to them, to resemble the behavior of the former CUPS
       broadcasting/browsing which was dropped in CUPS 1.6. Now remote
       printers appear as local print queues as before, but with the
       standardized Bonjour broadcasting (LP: #1086019, LP: #1061063,
       LP: #1061069).
  * debian/control: Added build dependencies on Avahi, needed for the new
    cups-browsed and also a new binary package named cups-browsed.
  * debian/cups-browsed.install: New cups-browsed binary package.
  * debian/local/cups-browsed.upstart: Upstart configuration file for
    cups-browsed.
  * debian/rules: Added everything needed so that the cups-browsed daemon gets
    started automatically.
  * debian/cups-filters.install: Include files from /usr/bin.
 -- Till Kamppeter <email address hidden> Sat, 29 Dec 2012 14:30:07 +0100

Everyone who suffers this problem (probably only Quantal users), please replace your file /etc/logrotate.d/cups by the attached file:

sudo cp cups.logrotate /etc/logrotate.d/cups

Now observe for some days. Do the daily crashes go away?

cupsd crashes nearly everyday on my machine.
I just replaced /etc/logrotate.d/cups by your file and I will tell you Monday whether the crash still occurs.

Hi,
It is now Tuesday and cupsd has not crashed since I replaced '/etc/logrotate.d/cups' , my machine was online all the week end.

Cedric, thanks for the test. I will apply the new logrotate script for CUPS soon, also for newer versions to avoid crashes and problems in the future, simply by having a cleaner logrotate script.

Everyone testing my new logrotate script (comment #15), please do not forget to remove the /etc/apport/blacklist.d/cups file from the initially proposed package.

Uploaded a new fixed package to quantal-proposed, this time using the improved logrotate script. As soon as the package gets approved it will be available for testing and a comment with instructions will be posted here. Please test it and give us feedback as your feedback is required to make the package an official update.

SRU Team: debdiff is attached.

Cedric, and everyone else who tested my improved logrotate script, as soon as the proposed package gets available please run the following commands in a terminal window to remove my script before installing the proposed package:

sudo rm /etc/logrotate.d/cups /etc/apport/blacklist.d/cups
mkdir tmpdir
cd tmpdir
apt-get download cups
sudo dpkg -i --force-confmiss cups_*.deb
cd ..
rm -rf tmpdir

Then install the package as described in the instructions which will get posted here.

Note that testing the proposed package and giving feedback here is required for the proposed package to get an official update.

When testing using the commands above, please replace

sudo dpkg -i --force-confmiss cups_*.deb

by

sudo dpkg -i --force-confmiss cups-daemon_*.deb
sudo dpkg -i --force-confmiss cups_*.deb

I think I missed something, where do I find the 'proposed package' ?
Will automatic update propose it to me? Do I have to upgrade Ubuntu from 12.10 to 13.04 ?

Cedric, you do not need to upgrade to 13.04.

The proposed package is not yet made available. It will be announced here. I will try to make the SRU aware of the fact that they still did not approve this package.

Hello Steve, or anyone else affected,

Accepted cups into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cups/1.6.1-0ubuntu11.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Cedric, please follow the instructions of comment #25, comment #21, and comment #22 and tell us your results here. First, do the steps of BOTH comment #21 and comment #22 to clean up the configuration changes of your first testing/workaround. ONLY AFTER THAT, follow the instructions of comment #25.

Please report your results here. Thanks.

Already did #21 two weeks ago and crashes came back.

#22 does not work:
--------------------
cmarin@BAT:~/tmpdir$ sudo dpkg -i --force-confmiss cups-daemon_*.deb

dpkg: erreur de traitement de cups-daemon_*.deb (--install) :
 ne peut pas accéder à l'archive: Aucun fichier ou dossier de ce type
Des erreurs ont été rencontrées pendant l'exécution :
 cups-daemon_*.deb
---------------------
in english:
 cannot access to archive: no such file or directory

I tried to add a "apt-get download cups-daemon" but it does not work either.

Cedric, sorry, forget about comment #22. The cups-daem,on package was only introduced with 13.04. So you need to follow only the instructions of comment #21 and this you have done two weeks ago. Now you should observe the daily crashes, which you already confirmed.

So now please install the proposed package as described in comment #25. After that you should not observe the crashes any more. Please tell us whether this is the case.

#25 done via automatic update: it worked but a dialog box asked me whether to replace or not the content of a file like "logrotate.d/cups" (I didn't note it, sorry). The file changes text field was empty.

I choose "replace" as recommended by the dialog but if this file is related to the crash maybe the replace should be forced or at least it should be told that 'replace' is needed to fix a crash.

Cedric, the pop-up about replacing the file does not appear for normal users but only for you because you have manually put logrotate.d/cups file in place for doing the first tests I asked you for. Having answered "Replace" was correct. You are now using the file of the package. So now you are all set for the test. If you have no crash report tomorrow, the fix in the proposed package is verified.

Bad news : I had a cupsd crash this morning.
I will try to post the apport crash here.

Here is the crash log found on /var/crash .

Cedric, can you run

ls -l /etc/apport/blacklist.d/cups*

post the output here, and attach all files which get listed by this command.

Please attach the files one by one, do not compress or package them.

I just checked the '/etc/logrotate.d/cups' file and it has the same content as the improved script you provided.

>> ls -l /etc/apport/blacklist.d/cups*
ls: impossible d'accéder à /etc/apport/blacklist.d/cups*: Aucun fichier ou dossier de ce type

==> No such file or directory

>> ls -l /etc/apport/blacklist.d/*
-rw-r--r-- 1 root root 216 oct. 10 2012 /etc/apport/blacklist.d/apport
-rw-r--r-- 1 root root 59 oct. 11 2012 /etc/apport/blacklist.d/firefox
-rw-r--r-- 1 root root 217 oct. 1 2012 /etc/apport/blacklist.d/README.blacklist
-rw-r--r-- 1 root root 71 oct. 11 2012 /etc/apport/blacklist.d/thunderbird

Cedric, sorry, forget about the previous comment, I wanted to ask you another thing.

You are sure that your file /etc/logrotate.d/cups is exactly the same as the one I asked you for testing in comment #15.

Can you attach your /etc/logrotate.d/cups file here? Thanks.

Here is my '/etc/logrotate.d/cups' file.

My machine will be online the 3 following days and idle.
I will see Tuesday if other crashes occurred.
If you want I can also reboot the machine before leaving the office (in one hour).

Cedric, the script is the correct one. Please do the reboot and then we will see whether you get further crashes.

Hello, there is no more crash since the reboot last week.
I needed to wait more because we had a general power failure nearly at the hour where cupsd would usually crash.
( I needed to check your fix didn't cause the power failure :-P )
I think you can set the bug as fixed now.

Cedric, thank you very much. I have marked the fix as verified now.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package cups - 1.6.1-0ubuntu11.4

---------------
cups (1.6.1-0ubuntu11.4) quantal-proposed; urgency=low

  * debian/cups.logrotate: Do not remove CUPS' log files while the CUPS
    daemon is running. Stop CUPS, move the files, and then start it again.
    This avoids crashes during the log rotation process (LP: #1086019).
 -- Till Kamppeter <email address hidden> Tue, 23 Apr 2013 12:12:12 +0200

all users can't print from libreoffice (oodt) files - only PDF printing works