Make duplicate signature more specific for DBusException and OSError

I believe the crash signature used by daisy is actually just the crash signature apport provides it. So the fix for this bug would need to happen in apport and its crash_signature function from report.py.

In bug 772083 we can see that the retracer used the incomplete signature to mark it as a duplicate of bug 781360.

We can see how the crash signature is created by apport for python tracebacks here: http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu/quantal/apport/ubuntu/view/head:/apport/report.py#L1206.

The problematic bit follows:

return self['ExecutablePath'] + ':' + trace[-1].split(':')[0] + sig

Here were just using the first part of the last line of the trace to create the signature. Considering just the trace[-1].split(':')[0] portion we have the same partial signature for bugs 772083 781360 - 'SystemError'. However, if we expand this out a bit we have:

772083: SystemError: E:Encountered a section with no Package: header, E:Problem with MergeList /var/lib/apt/lists/archive.canonical.com_ubuntu_dists_natty_partner_binary-i386_Packages, E:The package lists or status file could not be parsed or opened.
781630: SystemError: E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened.

which is quite different. For the partial signature we could use all of trace[-1] or a portion there of. I have code that will download all the tracebacks for a package and compare their old partial signatures to a new one.

Due to the nature of Python exceptions this indeed can only ever be a heuristic. Some Python errors are already very specific, and we *don't* want their message be part of the signature. For example, we want to treat PermissionError('/home/fred/foo') and PermissionError('/home/joe/bar') as one and the same bug. Likewise, in the example in the description we would want "org.freedesktop.Accounts.Error.Failed: 'bg_BG.UTF-8' is not a valid locale name" to be treated exactly the same as any other locale name. Exception messages also often have localized or otherwise user/system specific context in them.

For DBusError the situation is still relatively easy -- this is a well-known one from dbus-python which actually wraps a "real" exception on the server. For the duplicate signature we certainly should unwrap that inner exception, so that we end up with something like "DBusException:org.freedesktop.Accounts.Error.PermissionDenied".

The other case of pyapt's SystemErrors are harder to dissect as there is no structure in the exceptions and their messages. So the only real options that we have are to keep the status quo (don't use the message at all) or take it wholesale. But I think the latter wouldn't be helpful as it would consider too many things as "not the same bug". One could go further and just take the last word, but I think that would be completely arbitrary and still not help (translations/not capturing the important part).

So I propose to do one step at a time and unwrap DBusExceptions for now.

We should also dissect OSErrors and append errno. Python 3 already does that by having specific subclasses such as PermissionError for EACCES, but this is still useful for Python 2 or if we run into an errno which doesn't have a subclass.

Implemented in apport trunk r3066.

This should also be SRUed.

This bug was fixed in the package apport - 2.20.1-0ubuntu1

---------------
apport (2.20.1-0ubuntu1) xenial; urgency=medium

  * New upstream release. Changes since our previous snapshot:
    - crash-digger: Untag bugs which cannot be retraced instead of stopping
      crash-digger. This led to too many pointless manual restarts on broken bug
      reports.
    * Disambiguate overly generic Python exceptions in duplicate signature
      computation: dbus-glib's DBusException wraps a "real" server-side
      exception, so add the class of that to disambiguate different crashes;
      for OSError that is not a known subclass like FileNotFoundError, add the
      errno. (LP: #989819)

 -- Martin Pitt <email address hidden> Thu, 31 Mar 2016 16:16:37 +0200

Trusty backport of http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/3066 uploaded to the SRU review queue.

Hello Sebastien, or anyone else affected,

Accepted apport into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.14.1-0ubuntu3.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

bdmurray@clean-trusty-amd64:~$ apt-cache policy apport
apport:
  Installed: 2.14.1-0ubuntu3.20
  Candidate: 2.14.1-0ubuntu3.20
  Version table:
 *** 2.14.1-0ubuntu3.20 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.14.1-0ubuntu3.19 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty-updates/main amd64 Packages
     2.14.1-0ubuntu3.18 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty-security/main amd64 Packages
     2.14.1-0ubuntu3 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty/main amd64 Packages
bdmurray@clean-trusty-amd64:~$ python3 -c 'import apport; r = apport.Report(); r.load(open("/var/crash/_usr_share_dh-python_pybuild.1000.crash", "rb")); print(r.crash_signature())'
/usr/share/dh-python/pybuild:OSError(99):/usr/share/dh-python/pybuild@449:main

This bug was fixed in the package apport - 2.14.1-0ubuntu3.20

---------------
apport (2.14.1-0ubuntu3.20) trusty-proposed; urgency=medium

  * Disambiguate overly generic Python exceptions in duplicate signature
    computation: dbus-glib's DBusException wraps a "real" server-side
    exception, so add the class of that to disambiguate different crashes;
    for OSError that is not a known subclass like FileNotFoundError, add the
    errno. (LP: #989819)

 -- Martin Pitt <email address hidden> Fri, 01 Apr 2016 16:27:39 +0200

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.