apport-retrace ignores warnings from gdb

Bug #1354571 reported by Brian Murray
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apport (Ubuntu)
Fix Released
Medium
Martin Pitt
Precise
Fix Released
Medium
Unassigned
Trusty
Fix Released
Medium
Brian Murray

Bug Description

[Impact]
Crash reports with damaged core dumps are being sent to the Ubuntu Error Tracker, where they are added to the retracing queue and then an attempt is made to retrace them. These retraces fail because the core dump is damaged so resources are wasted. We should stop sending these to the Error Tracker.

[Test Case]
apport's test_report.py has gained a new test called test_add_gdb_info_damaged which tests that a crash report with a damaged core dump raises an IOError.
Running this test with the version of the package in trusty-updates will fail with an AssertionError.
Running the test with the version of the package from trusty-proposed will pass.

I was trying to investigate a failed retrace on armhf and received the following when running apport-retrace with -g.

Calling gdb command: gdb-multiarch --ex 'set architecture arm' --ex 'set gnutarget elf32-littlearm' --ex 'set debug-file-directory /tmp/apport
-utopic/usr/lib/debug' --ex 'set solib-absolute-prefix /tmp/apport-utopic' --ex 'file "/tmp/apport-utopic//usr/lib/ubuntu-push-client/signing-
helper"' --ex 'core-file /tmp/apport_core_36z3Va'
GNU gdb (Ubuntu 7.7-0ubuntu3.2) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
The target architecture is assumed to be arm
Reading symbols from /tmp/apport-utopic//usr/lib/ubuntu-push-client/signing-helper...Reading symbols from /tmp/apport-utopic/usr/lib/debug/.bu
ild-id/0d/28e75c6b9149132f8940077618a904989a75ed.debug...done.
done.
BFD: Warning: /tmp/apport_core_36z3Va is truncated: expected core file size >= 19316736, found: 9502720.
[New LWP 15327]
[New LWP 15530]
[New LWP 15321]
Cannot access memory at address 0xb6fa7948
Cannot access memory at address 0xb6fa7944

When running apport-retrace with -o, a new crash file is created with a broken Stacktrace. This makes sense given that the core file is corrupt.

However, I think apport-retrace should do something different here, as you have no idea why the retraced report is bad. Maybe it should write a RetraceFailure key to the report, not write the report at all, or just exit with an error regarding the core file size difference.

Revision history for this message
Brian Murray (brian-murray) wrote :

This problem exists in apport/report.py:

        part_re = re.compile('^\$\d+\s*=\s*-99$', re.MULTILINE)
        parts = part_re.split(out)
        # drop the gdb startup text prior to first separator
        parts.pop(0)
        for part in parts:
            self[value_keys.pop(0)] = part.replace('\n\n', '\n.\n').strip()

Using the relevant core file we see the warning in parts[0].

ipdb> parts[0]
u'The target architecture is assumed to be arm\nBFD: Warning: /tmp/apport_core_U3847R is truncated: expected core file size >= 19316736, found: 9502720.\n[New LWP 15327]\n[New LWP 15530]\n[New LWP 15321]\nCannot access memory at address 0xb6fa7948\nCannot access memory at address 0xb6fa7944\n'

description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

Here is the log file from the Error Tracker retracer trying to retrace the crash:

e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:21:45,422:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Processing.
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:21:45,447:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:swift token: de4d505d664547f086a4ee328ba87631
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:21:45,575:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Decompressing to /tmp/tmpxKHbHR-swift.f158f61e-1eec-11e4-9491-fa163e75317b.oopsid.core
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:21:45,887:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Retracing f158f61e-1eec-11e4-9491-fa163e75317b:swift
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:02,312:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Writing back to Cassandra
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:02,344:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Apport did not return a crash_signature.
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:02,345:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:StacktraceTop:
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:02,345:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:?? ()
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:02,345:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:?? ()
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:02,346:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Could not retrace.
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:03,108:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:Done processing /tmp/tmpxKHbHR-swift.f158f61e-1eec-11e4-9491-fa163e75317b.oopsid
e-t-retracer-app-5bb3de66-d6c1-412d-b8fc-fc855569978d/production-logs/retracer-armhf.log:2014-08-08 11:22:03,108:2852:140155919730432:INFO:root:f158f61e-1eec-11e4-9491-fa163e75317b:swift:swift token: de4d505d664547f086a4ee328ba87631

Changed in apport (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
tags: added: utopic
Martin Pitt (pitti)
Changed in apport (Ubuntu):
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Martin Pitt (pitti) wrote :

Fixed in trunk r2853.

Changed in apport (Ubuntu):
importance: Low → Medium
status: Triaged → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.14.7-0ubuntu1

---------------
apport (2.14.7-0ubuntu1) utopic; urgency=medium

  * New upstream release:
    - Fix interpretation of core dump ulimits: they are in bytes, not KiB.
      (LP: #1346497)
    - apport-retrace: Don't require specifying an operation; default to
      updating the given .crash file. (LP: #1361242)
    - Write report even on UnreportableReasons, so that whoopsie can still
      upload them. (LP: #1360417)
    - apt/dpkg install_packages(): Write a "packages.txt" into sandbox root
      directory to keep track of installed packages and their versions.
      Prerequisite for LP #1352591.
    - apt/dpkg install_packages(): Avoid re-downloading/installing packages
      which are already installed into a permanent sandbox. Prerequisite for
      LP #1352591.
    - sandboxutils.py, make_sandbox(): Drop check for already existing files
      when dynamically resolving libraries and ExecutablePaths; with that,
      these packages would never get updated in a permanent sandbox. The new
      version tracking from above now takes care of that. (LP: #1352591)
    - Fix report.test_add_proc_info test to stop assuming that pid 1 is named
      "init", as one can specify a different one on the kernel command line.
    - report.py, add_gdb_info(): Check for truncated core dumps, and set
      UnreportableReason and raise an IOError on them. Handle this in
      apport-retrace and whoopsie-upload-all to fail properly instead of
      silently producing broken Stacktraces. (LP: #1354571)
 -- Martin Pitt <email address hidden> Fri, 29 Aug 2014 12:38:54 +0200

Changed in apport (Ubuntu):
status: Fix Committed → Fix Released
Changed in apport (Ubuntu Trusty):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Brian Murray (brian-murray)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

We should also see a decrease in the growth rate of the graphite metric missing.missing_sas_amd64.count at least when the fixed version of the package moves to -updates.

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Brian, or anyone else affected,

Accepted apport into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apport/2.14.1-0ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apport (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

This is probably also worth SRU'ing to Precise since we seem to get some invalid core dumps from that release too.

Revision history for this message
Brian Murray (brian-murray) wrote :

bdmurray@clean-trusty-amd64:~/trusty$ PYTHON=python3 test/run report.test_add_gdb_info_damaged
Testing local source tree.
Skipping PEP 8 tests, pep8 is not installed
Skipping pyflakes tests, pyflakes is not installed
--- Testing report.test_add_gdb_info_damaged ---
test_add_gdb_info_damaged (__main__.T)
add_gdb_info() with damaged core dump ... ok

----------------------------------------------------------------------
Ran 1 test in 0.213s

OK
bdmurray@clean-trusty-amd64:~/trusty$ apt-cache policy apport
apport:
  Installed: 2.14.1-0ubuntu3.5
  Candidate: 2.14.1-0ubuntu3.5
  Version table:
 *** 2.14.1-0ubuntu3.5 0
        500 http://192.168.10.1/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.14.1-0ubuntu3.4 0
        500 http://192.168.10.1/ubuntu/ trusty-updates/main amd64 Packages
     2.14.1-0ubuntu3 0
        500 http://192.168.10.1/ubuntu/ trusty/main amd64 Packages

tags: added: verification-done
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

From #ubuntu-release:

09:14 < bdmurray> any SRU team member I'd like to have
                  a 2nd opinion on releasing the fix
                  for bug 1354571 early.
09:14 < ubot2> bug 1354571 in apport (Ubuntu Trusty)
               "apport-retrace ignores warnings from
               gdb" [Medium,Fix committed]
               https://launchpad.net/bugs/1354571
09:47 < infinity> bdmurray: apport looks fine.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.14.1-0ubuntu3.5

---------------
apport (2.14.1-0ubuntu3.5) trusty-proposed; urgency=medium

  [ Martin Pitt ]
  * report.py, add_gdb_info(): Check for truncated core dumps, and set
    UnreportableReason and raise an IOError on them. Handle this in
    apport-retrace and whoopsie-upload-all to fail properly instead of
    silently producing broken Stacktraces. (LP: #1354571)
 -- Brian Murray <email address hidden> Wed, 01 Oct 2014 11:28:57 -0700

Changed in apport (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in apport (Ubuntu Precise):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Brian Murray (brian-murray) wrote :

In Precise the output of gdb contains the following '"/tmp/tmp6nupat/core" is not a core dump: File format not recognized' as opposed to Trusty's message regarding "is truncated: expected core file size" message. It'd probably be best to match on either string.

Revision history for this message
Brian Murray (brian-murray) wrote :

I've updated the debdiff to match either string. Additionally, the test case needs changing as os.truncate() is only available in python3. Switching to the following will properly truncate the CoreDump in python2.

pr['CoreDump'] = pr['CoreDump'][0][:100]

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Brian, or anyone else affected,

Accepted apport into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apport/2.0.1-0ubuntu17.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apport (Ubuntu Precise):
status: Triaged → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

bdmurray@clean-precise-amd64:~/precise$ PYTHON=python2 test/run report.test_add_gdb_info_damaged
Testing local source tree.
--- Testing report.test_add_gdb_info_damaged ---
test_add_gdb_info_damaged (__main__.T)
add_gdb_info() with damaged core dump ... warning: Memory read failed for corefile section, 1048576 bytes at 0x7ffff7bcf000.
ok

----------------------------------------------------------------------
Ran 1 test in 0.151s

OK
bdmurray@clean-precise-amd64:~/precise$ apt-cache policy apport
apport:
  Installed: 2.0.1-0ubuntu17.8
  Candidate: 2.0.1-0ubuntu17.8
  Version table:
 *** 2.0.1-0ubuntu17.8 0
        500 http://192.168.10.1/ubuntu/ precise-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.1-0ubuntu17.7 0
        500 http://192.168.10.1/ubuntu/ precise-updates/main amd64 Packages
     2.0.1-0ubuntu17.6 0
        500 http://192.168.10.1/ubuntu/ precise-security/main amd64 Packages
     2.0.1-0ubuntu5 0
        500 http://192.168.10.1/ubuntu/ precise/main amd64 Packages

I've verified the fix in Precise.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.0.1-0ubuntu17.8

---------------
apport (2.0.1-0ubuntu17.8) precise-proposed; urgency=medium

  [ Martin Pitt ]
  * report.py, add_gdb_info(): Check for truncated core dumps, and set
    UnreportableReason and raise an IOError on them. Handle this in
    apport-retrace and whoopsie-upload-all to fail properly instead of
    silently producing broken Stacktraces. (LP: #1354571)
 -- Brian Murray <email address hidden> Tue, 28 Oct 2014 14:34:44 -0700

Changed in apport (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.