Apport

apport-retrace does not update libraries or executable in a sandbox

Bug #1352591 reported by Brian Murray
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apport
Fix Released
Undecided
Martin Pitt
apport (Ubuntu)
Fix Released
Undecided
Martin Pitt

Bug Description

I was trying to retrace a crash that failed with the retracers in the Ubuntu Error Tracker. I'm using gdb-multiarch from the following PPA (https://launchpad.net/~daisy-pluckers/+archive/ubuntu/daisy-seeds) and apport from bzr revision number 2818. The command follows:

PYTHONPATH=$PYTHONPATH:/srv/daisy.staging.ubuntu.com/prod
uction/apport python /srv/daisy.staging.ubuntu.com/production/apport/bin/apport-retrace ~/4d9d6984-1883-11e4-84c3-fa163e22e467.crash -S /srv/da
isy.staging.ubuntu.com/production/daisy/retracer/config --sandbox-dir /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu\ 14.10/cache-DhmXbj/sandbox -C /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu\ 14.10/cache-DhmXbj/cache -v -g

With that bt full would return:

Thread 1 (LWP 2270):
#0 __libc_do_syscall () at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:44
No locals.
#1 0xb6a3805e in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        _a1 = 0
        _a3tmp = 6
        _a1tmp = 0
        _a3 = 6
        _nametmp = 268
        _a2tmp = 2270
        _a2 = 2270
        _name = 268
        _sys_result = <optimized out>
        pd = 0xb680e000
        pid = 0
        selftid = 2270
#2 0xb6a38d4e in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {1, 3064945215, 3064945144, 3064575611,
              3064342309, 3069180536, 3061899264, 3069180712, 3061899264, 3065241600, 729240, 0, 1, 3199521020, 0, 3069123061, 3069180712,
              1, 5, 0, 3199521440, 3064036264, 1, 3064946200, 1, 10212828, 3199521044, 3069005824, 3199521440, 3069141488, 10212360,
              3061900488}}, sa_flags = 2, sa_restorer = 0x0}
        sigs = {__val = {32, 0 <repeats 31 times>}}
#3 0xb6b892c4 in ?? ()
   from /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu 14.10/cache-DhmXbj/sandbox-corrupt/usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

However, using apport-retrace without the sandbox-dir command creates a more complete backtrace like in http://paste.ubuntu.com/7929061/.

I've tracked this down to the following from needed_runtime_packages in apport/sandboxutils.py:

    # grab as much as we can
    for l in libs:
        if os.path.exists(sandbox + l):
            continue

There is no check to see if the sandbox version of the library is current or not, subsequently the library may be out of date which was causing the retrace to fail.

I've verified this by checking the ctime of the file before and after commenting out the "if os.path.exists()" check.

ls -lch /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu\ 14.10/cache-DhmXbj/sandbox-old/usr/lib/arm-linux-gnueabihf/libstdc++.so.6
lrwxrwxrwx 1 root root 19 Jul 11 02:19 /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu 14.10/cache-DhmXbj/sandbox-old/usr/lib/arm-linux-gnueabihf/libstdc++.so.6 -> libstdc++.so.6.0.20

ls -lch /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu\ 14.10/cache-DhmXbj/sandbox-corrupt/usr/lib/arm-linux-gnueabihf/libstdc++.so.6
lrwxrwxrwx 1 root root 19 Aug 4 22:27 /srv/daisy.staging.ubuntu.com/production/cache/Ubuntu 14.10/cache-DhmXbj/sandbox-old/usr/lib/arm-linux-gnueabihf/libstdc++.so.6 -> libstdc++.so.6.0.20

Revision history for this message
Brian Murray (brian-murray) wrote :

You can also see the change in behavior by running apport-retrace with the -v switch and watching for messages like the following:

dynamically loaded /lib/arm-linux-gnueabihf/libbz2.so.1.0.4 needs package libbz2-1.0, queueing
dynamically loaded /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.20 needs package libstdc++6, queueing
dynamically loaded /lib/arm-linux-gnueabihf/libglib-2.0.so.0.4101.0 needs package libglib2.0-0, queueing

This ends up being a problem for the Error Tracker retracers as they try to use the same sandbox directory and don't get restarted very often.

Brian Murray (brian-murray)
Changed in apport (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Brian Murray (brian-murray) wrote :

There is also not a check to see whether or not the path in InterpreterPath or ExecutablePath is current. From sandboxutils.py:

   # package hooks might reassign Package:, check that we have the originally
    # crashing binary
    for path in ('InterpreterPath', 'ExecutablePath'):
        if path in report and not os.path.exists(sandbox_dir + report[path]):

So we only check for the existence not whether or not it is recent. Subsequently, a retrace with a old executable can fail like so.

Reading symbols from /tmp/apport-utopic//usr/lib/arm-linux-gnueabihf/indicator-network/indicator-network-service...Reading symbols from /tmp/ap
port-utopic/usr/lib/debug/.build-id/5a/851287dfe6af740803822fa635ea6fa5fa5f19.debug...done.
done.
warning: core file may not match specified executable file.

summary: - apport-retrace does not update libraries in a sandbox
+ apport-retrace does not update libraries or executable in a sandbox
Revision history for this message
Martin Pitt (pitti) wrote :

So for permanent sandboxes this requires maintaining a package/version map in the sandbox.

Changed in apport:
status: New → Triaged
Changed in apport (Ubuntu):
status: New → Triaged
Revision history for this message
Martin Pitt (pitti) wrote :

Commits r2847 and r2848 now provide proper version tracking in apt.packaging.install_packages().

Changed in apport:
assignee: nobody → Martin Pitt (pitti)
status: Triaged → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :

With the version tracking in install_packages(), the changes to sandboxutils are now trivial. Done in r2849, so if you run from trunk you can now use this right away.

Please note that for existing sandboxes there is no "packages.txt" yet, thus the next retraces will re-download/install everything which has already been installed (but might be too old). To clean up, it's probably best to purge all the existing sandboxes from the daisy retracers and start over, to get rid of old cruft.

Changed in apport:
status: In Progress → Fix Released
Changed in apport (Ubuntu):
status: Triaged → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.14.7-0ubuntu1

---------------
apport (2.14.7-0ubuntu1) utopic; urgency=medium

  * New upstream release:
    - Fix interpretation of core dump ulimits: they are in bytes, not KiB.
      (LP: #1346497)
    - apport-retrace: Don't require specifying an operation; default to
      updating the given .crash file. (LP: #1361242)
    - Write report even on UnreportableReasons, so that whoopsie can still
      upload them. (LP: #1360417)
    - apt/dpkg install_packages(): Write a "packages.txt" into sandbox root
      directory to keep track of installed packages and their versions.
      Prerequisite for LP #1352591.
    - apt/dpkg install_packages(): Avoid re-downloading/installing packages
      which are already installed into a permanent sandbox. Prerequisite for
      LP #1352591.
    - sandboxutils.py, make_sandbox(): Drop check for already existing files
      when dynamically resolving libraries and ExecutablePaths; with that,
      these packages would never get updated in a permanent sandbox. The new
      version tracking from above now takes care of that. (LP: #1352591)
    - Fix report.test_add_proc_info test to stop assuming that pid 1 is named
      "init", as one can specify a different one on the kernel command line.
    - report.py, add_gdb_info(): Check for truncated core dumps, and set
      UnreportableReason and raise an IOError on them. Handle this in
      apport-retrace and whoopsie-upload-all to fail properly instead of
      silently producing broken Stacktraces. (LP: #1354571)
 -- Martin Pitt <email address hidden> Fri, 29 Aug 2014 12:38:54 +0200

Changed in apport (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.