Retracer ARM support

Bug #1044437 reported by Evan
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apport
Fix Released
Undecided
Martin Pitt
Daisy
Fix Released
Low
Martin Pitt

Bug Description

We have a queue full of ARM retraces that we should probably empty, since the ddebs for these packages are likely long gone. However, we should eventually bring an ARM retracer online. This seems fairly straightforward:

[13:54:43] <ev> pitti: while I have you here, might I ask what the historical reason is for not having ARM retracers?
[13:54:47] <ev> was it lack of space on ddebs?
[13:55:23] <pitti> ev: we actually had had some for a while, but since then we just never got around to setting some up
[13:55:35] <pitti> ev: the main blocker is that RT ticket to move to a proper postgresql duplicate DB
[13:55:46] <pitti> ev: as we cannot share the sqlite one between different hosts
[13:55:52] <ev> right
[13:55:58] <pitti> ev: i. e. pretty much the same problem that we have between "your" and "my" retracers now
[13:56:05] <ev> so apport-retrace should be able to handle ARM just fine
[13:56:22] <pitti> yes, it does
[13:56:42] <ogra_> do the backends actually work nowadays ?
[13:56:50] <pitti> ogra_: backends?
[13:57:00] <pitti> the packaging backend is exactly the same for all arches
[13:57:06] ogra_ remembers when NCommander maintained the arm retracers we simply didnt have any
[13:57:28] <ogra_> pitti, doesnt the retracer need to run the package too ?
[13:57:31] <pitti> there is not much that's platform specific, except perhaps the crash analysis from the disassembl
[13:57:33] <pitti> y
[13:58:17] <pitti> ogra_: which package?
[13:58:29] <ogra_> pitti, dunno, any arm packages actually
[13:58:36] <pitti> ogra_: a retracer by and large needs a checkout of apport trunk, python-launchpadlib, and a good deal of CPU and IO capacity
[13:58:52] <pitti> ogra_: sure, but fortunately we have them in an archive? :-)

Steve points out that with gdb-multiarch, we don't even need ARM hardware.

Evan (ev)
Changed in daisy:
importance: Undecided → Low
status: New → Confirmed
Martin Pitt (pitti)
Changed in apport:
assignee: nobody → Martin Pitt (pitti)
status: New → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :

We can do this with really low overhead. We already construct a sandbox with all the target debs unpacked. The only thing we need to do is to additionally unpack the target system's gdb and its few dependencies, install qemu-user on the retracer system, and run the target gdb through qemu-arm. I tried this approach on bug 1070952, with this:

  $ qemu-arm -L /tmp/sandbox usr/bin/gdb --ex 'set debug-file-directory /tmp/sandbox/usr/lib/debug' --ex 'set solib-absolute-prefix /tmp/sandbox' --ex 'file /tmp/sandbox/usr/lib/nux/unity_support_test' --ex 'core-file /tmp/tmpmycDVy'

It doesn't give me an useful stack trace as we unfortunately do not have any debug symbols for armhf on quantal and raring, and the dependencies of the report might also be slightly outdated (as we do not get Launchpad crash reports from stable releases).

There's a couple of details which need to be changed, such as an arch specific sources.list in the retracer's config directory etc., but those are a SMOP, but I believe this approach works and avoids involving a second chroot, lxc, etc.

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 1044437] Re: Retracer ARM support

On Mon, Dec 03, 2012 at 04:12:53PM -0000, Martin Pitt wrote:
> We can do this with really low overhead. We already construct a sandbox
> with all the target debs unpacked. The only thing we need to do is to
> additionally unpack the target system's gdb and its few dependencies,
> install qemu-user on the retracer system, and run the target gdb through
> qemu-arm. I tried this approach on bug 1070952, with this:

Even this should be unnecessary. You should just be able to install
gdb-multiarch instead of gdb, set the target, and use gdb's ability to
retrace stacks of foreign architectures. That should be much more efficient
than running gdb under qemu.

Revision history for this message
Martin Pitt (pitti) wrote :

I tried this as well, but couldn't make it work.

(gdb) set architecture arm
The target architecture is assumed to be arm
(gdb) core-file /tmp/tmpQ9sZpI
"/tmp/tmpQ9sZpI" is not a core dump: File format is ambiguous

gdb on arm has "show architecture" == "arm", and values like "arm-linux-gnueabihf" are not valid.

Revision history for this message
Martin Pitt (pitti) wrote :

"set gnutarget elf32-littlearm" seems to do the trick, thanks to Steve for figuring this out!

Revision history for this message
Martin Pitt (pitti) wrote :

Current Apport trunk now got a range of commits to add optional "architecture" arguments to API like get_file_package() and install_packages(), gdb_command() now uses gdb-multiarch and configures gdb for the target architecture, and the Launchpad backend now supports an "architecture" option as well. This has been rolled out to the Launchpad retracers. Bug 1088428 is an example of a submitted armhf crash report.

Changed in apport:
status: In Progress → Fix Released
Martin Pitt (pitti)
Changed in daisy:
status: Confirmed → In Progress
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Martin Pitt (pitti) wrote :

With lp:daisy r203 we now have all bits in place on the daisy side. process_core.py has an --architecture option, and with apport 2.7 it is able to process reports from any architecture.

What remains is to set up appropriate upstart jobs in the daisy-retracer charm.

Revision history for this message
Martin Pitt (pitti) wrote :

As of revision 20 of lp:~daisy-pluckers/charms/precise/daisy-retracer/trunk, a stock deployment of the error tracker in Juju is now able to process i386, amd64, and armhf reports.

We now have all pieces together, this just needs to be actually deployed into the data center.

Changed in daisy:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.