errors indicates a retraced crash but isn't showing a stack trace

I've looked into this a bit and manually queried the error tracker in an attempt to find the stack trace. One thing of note and that needs fixing is that get_stacktrace_for_bucket in errors/cassandra.py (https://bazaar.launchpad.net/~daisy-pluckers/errors/trunk/view/head:/errors/cassandra.py#L168) only checks 10 crashes. I manually changed my version of cassandra.py to get 300 crashes but I was still unable to find a Stacktrace or ThreadStacktrace for this particular problem.

Additionally, I grep'ed the retracing logs for the two instances (4b5ea464-a941-11e3-b525-2c768aafd08c, d25d6b60-a943-11e3-ae27-2c768aafd08c) of the crash that jibel reported the other day and did not find them being retraced at all.

Additionally stacktrace_cf.get(sas, colums=cols) always returns a NotFoundException for any StacktraceAddressSignature associated with the crash.

We really should at some point create a direct mapping of bucket to Stacktrace (maybe as a new CF called bucket_stacktrace with a key of the bucket, column names of the SASes, and column values of the stacktraces). This going via all the OOPSes to get an SAS to get a Stacktrace was an awful mistake on my part.

Looking around I've managed to find other problems without a stacktrace even after checking 300 crashes.

https://errors.ubuntu.com/problem/23eed577b86c959a28589b8bb1a3bc6ee15b8694
https://errors.ubuntu.com/problem/967af354aa85426ff0d738d1355166dbfc6b41a5
https://errors.ubuntu.com/problem/6366665e5bf8aacf144dbbed48d4662015571f4a

Curiously, most of these first appeared with trusty.

After looking at the retracer code for a while, I think these could be disappearing as a result of the the general exception handling in the retracing process.

https://bazaar.launchpad.net/~daisy-pluckers/daisy/trunk/view/head:/daisy/retracer.py#L538

In the event that there is any exception the .new file will be removed and then if the .new file does not exist we'll go to failed_to_process() and remove it from retracing and delete it from the queue.

The log message associated with .new not being there is '.new did not exist' and:

brian@snakefruit:/srv/app-logs/cassabanana/production-crashdb-retracer-logs$ grep 'new did not exist' * | wc -l
1328

this is a significant number as in the same log file 2455 were successfully retraced.

I've landed a logging change to daisy that might reveal the problem, jibel could you re-run the tests with the workaround disabled?

2 more crashes during an installation of Ubuntu and Ubuntu Gnome today:

https://errors.ubuntu.com/oops/3cde31a8-adf0-11e3-aa3d-2c768aafd08c
https://errors.ubuntu.com/oops/743f35a8-adfe-11e3-8cb6-e4115b0f8a4a

Well, the error message I'd expect to see in the logs hasn't appeared at all. Can you give me a .crash file from one of those failing install tests so I can try manually retracing it?

Crash file from https://jenkins.qa.ubuntu.com/job/ubiquity_ap-ubuntu_devel_daily-test_nonenglish_lvm/ARCH=amd64,label=rabisu/103/

I was able to manually retrace this using canonistack and the following command:

PYTHONPATH=$PYTHONPATH:/srv/daisy.staging.ubuntu.com/production/apport /srv/daisy.staging.ubuntu.com/production/apport/bin/apport-retrace -v -S /srv/daisy.staging.ubuntu.com/production/daisy/retracer/config ~/_usr_lib_ubiquity_bin_ubiquity.0.ubuntu_nonenglish_lvm_amd64_103.crash -o /tmp/updated.crash

Ah-ha! Loading the .crash with apport reveals that:

ipdb> report.crash_signature()
'/usr/lib/ubiquity/bin/ubiquity:11:GtkNode::MatchStringProperty:xpathselect::XPathQueryPart::Matches:SelectNodes:GetNodesThatMatchQuery:Introspect'

Because crash_signature exists the crash is treated as a python one and bucketing then fails.
https://bazaar.launchpad.net/~daisy-pluckers/daisy/trunk/view/head:/daisy/submit.py#L167

This has landed now, please regenerate the crashes and we should be able get a Stacktrace for them now.

jibel resubmited the error - https://errors.ubuntu.com/oops/5fe06f9e-b053-11e3-8067-fa163e707a72 and it can be found in the following bucket

https://errors.ubuntu.com/bucket/?id=/usr/lib/ubiquity/bin/ubiquity:11:GtkNode::MatchStringProperty:xpathselect::XPathQueryPart::Matches:SearchTreeForNode:xpathselect::SelectNodes:GetNodesThatMatchQuery