BugNomination:+editstatus timeout for bugs with many tasks

This should be marked with Importance 'High' as this is affecting Ubuntu's ability and effectiveness in maintaining the kernel cadence.

Can you get an OOPS number?

(Error ID: OOPS-2113CG81)

and

(Error ID: OOPS-2113AQ86)

Please consider this bug as escalated on behalf of ubuntu engineering team.

After talking with the security team and the kernel team, this issue is causing a lot of problems with the kernel cadence in Ubuntu Engineering. The kernel cadence is a process outlined here: https://wiki.ubuntu.com/Kernel/kernel-sru-workflow

While kernel workflow bugs seem to be ok for the moment, more and more CVE tracking bugs are broken because precise and the oneiric backport kernel were added resulting in I'm told 11 extra tasks to the bugs. Bugs are timing out and can no longer be updated, which ultimately wastes developer resources and reduces parallelism. What this means is that bug reports are inaccurate so they can't be trusted and one person becomes a blocker on manually fixing them. This makes it more difficult for people to work together on the security fixes and security fixes therefore lag (meaning our users aren't getting them as quickly).

I realize this bug is already a distro-escalated Critical bug, but the recent opening of Precise and the newly added backport kernel are aggravating the issue, so I wanted to make sure this was captured.

Fixed in stable r14300 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14300>.

Fixed in stable r14301 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14301>.

The fix in lp:~allenap/launchpad/bugnomination-timeout-bug-874250 that
I have just marked as qa-ok does not completely fix this bug, though
it does improve things. There are two more branches to go.

Marc helped QA this branch. He was not able to get an approval to work, but the OOPS from the final attempt was OOPS-767ea6a157fe7d4ab3d7692ea09a9dda, which shows a large reduction in queries for DistributionSourcePackage (from 270 to 10, iirc). This saves about 0.7s.

Fixed in stable r14326 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14326>.

Recent attempts to approve nominations with the lazy-heat branch in
place have resulted in more OOPSes, but I think we're heading in the
right direction.

OOPS-2fe32d4d689d2e12b2858678ef72e916
OOPS-95d6996501ebcbc29e8b714f82e39269
OOPS-34fdbd3321178b447f549836429f7d91
OOPS-31c20c55f255f58540c8f0be0a97fa06

Remaining issues:

- Email addresses are being loaded individually.

- updateHeat gets called a *lot*.

Recent OOPSes:
  OOPS-8be0f33b472a707e0e15394fda5aa328
  OOPS-c7ce6aef19d617d8ac91aaba5b90c911

Fixed in stable r14638 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14638>.

lp:~allenap/launchpad/bugnomination-timeout-bug-874250-preload-email is okay, but there are still problems with bug heat calculations, and I now think that lp:~allenap/launchpad/bugnomination-timeout-bug-874250-heat-death will actually make things worse. I suspect we need to move all heat calculations out of the web request and accept that there it will lag slightly.

On Sat, Jan 7, 2012 at 4:14 AM, Gavin Panella
<email address hidden> wrote:
> lp:~allenap/launchpad/bugnomination-timeout-bug-874250-preload-email is
> okay, but there are still problems with bug heat calculations, and I now
> think that lp:~allenap/launchpad/bugnomination-timeout-bug-874250-heat-
> death will actually make things worse. I suspect we need to move all
> heat calculations out of the web request and accept that there it will
> lag slightly.

I have initiated a dicussion to disable heat updates on the pillars;
AFAICT this is the primary cause of heat death during updates -
contention on the contexts.

So I'd hold off bringing in an offline system when we may have a much
simpler way forward.

-Rob

The remaining problem here is bug heat updates.

One problem is that there are a lot of heat calculations being done
for bugs with lots of bugtasks and nominations.

Another is that these queries aggregate quite a lot of data and can be
a little slow even when tuned.

The third problem is the one that Rob mentions: these aggregate
results are used to update heavily used rows (e.g. Product,
ProductSeries), and any other activity that also updates these rows
will contend for locks.

Fixed in stable r14760 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14760>.

This is still not fixed for us

when trying to approve the oneiric nomination for 706999, the api timesout returning the attached error

Manually approving the nominations for 706999 and 732628 in a web browser results in the following error message being returned

Timeout error
Sorry, something just went wrong in Launchpad.

We’ve recorded what happened, and we’ll fix it as soon as possible. Apologies for the inconvenience.

Trying again in a couple of minutes might work.

(Error ID: OOPS-6e6b4b29ef635939095468528e92c7ec)

It looks like about half the SQL time is going to BugTask inserts, probably due to the BugSummary triggers not handling the bug well -- it has 73 tasks, an order of magnitude more than this was all designed for.

On Tue, Feb 28, 2012 at 11:59 AM, William Grant <email address hidden> wrote:
> It looks like about half the SQL time is going to BugTask inserts,
> probably due to the BugSummary triggers not handling the bug well -- it
> has 73 tasks, an order of magnitude more than this was all designed for.

FWIW bugsummary was intended to handle such outlier bugs - because we
know we had them in the system. I'd expect it to be a bug, not a
system issue; but suggest whomever works on it reproduce locally and
gather a profile.

Fixed in stable r15227 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15227>.

We has not see this timeout in recent weeks because Lp has a timeout feature flag set for this page. This is the last oops I can find and it is in the old oops site which dates from 2012-06: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-6e6b4b29ef635939095468528e92c7ec . It shows repeated event subscriber actions:
    * Inserts into bugtask
    * inserts into bug activity
    * bug subscription notifications

The repeat info implies 1s or 1/10 of the SQL time could be saved by building the bug notification recipient set once. There are two general calls to find recipients, one for the bug, and the other for the structural subscriptions. I think the next optimisation would be to ensure that the notification recipient set is created once for this action/request because we know that the set cannot mutate after is it first created.

Fixed in stable r16164 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/16164>.

The potential fix is now in production. We are going to watch the oopses for a while to be certain this long standing issue is fixed.

Thanks!

We think the majority of issues related to this bug are fixed. Bug #857109 and Bug #110195 overlap with a few issues in this bug, and we believe they are the proper place to track subsequent fixes that relate to why Lp wants to nominate/target so many packages at once.

This really doesn't appear to be fixed...for example, I'm unable to switch some of the statuses in bug 706999 without hitting a timeout...

For example: OOPS-91d11c991cbabec55e1af4d24a337656

OK, I've opened bug 1076512 with this new issue.

The oops shows you were editing a bugtask, not a bugnomination.

Yeah, sorry about that, I noticed the title of this bug after I wrote that, which is why I opened a new one.