100s/1000s breaks +bug and +subscribe with SQL timeout

Bug #471974 reported by Michael Hudson-Doyle
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Graham Binns

Bug Description

This bug:

https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/417842

is timing out repeatedly, but the oopses (e.g. https://lp-oops.canonical.com/oops.py/?oopsid=1402D3725) only show a moderate amount of SQL time. It seems better on lpnet rather than edge -- but lpnet has loads of appservers now so maybe it's an appserver CPU thing. Being logged in or not doesn't seem to matter. Here's a ++oops++ for the page rendering successfully on lpnet:

https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1403G78

There are some quite large jumps between the query times: for example, 2.2 seconds between queries 13 and 14, another 2.6 between 14 and 15, 1.9 between 50 and 51, ... I have no idea what stage of rendering these correspond to.

The bug has a large number of duplicates (300+) but creating a bug with lots of duplicates locally isn't enough to reproduce the problem.

Related branches

Deryck Hodge (deryck)
Changed in malone:
status: New → Triaged
importance: Undecided → High
tags: added: bug-page
Revision history for this message
Graham Binns (gmb) wrote :

I've seen this a couple of times with bug 131679, too (OOPS-1404EA298). In this case, it's more obvious to me that it's an dupe subscription problem. There's a jump between statement 137 (fetching the BugSubscriptions from dupes) and statement 138 (footling around with ValidPersonCache) of nearly 12 seconds.

Bug 131679 has 292 dupes which bring a grand total of 1061 extra subscriptions with them. The trouble is that the results set from the dupe subscription query get iterated over more than once in the code before they get displayed (see Bug.getSubscriptionsFromDuplicates(), for example).

I'm not entirely clear why all the subscribers' details are getting loaded with each page load, since we don't appear to display them any more unless you're viewing the page with a JS-enabled browser, in which case the subscribers list is loaded asynchronously anyway.

The simple way to solve this problem, then, is to find the piece of code that's getting the subscribers list unnecessarily and excise it.

Revision history for this message
Graham Binns (gmb) wrote :

So, glancing at this before EoDing, I think the offending code is this statement in bugtask-index.pt:

 <script type="text/javascript" tal:content="string:var subscriber_ids = ${view/subscriber_ids_js};" />

That statement causes the view load all the bug's subscribers for the sake of creating a JS structure containing all the subscriber IDs, which pretty much negates the advantage we have in loading the subscribers asynchronously.

I'm too knackered to think of a solution tonight; I'll look at it again in the morning, though suggested solutions are welcome in the mean time...

Changed in malone:
assignee: nobody → Graham Binns (gmb)
milestone: none → 3.1.11
Graham Binns (gmb)
Changed in malone:
status: Triaged → In Progress
Revision history for this message
Paul Sladen (sladen) wrote :

Also occurring on:

  https://bugs.launchpad.net/ubuntu/+source/seahorse-plugins/+bug/429322

(a user just complained that they were unable to unsubscribe from the bug report, as the page to do so is timing out).

Revision history for this message
Paul Sladen (sladen) wrote :

< gmb> sladen: Jaysus. That bug 429322 has 2030 subscribers. No wonder it times out.
< gmb> And all but 51 of those are from duplicates.

Revision history for this message
Paul Sladen (sladen) wrote :

It would appear that the submission of /+subscribe also times out... although I have no idea why it should/would.

summary: - bug timing out without much sql time
+ 100s/1000s breaks +bug and +subscribe with SQL timeout
Revision history for this message
Graham Binns (gmb) wrote :

I've filed bug 481437 about doing the first part of this work.

Revision history for this message
Diogo Matsubara (matsubara) wrote : Bug fixed by a commit
Changed in malone:
status: In Progress → Fix Committed
Graham Binns (gmb)
Changed in malone:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.