structural subscription bug search timeouts (affects API, Person:+patches, etc) when many subscriptions are held timeouts

 Aggregate (cost=816866.73..816866.74 rows=1 width=0)
   -> Nested Loop (cost=705926.62..816816.29 rows=20177 width=0)
         -> HashAggregate (cost=705926.62..706128.39 rows=20177 width=280)
               -> Append (cost=0.00..704564.67 rows=20177 width=280)
                     -> Nested Loop Left Join (cost=0.00..71.68 rows=3 width=280)
                           Filter: ((public.bugtask.product IS NULL) OR public.product.active)
                           -> Nested Loop (cost=0.00..52.83 rows=3 width=280)
                                 -> Index Scan using bugtask__assignee__idx on bugtask (cost=0.00..32.64 rows=3 width=280)
                                       Index Cond: (assignee = 343381)
                                       Filter: ((status = 10) OR (status = 15) OR (status = 20) OR (status = 21) OR (status = 22) OR (status = 25))
                                 -> Index Scan using bug_pkey on bug (cost=0.00..6.72 rows=1 width=4)
                                       Index Cond: (public.bug.id = public.bugtask.bug)
                                       Filter: ((public.bug.duplicateof IS NULL) AND (public.bug.latest_patch_uploaded IS NOT NULL) AND (NOT public.bug.private))
                           -> Index Scan using product_pkey on product (cost=0.00..6.27 rows=1 width=5)
                                 Index Cond: (public.bugtask.product = public.product.id)
                                 Filter: public.product.activeh, BugTask.date_assigned, BugTask.date_closed, BugTask.date_confirmed, BugTask.date_fix_committed, BugTask.date_fix_released, BugTask.date_incomplete, BugTask.date_inprogress, B -> Nested Loop (cost=0.00..1304.50 rows=4 width=280)sk.datecreated, BugTask.distribution, BugTask.distroseries, BugTask.heat, BugTask.id, BugTask.importance, BugTask.milestone, BugTask.OWNER, BugTask.product, BugTask -> Nested Loop Left Join (cost=0.00..1008.40 rows=44 width=284)k.targetnamecache
                                 Filter: ((public.bugtask.product IS NULL) OR public.product.active)
                                 -> Nested Loop (cost=0.00..993.50 rows=48 width=284)
                                       -> Index Scan using bugsubscription_person_idx on bugsubscription (cost=0.00..255.56 rows=115 width=4)
                                             Index Cond: (person = 343381)
                                       -> Index Scan using bugtask__bug__idx on bugtask (cost=0.00..6.40 rows=1 width=280)
                                             Index Cond: (public.bugtask.bug = bugsubscription.bug)
                                             Filter: ((public.bugtask.status = 10) OR (public.bugtask.status = 15) OR (public.bugtask.status = 20) OR (public.bugtask.status = 21) OR (public.bugtask.status = 22) OR (public.bugtask.status = 25)) Bug ON BugTask.bug = Bug.id;E)) AS BugTaskr = 343381lestone, BugsepackagenameroductgTask.targetnamecacheon, BugTask.distroseries, BugTask.heat, BugTask.id, BugTask.importance, BugTask.milestone, BugTask.OWNER, BugTask.product, BugTask
                                 -> Index Scan using product_pkey on product...

                           -> Nested Loop Left Join (cost=43387.94..702110.46 rows=20168 width=288)
                                 Filter: ((public.bugtask.product IS NULL) OR public.product.active)
                                 -> Nested Loop (cost=43387.94..695244.24 rows=22125 width=280)
                                       -> Bitmap Heap Scan on bug (cost=43387.94..614073.75 rows=22125 width=4)
                                             Recheck Cond: (duplicateof IS NULL)
                                             Filter: ((latest_patch_uploaded IS NOT NULL) AND (NOT private))
                                             -> Bitmap Index Scan on bug_duplicateof_idx (cost=0.00..43382.41 rows=577824 width=0)
                                                   Index Cond: (duplicateof IS NULL)
                                       -> Index Scan using bugtask__bug__idx on bugtask (cost=0.00..3.66 rows=1 width=280)
                                             Index Cond: (public.bugtask.bug = public.bug.id)
                                             Filter: ((public.bugtask.status = 10) OR (public.bugtask.status = 15) OR (public.bugtask.status = 20) OR (public.bugtask.status = 21) OR (public.bugtask.status = 22) OR (public.bugtask.status = 25))
                                 -> Index Scan using product_pkey on product (cost=0.00..0.30 rows=1 width=9)
                                       Index Cond: (public.bugtask.product = public.product.id)
                                       Filter: public.product.active

is the core of the slowness

Structural sub lookup is the culprit:
SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_closed, BugTask.date_confirmed, BugTask.date_fix_committed, BugTask.date_fix_released, BugTask.date_incomplete, BugTask.date_inprogress, BugTask.date_left_closed, BugTask.date_left_new, BugTask.date_triaged, BugTask.datecreated, BugTask.distribution, BugTask.distroseries, BugTask.heat, BugTask.id, BugTask.importance, BugTask.milestone, BugTask.OWNER, BugTask.product, BugTask.productseries, BugTask.sourcepackagename, BugTask.status, BugTask.statusexplanation, BugTask.targetnamecache
         FROM BugTask
         LEFT JOIN Product ON BugTask.product = Product.id
         AND Product.active
         JOIN StructuralSubscription ON BugTask.product = StructuralSubscription.product
         OR BugTask.productseries = StructuralSubscription.productseries
         OR Product.project = StructuralSubscription.project
         AND BugTask.product = Product.id
         OR BugTask.distribution = StructuralSubscription.distribution
         AND (BugTask.sourcepackagename = StructuralSubscription.sourcepackagename
              OR StructuralSubscription.sourcepackagename IS NULL)
         OR BugTask.distroseries = StructuralSubscription.distroseries
         OR BugTask.milestone = StructuralSubscription.milestone, Bug
         WHERE Bug.id = BugTask.bug
           AND ((BugTask.status = 10)
                OR (BugTask.status = 15)
                OR (BugTask.status = 20)
                OR (BugTask.status = 21)
                OR (BugTask.status = 22)
                OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND StructuralSubscription.subscriber = 343381
           AND (Bugtask.product IS NULL
                OR Product.active = TRUE)
           AND Bug.private = FALSE)

->

 Nested Loop (cost=43392.43..702903.59 rows=20168 width=280)
   Join Filter: ((bugtask.product = structuralsubscription.product) OR (bugtask.productseries = structuralsubscription.productseries) OR ((product.project = structuralsubscription.project) AND (bugtask.product = product.id)) OR ((bugtask.distribution = structuralsubscription.distribution) AND ((bugtask.sourcepackagename = structuralsubscription.sourcepackagename) OR (structuralsubscription.sourcepackagename IS NULL))) OR (bugtask.distroseries = structuralsubscription.distroseries) OR (bugtask.milestone = structuralsubscription.milestone))
   -> Bitmap Heap Scan on structuralsubscription (cost=4.48..188.09 rows=1 width=28)
         Recheck Cond: (subscriber = 343381)
         Filter: (((subscriber = 343381) AND (product IS NOT NULL)) OR ((subscriber = 343381) AND (productseries IS NOT NULL)) OR ((subscriber = 343381) AND (project IS NOT NULL)) OR ((subscriber = 343381) AND (distribution IS NOT NULL) AND (sourcepackagename IS NOT NULL)) OR ((subscriber = 343381) AND (distribution IS NOT NULL) AND (sourcepackagename IS NULL)) OR ((subscriber = 343381) AND (distroseries IS NOT NULL)) OR ((subscriber = 343381) AND (milestone IS NOT NULL)))
         -> Bitmap Index Scan on structuralsubscription__subscriber__idx (...

SELECT BugTask.bug FROM BugTask
          JOIN StructuralSubscription ON BugTask.product = StructuralSubscription.product
         OR BugTask.productseries = StructuralSubscription.productseries
         OR (BugTask.distribution = StructuralSubscription.distribution
         AND (BugTask.sourcepackagename = StructuralSubscription.sourcepackagename
              OR StructuralSubscription.sourcepackagename IS NULL))
         OR BugTask.distroseries = StructuralSubscription.distroseries
         OR BugTask.milestone = StructuralSubscription.milestone
         WHERE
            BugTask.status IN (10, 15, 20, 21, 22,25)
           AND StructuralSubscription.subscriber = 343381;

- even though the plan before was suggesting that the bug scan was the culprit, this query itself is the crux of the issue.
Time: 81575.535 ms
2596 rows

This query completes in 7.7 seconds (not great, but better):
SELECT distinct BugTask.id FROM BugTask inner join bug on bug.id=bugtask.bug LEFT JOIN Product ON BugTask.product = Product.id AND Product.active inner join structuralsubscription ss on (
ss.subscriber = 343381
 and
         BugTask.product is NULL = ss.product
         OR BugTask.productseries = ss.productseries
         OR Product.project = ss.project
         AND BugTask.product = Product.id
         OR BugTask.distribution = ss.distribution
         AND (BugTask.sourcepackagename = ss.sourcepackagename
              OR ss.sourcepackagename IS NULL)
         OR BugTask.distroseries = ss.distroseries
         OR BugTask.milestone = ss.milestone
)
         WHERE ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND Bug.private = FALSE;

key changes: remove redundant product.active check - the constrained left join takes care of that. However, I suspect the join is buggy (= and NULL handling)

scratch the last comment about null handling, been staring at this too long. (And note the syntax error where I left a stray is NULL in place)

This clause:
product.project = ss.project and bugtask.product=product.id

- the bugtask.product constraint is strictly redundant.

Bah, clearly tired. Take 2:

SELECT distinct BugTask.bug FROM BugTask inner join bug on bug.id=bugtask.bug LEFT JOIN Product ON BugTask.product = Product.id AND Product.active inner join structuralsubscription ss on (
ss.subscriber = 343381
 and(
         BugTask.product = ss.product
         OR BugTask.productseries = ss.productseries
         OR Product.project = ss.project
         OR BugTask.distribution = ss.distribution
         AND (BugTask.sourcepackagename = ss.sourcepackagename
              OR ss.sourcepackagename IS NULL)
         OR BugTask.distroseries = ss.distroseries
         OR BugTask.milestone = ss.milestone)
)
         WHERE ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND Bug.private = FALSE;

(this restores the subscriber -> all of the other constraints, and blows the time out again).
 HashAggregate (cost=702962.30..703183.55 rows=22125 width=4)

however this:
SELECT distinct BugTask.bug FROM BugTask inner join bug on bug.id=bugtask.bug LEFT JOIN Product ON BugTask.product = Product.id AND Product.active inner join structuralsubscription ss on (
ss.subscriber = 343381
 and (BugTask.distribution = ss.distribution AND (BugTask.sourcepackagename = ss.sourcepackagename
              OR ss.sourcepackagename IS NULL)
  )
)
         WHERE ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND Bug.private = FALSE;

only finds 100 bugs. So we're finding too many bugs with the long structsub clause

starting over - qastaging was missing the new_patches index.

SELECT distinct BugTask.bug FROM BugTask inner join bug on bug.id=bugtask.bug LEFT JOIN Product ON BugTask.product = Product.id AND Product.active inner join structuralsubscription ss on (
ss.subscriber = 343381
 and(
         BugTask.distribution = ss.distribution
         AND (BugTask.sourcepackagename = ss.sourcepackagename
              OR ss.sourcepackagename IS NULL)

         )
)
         WHERE ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND Bug.private = FALSE;

(the minimal case here) is 2 seconds on staging.

SELECT distinct BugTask.bug FROM BugTask inner join bug on bug.id=bugtask.bug LEFT JOIN Product ON BugTask.product = Product.id AND Product.active inner join structuralsubscription ss on (
ss.subscriber = 343381
 and(
         BugTask.product = ss.product
         OR BugTask.productseries = ss.productseries
         OR Product.project = ss.project
         OR BugTask.distribution = ss.distribution
         AND (BugTask.sourcepackagename = ss.sourcepackagename
              OR ss.sourcepackagename IS NULL)
         OR BugTask.distroseries = ss.distroseries
         OR BugTask.milestone = ss.milestone)
)
         WHERE ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND Bug.private = FALSE;
has a cost estimate of 100K and a cold query time exceeding 1 minute :(

88 seconds:

 HashAggregate (cost=112166.83..112397.51 rows=23068 width=4)
   -> Nested Loop (cost=61556.62..112109.16 rows=23068 width=4)
         Join Filter: ((bugtask.product = ss.product) OR (bugtask.productseries = ss.productseries) OR (product.project = ss.project) OR ((bugtask.distribution = ss.distribution) AND ((bugtask.sourcepackagename = ss.sourcepackagename) OR (ss.sourcepackagename IS NULL))) OR (bugtask.distroseries = ss.distroseries) OR (bugtask.milestone = ss.milestone))
         -> Bitmap Heap Scan on structuralsubscription ss (cost=4.50..196.24 rows=1 width=28)
               Recheck Cond: (subscriber = 343381)
               Filter: (((subscriber = 343381) AND (product IS NOT NULL)) OR ((subscriber = 343381) AND (productseries IS NOT NULL)) OR ((subscriber = 343381) AND (project IS NOT NULL)) OR ((subscriber = 343381) AND (distribution IS NOT NULL) AND (sourcepackagename IS NOT NULL)) OR ((subscriber = 343381) AND (distribution IS NOT NULL) AND (sourcepackagename IS NULL)) OR ((subscriber = 343381) AND (distroseries IS NOT NULL)) OR ((subscriber = 343381) AND (milestone IS NOT NULL)))
               -> Bitmap Index Scan on structuralsubscription__subscriber__idx (cost=0.00..4.50 rows=167 width=0)
                     Index Cond: (subscriber = 343381)
         -> Hash Left Join (cost=61552.12..111278.55 rows=23068 width=32)
               Hash Cond: (bugtask.product = product.id)
               -> Hash Join (cost=58241.23..107655.41 rows=23068 width=28)
                     Hash Cond: (bugtask.bug = bug.id)
                     -> Bitmap Heap Scan on bugtask (cost=6436.30..52503.98 rows=311582 width=28)
                           Recheck Cond: ((status = 10) OR (status = 15) OR (status = 20) OR (status = 21) OR (status = 22) OR (status = 25))
                           -> BitmapOr (cost=6436.30..6436.30 rows=348787 width=0)
                                 -> Bitmap Index Scan on bugtask__status__idx (cost=0.00..3610.45 rows=211349 width=0)
                                       Index Cond: (status = 10)
                                 -> Bitmap Index Scan on bugtask__status__idx (cost=0.00..346.62 rows=20171 width=0)
                                       Index Cond: (status = 15)
                                 -> Bitmap Index Scan on bugtask__status__idx (cost=0.00..1085.74 rows=63520 width=0)
                                       Index Cond: (status = 20)
                                 -> Bitmap Index Scan on bugtask__status__idx (cost=0.00..554.49 rows=32287 width=0)
                                       Index Cond: (status = 21)
                                 -> Bitmap Index Scan on bugtask__status__idx (cost=0.00..99.87 rows=5671 width=0)
                                       Index Cond: (status = 22)
                                 -> Bitmap Index Scan on bugtask__status__idx (cost=0.00..271.75 rows=15789 width=0)
                                       Index Cond: (status = 25)
                     -> Hash (cost=51516.58..51516.58 rows=23068 width=4)
                           -> Bitmap Heap Scan on bug (cost=665.02..51516.58 rows=23068 width=4)
                              ...

recasting as a wide filter:
with ss as (select * from structuralsubscription where subscriber=343381)
SELECT distinct BugTask.bug FROM BugTask inner join bug on bug.id=bugtask.bug LEFT JOIN Product ON BugTask.product = Product.id AND Product.active left join ss ss1 on BugTask.product = ss1.product left join ss ss2 on BugTask.productseries = ss2.productseries left join ss ss3 on Product.project = ss3.project left join ss ss4 on BugTask.distribution = ss4.distribution
         AND (BugTask.sourcepackagename = ss4.sourcepackagename
              OR ss4.sourcepackagename IS NULL) left join ss ss5 on BugTask.distroseries = ss5.distroseries left join ss ss6 on BugTask.milestone = ss6.milestone
         WHERE ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25))
           AND Bug.duplicateof IS NULL
           AND Bug.latest_patch_uploaded IS NOT NULL
           AND Bug.private = FALSE
          AND (Bugtask.product IS NULL
                OR Product.active = TRUE)
AND null_count(ARRAY[ss1.id, ss2.id, ss3.id, ss4.id, ss5.id, ss6.id]) < 6;

-> 1.7 seconds. WIN.

Fixed in stable r13356 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13356>.

The tuned query is still 822ms, but the full query the new code generated behaved badly. plan coming up.

 Aggregate (cost=252894.18..252894.19 rows=1 width=0)
   -> Nested Loop (cost=252793.86..252894.11 rows=24 width=0)
         -> HashAggregate (cost=252793.86..252794.10 rows=24 width=280)
               -> Append (cost=0.00..252792.24 rows=24 width=280)
                     -> Nested Loop Left Join (cost=0.00..150.86 rows=4 width=280)
                           Filter: ((public.bugtask.product IS NULL) OR public.product.active)
                           -> Nested Loop (cost=0.00..135.34 rows=4 width=280)
                                 -> Index Scan using bugtask__assignee__idx on bugtask (cost=0.00..22.48 rows=4 width=280)
                                       Index Cond: (assignee = 343381)
                                       Filter: ((status = 10) OR (status = 15) OR (status = 20) OR (status = 21) OR (status = 22) OR (status = 25))
                                 -> Index Scan using bug__new_patches__idx on bug (cost=0.00..28.20 rows=1 width=4)
                                       Index Cond: (public.bug.id = public.bugtask.bug)
                                       Filter: ((NOT public.bug.private) OR (SubPlan 6))
                                       SubPlan 6
                                         -> Unique (cost=24.32..24.33 rows=2 width=4)
                                               -> Sort (cost=24.32..24.33 rows=2 width=4)
                                                     Sort Key: public.bugsubscription.bug
                                                     -> Append (cost=0.00..24.31 rows=2 width=4)
                                                           -> Nested Loop (cost=0.00..16.36 rows=1 width=4)
                                                                 -> Index Scan using bugsubscription_bug_idx on bugsubscription (cost=0.00..4.46 rows=3 width=8)
                                                                       Index Cond: (bug = $1)
                                                                 -> Index Scan using teamparticipation_team_key on teamparticipation (cost=0.00..3.95 rows=1 width=4)
                                                                       Index Cond: ((public.teamparticipation.team = public.bugsubscription.person) AND (public.teamparticipation.person = 6874))
                                                           -> Nested Loop (cost=0.00..7.93 rows=1 width=4)
                                                                 -> Index Scan using bugtask__bug__idx on bugtask (cost=0.00..3.97 rows=1 width=8)
                                                                       Index Cond: (bug = $1)
                                                                 -> Index Scan using teamparticipation_team_key on teamparticipation (cost=0.00..3.95 rows=1 width=4)
                                                                       Index Cond: ((public.teamparticipation.team = public.bugtask.assignee) AND (public.teamparticipation.person = 6874))
                           -> Index Scan using product_pkey on product (cost=0.00..3.87 rows=1 width=5)
                                 Index Cond: (public.bugtask.product = public....

For what is worth the current query is now returning results and not timing out at least for simple queries.

On Wed, Jul 20, 2011 at 8:50 AM, Brian Murray <email address hidden> wrote:
> For what is worth the current query is now returning results and not
> timing out at least for simple queries.

We've just rebuilt the entire database, so we have fresh indices
everywhere. Thats a very hard situation to retain.

The query that failed in qastaging (comment 13) was significantly different than the one that Robert showed in comment 11. In addition to an extraneous WITH clause at the very beginning of the whole query, the problematic structural subscription unioned query has the pertinent differences.

The query as found in qastaging was this. It consistently takes over 35 seconds on staging.

  WITH ss as (SELECT * from StructuralSubscription
 WHERE StructuralSubscription.subscriber = 343381)
 SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_closed, BugTask.date_confirmed, BugTask.date_fix_committed, BugTask.date_fix_released, BugTask.date_incomplete, BugTask.date_inprogress, BugTask.date_left_closed, BugTask.date_left_new, BugTask.date_triaged, BugTask.datecreated, BugTask.distribution, BugTask.distroseries, BugTask.heat, BugTask.id, BugTask.importance, BugTask.milestone, BugTask.owner, BugTask.product, BugTask.productseries, BugTask.sourcepackagename, BugTask.status, BugTask.statusexplanation, BugTask.targetnamecache
 FROM BugTask
 LEFT JOIN Product ON BugTask.product = Product.id AND Product.active, Bug
 WHERE Bug.id = BugTask.bug AND ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25)) AND Bug.duplicateof is NULL AND Bug.latest_patch_uploaded IS NOT NULL AND BugTask.id IN (
 SELECT DISTINCT BugTask.id
 FROM BugTask
 LEFT JOIN Product ON (
 BugTask.product = Product.id AND Product.active)
 LEFT JOIN ss ss1 ON BugTask.product = ss1.product
 LEFT JOIN ss ss2 ON BugTask.productseries = ss2.productseries
 LEFT JOIN ss ss3 ON Product.project = ss3.project
 LEFT JOIN ss ss4 ON (
 BugTask.distribution = ss4.distribution AND (
 BugTask.sourcepackagename = ss4.sourcepackagename OR
 ss4.sourcepackagename IS NULL))
 LEFT JOIN ss ss5 ON BugTask.distroseries = ss5.distroseries
 LEFT JOIN ss ss6 ON BugTask.milestone = ss6.milestone
 WHERE
 NULL_COUNT(ARRAY[ss1.id, ss2.id, ss3.id,
 ss4.id, ss5.id, ss6.id]) < 6)
 AND (Bugtask.product IS NULL OR Product.active = TRUE) AND
 (Bug.private = FALSE OR EXISTS (
 SELECT BugSubscription.bug
 FROM BugSubscription, TeamParticipation
 WHERE TeamParticipation.person = 6874 AND
 TeamParticipation.team = BugSubscription.person AND
 BugSubscription.bug = Bug.id
 UNION
 SELECT BugTask.bug
 FROM BugTask, TeamParticipation
 WHERE TeamParticipation.person = 6874 AND
 TeamParticipation.team = BugTask.assignee AND
 BugTask.bug = Bug.id
 ));

Recasting this to the spelling Robert gave, I get the following, which takes around 1 second on staging.

WITH ss as (SELECT * from StructuralSubscription
 WHERE StructuralSubscription.subscriber = 343381)
 SELECT DISTINCT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_closed, BugTask.date_confirmed, BugTask.date_fix_committed, BugTask.date_fix_released, BugTask.date_incomplete, BugTask.date_inprogress, BugTask.date_left_closed, BugTask.date_left_new, BugTask.date_triaged, BugTask.datecreated, BugTask.distribution, BugTask.distroseries, BugTask.heat, BugTask.id, BugTask.importance, BugTask.milestone, BugTask.owner, BugTa...

Fixed in stable r13572 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13572>.

The page is no speed demon now, but it does load now on qastaging after the cache has been warmed up. "280 queries/external actions issued in 5.78 seconds".

That query generation can be improved further--the permission stuff looks like it is slower than it needs to be, for instance--but for now this nominally fits our goals.