distribution:+bugs timeout searching for bugs by component

Bug #731679 reported by Robert Collins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Robert Collins

Bug Description

Another search timeout, but not high-offset this time.

 29 SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_close ... $INT AND Bug.private = FALSE ORDER BY BugTask.importance DESC, BugTask.id LIMIT $INT OFFSET $INT:
   GET: 29 Robots: 0 Local: 0
     29 https://bugs.launchpad.net/ubuntu/+bugs (Distribution:+bugs)
       OOPS-1892A158, OOPS-1892B1957, OOPS-1892B587, OOPS-1892C2069, OOPS-1892D1181

Branch: launchpad-rev-12515
Revno: 12515
SQL time: 16970 ms
Non-sql time: 698 ms
Total time: 17668 ms
Statement Count: 21

QUERY_STRING: field.searchtext=&orderby=-importance&assignee_option=any&field.assignee=&field.bug_reporter=&fie ... ed=&field.tag=&field.has_cve.used=&field.has_no_package.used=&search=Search&field.status:list=New
REMOTE_ADDR: 91.189.89.30
REQUEST_METHOD: GET
SCRIPT_NAME:
SERVER_NAME: soybean
SERVER_PORT: 9031
SERVER_PROTOCOL: HTTP/1.0
SERVER_SOFTWARE: zope.server.http (HTTP)
assignee_option: any
field.assignee:
field.bug_commenter:
field.bug_contact:
field.bug_reporter:
field.component: 3
field.component-empty-marker: 1
field.has_cve.used:
field.has_no_package.used:
field.has_patch.used:
field.omit_dupes: on
field.omit_dupes.used:
field.searchtext:
field.status: [u'New']
field.status_upstream-empty-marker: 1
field.subscriber:
field.tag:
orderby: -importance
search: Search

18. 2213 14919ms SQL-launchpad-main-slave
SELECT BugTask.assignee,
       BugTask.bug,
       BugTask.bugwatch,
       BugTask.date_assigned,
       BugTask.date_closed,
       BugTask.date_confirmed,
       BugTask.date_fix_committed,
       BugTask.date_fix_released,
       BugTask.date_incomplete,
       BugTask.date_inprogress,
       BugTask.date_left_closed,
       BugTask.date_left_new,
       BugTask.date_triaged,
       BugTask.datecreated,
       BugTask.distribution,
       BugTask.distroseries,
       BugTask.id,
       BugTask.importance,
       BugTask.milestone,
       BugTask.OWNER, BugTask.product,
                      BugTask.productseries,
                      BugTask.sourcepackagename,
                      BugTask.status,
                      BugTask.statusexplanation,
                      BugTask.targetnamecache,
                      Bug.date_last_message,
                      Bug.date_last_updated,
                      Bug.date_made_private,
                      Bug.datecreated,
                      Bug.description,
                      Bug.duplicateof,
                      Bug.heat,
                      Bug.heat_last_updated,
                      Bug.id,
                      Bug.latest_patch_uploaded,
                      Bug.message_count,
                      Bug.name,
                      Bug.number_of_duplicates,
                      Bug.OWNER, Bug.private,
                                 Bug.security_related,
                                 Bug.title,
                                 Bug.users_affected_count,
                                 Bug.users_unaffected_count,
                                 Bug.who_made_private
FROM BugTask
JOIN Bug ON BugTask.bug = Bug.id,
            SourcePackagePublishingHistory,
            SourcePackageRelease
WHERE Bug.id = BugTask.bug
  AND BugTask.distribution = 1
  AND ((BugTask.status = 10))
  AND Bug.duplicateof IS NULL
  AND BugTask.sourcepackagename = SourcePackageRelease.sourcepackagename
  AND SourcePackageRelease.id = SourcePackagePublishingHistory.sourcepackagerelease
  AND SourcePackagePublishingHistory.distroseries = 106
  AND SourcePackagePublishingHistory.archive IN (1,
                                                 534)
  AND SourcePackagePublishingHistory.component IN ('3')
  AND SourcePackagePublishingHistory.status = 2
  AND Bug.private = FALSE
ORDER BY BugTask.importance DESC, BugTask.id LIMIT 76
OFFSET 0

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=0.00..13405.41 rows=76 width=1017) (actual time=16037.847..149141.010 rows=76 loops=1)
   -> Nested Loop (cost=0.00..16735776.75 rows=94881 width=1017) (actual time=16037.845..149140.888 rows=76 loops=1)
         -> Nested Loop (cost=0.00..16476819.05 rows=94881 width=276) (actual time=6269.181..149113.784 rows=761 loops=1)
               -> Nested Loop (cost=0.00..672139.10 rows=31019067 width=280) (actual time=9.959..134427.302 rows=1276160 loops=1)
                     -> Index Scan Backward using bugtask_importance_idx on bugtask (cost=0.00..116380.60 rows=110413 width=276) (actual time=9.909..2039.202 rows=3788 loops=1)
                           Filter: ((distribution = 1) AND (status = 10))
                     -> Index Scan using sourcepackagerelease_sourcepackagename_idx on sourcepackagerelease (cost=0.00..4.77 rows=21 width=8) (actual time=0.508..34.643 rows=337 loops=3788)
                           Index Cond: (sourcepackagerelease.sourcepackagename = bugtask.sourcepackagename)
               -> Index Scan using securesourcepackagepublishinghistory_sourcepackagerelease_idx on sourcepackagepublishinghistory (cost=0.00..0.50 rows=1 width=4) (actual time=0.011..0.011 rows=0 loops=1276160)
                     Index Cond: (sourcepackagepublishinghistory.sourcepackagerelease = sourcepackagerelease.id)
                     Filter: ((sourcepackagepublishinghistory.archive = ANY ('{1,534}'::integer[])) AND (sourcepackagepublishinghistory.distroseries = 106) AND (sourcepackagepublishinghistory.component = 3) AND (sourcepackagepublishinghistory.status = 2))
         -> Index Scan using bug_pkey on bug (cost=0.00..2.72 rows=1 width=741) (actual time=0.033..0.033 rows=0 loops=761)
               Index Cond: (bug.id = bugtask.bug)
               Filter: ((bug.duplicateof IS NULL) AND (NOT bug.private))
 Total runtime: 149141.444 ms
(15 rows)

Tags: qa-ok timeout

Related branches

description: updated
summary: - distribution:+bugs timeout
+ distribution:+bugs timeout searching for bugs by component
Revision history for this message
Robert Collins (lifeless) wrote :

We can generate all spn's needed in this query by
select SourcePackageRelease.sourcepackagename from SourcePackagePublishingHistory, SourcePackageRelease where SourcePackageRelease.id = SourcePackagePublishingHistory.sourcepackagerelease
  AND SourcePackagePublishingHistory.distroseries = 106
  AND SourcePackagePublishingHistory.archive IN (1,
                                                 534)
  AND SourcePackagePublishingHistory.component IN ('3')
  AND SourcePackagePublishingHistory.status = 2;

and its tolerably fast:

(14134 rows)

Time: 342.589 ms

I'll see about a more constrained join

Revision history for this message
Robert Collins (lifeless) wrote :

select bug.id FROM BugTask
JOIN Bug ON BugTask.bug = Bug.id
join (SourcePackagePublishingHistory join
            SourcePackageRelease on SourcePackageRelease.id = SourcePackagePublishingHistory.sourcepackagerelease
  AND SourcePackagePublishingHistory.distroseries = 106
  AND SourcePackagePublishingHistory.archive IN (1,
                                                 534)
  AND SourcePackagePublishingHistory.component IN ('3')
  AND SourcePackagePublishingHistory.status = 2) on BugTask.sourcepackagename = SourcePackageRelease.sourcepackagename
WHERE Bug.id = BugTask.bug
  AND BugTask.distribution = 1
  AND ((BugTask.status = 10))
  AND Bug.duplicateof IS NULL
  AND Bug.private = FALSE
ORDER BY BugTask.importance DESC, BugTask.id LIMIT 76
OFFSET 0

doesn't work - it still things bug is a cheaper initial query.

Revision history for this message
Robert Collins (lifeless) wrote :

with spns as (select sourcepackagename from SourcePackagePublishingHistory join
            SourcePackageRelease on SourcePackageRelease.id = SourcePackagePublishingHistory.sourcepackagerelease
  AND SourcePackagePublishingHistory.distroseries = 106
  AND SourcePackagePublishingHistory.archive IN (1, 534)
  AND SourcePackagePublishingHistory.component IN ('3')
  AND SourcePackagePublishingHistory.status = 2) select bug.id FROM BugTask
JOIN Bug ON BugTask.bug = Bug.id
WHERE Bug.id = BugTask.bug
  AND BugTask.distribution = 1
  AND ((BugTask.status = 10))
  AND Bug.duplicateof IS NULL
  AND Bug.private = FALSE
  and bugtask.sourcepackagename in (select sourcepackagename from spns)
ORDER BY BugTask.importance DESC, BugTask.id LIMIT 76
OFFSET 0

Time: 860.792 ms

Revision history for this message
Robert Collins (lifeless) wrote :

An uncorrelated subquery results in a nested materialization so slower :(/

Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
Changed in launchpad:
assignee: nobody → Robert Collins (lifeless)
milestone: none → 11.04
tags: added: qa-needstesting
Changed in launchpad:
status: Triaged → Fix Committed
Revision history for this message
Robert Collins (lifeless) wrote :

7seconds on staging cold. Win.

tags: added: qa-ok
removed: qa-needstesting
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.