Distribution:+search (package search) timeouts

Bug #816870 reported by Eliah Kagan
68
This bug affects 10 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
William Grant

Bug Description

Over the past few weeks, I've been getting a lot of timeout errors when searching for Ubuntu packages at https://launchpad.net/ubuntu (typing the package name or other search term into the text box and clicking "Find a Package"). For significant stretches of time (hours), all my searches timeout and package search have been completely nonfunctional (for me) as a result.

I very strongly suspect that this affects only some packages and not others. Both Brad Crittenden and I (see https://answers.launchpad.net/launchpad/+question/165922) are unable to successfully search for linux-tools-common (recently: OOPS-2034EA35), which is a binary package provided by the source package linux. Searching for other binary packages provided by source packages with different names, such as alsa-base (provided by alsa-driver), are currently working perfectly well for me.

24. 255 8942ms SQL-launchpad-main-slave
SELECT COUNT(*)
FROM
  (SELECT DISTINCT DistributionSourcePackageCache.archive, DistributionSourcePackageCache.binpkgdescriptions, DistributionSourcePackageCache.binpkgnames, DistributionSourcePackageCache.binpkgsummaries, DistributionSourcePackageCache.changelog, DistributionSourcePackageCache.distribution, DistributionSourcePackageCache.id, DistributionSourcePackageCache.name, DistributionSourcePackageCache.sourcepackagename
   FROM BinaryPackageBuild, BinaryPackageName, BinaryPackageRelease, DistributionSourcePackageCache, DistroArchSeries, DistroSeries, SourcePackageName, SourcePackageRelease
   WHERE DistroSeries.distribution = 1
     AND DistroSeries.releasestatus != 6
     AND DistroArchSeries.distroseries = DistroSeries.id
     AND BinaryPackageBuild.distro_arch_series = DistroArchSeries.id
     AND BinaryPackageRelease.build = BinaryPackageBuild.id
     AND BinaryPackageBuild.source_package_release = SourcePackageRelease.id
     AND SourcePackageRelease.sourcepackagename = SourcePackageName.id
     AND DistributionSourcePackageCache.sourcepackagename = SourcePackageName.id
     AND DistributionSourcePackageCache.archive IN (1, 534)
     AND BinaryPackageRelease.binarypackagename = BinaryPackageName.id
     AND BinaryPackageName.name = 'linux-tools-common') AS "_tmp";

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate (cost=17040.81..17040.82 rows=1 width=0) (actual time=11349.983..11349.983 rows=1 loops=1)
   -> HashAggregate (cost=17037.90..17039.19 rows=129 width=855) (actual time=11349.963..11349.966 rows=1 loops=1)
         -> Nested Loop (cost=528.54..17035.00 rows=129 width=855) (actual time=208.869..259.804 rows=1119 loops=1)
               -> Hash Join (cost=528.54..16998.55 rows=129 width=859) (actual time=208.795..226.002 rows=1119 loops=1)
                     Hash Cond: (distributionsourcepackagecache.sourcepackagename = sourcepackagerelease.sourcepackagename)
                     -> Bitmap Heap Scan on distributionsourcepackagecache (cost=311.90..16690.06 rows=24153 width=855) (actual time=7.915..105.385 rows=24163 loops=1)
                           Recheck Cond: (archive = ANY ('{1,534}'::integer[]))
                           -> Bitmap Index Scan on distributionsourcepackagecache__archive__idx (cost=0.00..305.86 rows=24153 width=0) (actual time=5.029..5.029 rows=25147 loops=1)
                                 Index Cond: (archive = ANY ('{1,534}'::integer[]))
                     -> Hash (cost=215.92..215.92 rows=57 width=4) (actual time=92.959..92.959 rows=1119 loops=1)
                           -> Nested Loop (cost=6.67..215.92 rows=57 width=4) (actual time=0.866..91.618 rows=1119 loops=1)
                                 -> Hash Join (cost=6.67..189.62 rows=57 width=4) (actual time=0.758..66.512 rows=1119 loops=1)
                                       Hash Cond: (distroarchseries.distroseries = distroseries.id)
                                       -> Hash Join (cost=2.98..185.06 rows=78 width=8) (actual time=0.667..64.694 rows=1134 loops=1)
                                             Hash Cond: (binarypackagebuild.distro_arch_series = distroarchseries.id)
                                             -> Nested Loop (cost=0.00..181.01 rows=78 width=8) (actual time=0.488..62.445 rows=1134 loops=1)
                                                   -> Nested Loop (cost=0.00..148.87 rows=78 width=4) (actual time=0.377..26.758 rows=1134 loops=1)
                                                         -> Index Scan using binarypackagename_name_key on binarypackagename (cost=0.00..3.89 rows=1 width=4) (actual time=0.198..0.200 rows=1 loops=1)
                                                               Index Cond: (name = 'linux-tools-common'::text)
                                                         -> Index Scan using binarypackagerelease_binarypackagename_key on binarypackagerelease (cost=0.00..144.01 rows=78 width=8) (actual time=0.172..24.991 rows=1134 loops=1)
                                                               Index Cond: (binarypackagerelease.binarypackagename = binarypackagename.id)
                                                   -> Index Scan using binarypackagebuild_pkey on binarypackagebuild (cost=0.00..0.40 rows=1 width=12) (actual time=0.028..0.029 rows=1 loops=1134)
                                                         Index Cond: (binarypackagebuild.id = binarypackagerelease.build)
                                             -> Hash (cost=1.88..1.88 rows=88 width=8) (actual time=0.155..0.155 rows=88 loops=1)
                                                   -> Seq Scan on distroarchseries (cost=0.00..1.88 rows=88 width=8) (actual time=0.026..0.083 rows=88 loops=1)
                                       -> Hash (cost=3.55..3.55 rows=11 width=4) (actual time=0.072..0.072 rows=7 loops=1)
                                             -> Seq Scan on distroseries (cost=0.00..3.55 rows=11 width=4) (actual time=0.026..0.065 rows=7 loops=1)
                                                   Filter: ((releasestatus <> 6) AND (distribution = 1))
                                 -> Index Scan using sourcepackagerelease_pkey on sourcepackagerelease (cost=0.00..0.45 rows=1 width=8) (actual time=0.018..0.019 rows=1 loops=1119)
                                       Index Cond: (sourcepackagerelease.id = binarypackagebuild.source_package_release)
               -> Index Scan using sourcepackagename_pkey on sourcepackagename (cost=0.00..0.27 rows=1 width=4) (actual time=0.015..0.020 rows=1 loops=1119)
                     Index Cond: (sourcepackagename.id = distributionsourcepackagecache.sourcepackagename)
 Total runtime: 11350.728 ms

Related branches

Benji York (benji)
Changed in launchpad:
status: New → Triaged
importance: Undecided → Critical
summary: - package searches frequently time out
+ Distribution:+search (package search) timeouts
description: updated
description: updated
description: updated
tags: added: timeout
tags: added: critical-analysis
Revision history for this message
Scott Kitterman (kitterman) wrote :

(Error ID: OOPS-4e83b78b474fd43d28a1afca4ac0ad68)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

OOPS-fccf3ffb05288c02d9351b7b9328a575 happens every time for me. Is it possible to increase the timeout just for this page while this is being debugged?

Revision history for this message
Giorgio Wicklein (giowck) wrote :

Trying to search for eclipse:
https://launchpad.net/ubuntu/+search?text=eclipse

timeout error

Revision history for this message
William Grant (wgrant) wrote :

OOPS-0a167283b8de82115790872c75a0f611 is a recent example. This page has a lot of issues, of which I'll describe just a few:

Firstly, DistributionPackageSearchView.initialize() calls Distribution.has_published_binaries, which should be sub-ms but takes >1s on production. https://pastebin.canonical.com/67456/ reveals that it's doing a merge join from DistroSeries to DistroArchSeries, using distroseries__distribution__id__key. This will, of course, cause older DistroSeries to appear first. So when it subsequently does a nested loop to find active publications for the distroseries, the first few iterations find none, since all the publications are obsolete. This wouldn't be a problem if the BPPH lookup was indexed, but the query doesn't specify the archive, so it can't use binarypackagepublishinghistory__archive__distroarchseries__stat. If I restrict to just the primary archive, the index used and the query is lightning fast. If I restrict the archive to primary+partner, the index is no longer used.

Secondly, and the primary cause of timeouts on production today, DistributionPackageSearchView.has_exact_matches says "return self.exact_matches.count() > 0", which is going to be terribly slow. It in fact means to say "return not self.exact_matches.is_empty()".

Thirdly, the search queries are pretty bad. Distribution.searchBinaries with exact_match=True creates a monstrous join across 8 tables to filter DistributionSourcePackageCache on archive and binary name, when it has archive and binary name columns which eliminate the need for any joins at all. With exact_match=False it's probably even worse.

William Grant (wgrant)
Changed in launchpad:
assignee: nobody → William Grant (wgrant)
status: Triaged → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant)
tags: added: qa-ok
removed: qa-needstesting
Ian Booth (wallyworld)
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.