Launchpad itself

Archive.getPublishedBinaries(created_since_date=) is very slow

Bug #1441729 reported by Colin Watson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Colin Watson
lazr.restful
In Progress
High
Colin Watson

Bug Description

To cope with the new world order for ddebs, ddeb-retriever is going to need to use Archive.getPublishedBinaries(status="Published", created_since_date=threshold, ordered=False) to catch up with recent binary publications. Unfortunately, this is very slow right now; I've been trying to use 2015-03-15 as a fairly arbitrary test threshold (not remembering off the top of my head exactly when the qastaging database was last restored), and I can't get that to not time out on qastaging.

Example timeout: OOPS-4a160fb13a4f966d24123318cd4c46eb

I think it would be sensible to index BPPH.datecreated to speed this up.

Related branches

lp:~cjwatson/launchpad/db-index-bpph-datecreated
William Grant: Approve (db)
Stuart Bishop: Pending (db) requested
Diff: 13 lines (+9/-0)
1 file modified
database/schema/patch-2209-53-9.sql (+9/-0)
lp:~cjwatson/lazr.restful/range-factory
LAZR Developers: Pending requested
Diff: 415 lines (+146/-16)
9 files modified
setup.py (+1/-1)
src/lazr/restful/_operation.py (+10/-1)
src/lazr/restful/_resource.py (+16/-3)
src/lazr/restful/declarations.py (+11/-5)
src/lazr/restful/docs/webservice-declarations.txt (+47/-2)
src/lazr/restful/docs/webservice.txt (+51/-3)
src/lazr/restful/fields.py (+2/-1)
src/lazr/restful/interfaces/_fields.py (+5/-0)
src/lazr/restful/interfaces/_rest.py (+3/-0)
lp:~cjwatson/launchpad/getpublishedbinaries-sorting
William Grant: Approve (code)
Diff: 752 lines (+198/-148)
12 files modified
lib/lp/archivepublisher/tests/test_ftparchive.py (+2/-2)
lib/lp/archivepublisher/tests/test_publisher.py (+1/-1)
lib/lp/registry/doc/distroseries.txt (+2/-2)
lib/lp/soyuz/doc/archive.txt (+23/-23)
lib/lp/soyuz/interfaces/archive.py (+23/-4)
lib/lp/soyuz/model/archive.py (+110/-102)
lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py (+3/-3)
lib/lp/soyuz/stories/ppa/xx-ppa-packages.txt (+1/-1)
lib/lp/soyuz/stories/webservice/xx-binary-package-publishing.txt (+1/-1)
lib/lp/soyuz/stories/webservice/xx-source-package-publishing.txt (+1/-1)
lib/lp/soyuz/tests/test_archive.py (+28/-6)
lib/lp/soyuz/tests/test_packagecopyjob.py (+3/-2)
Colin Watson (cjwatson)
tags: added: oops performance soyuz-publish timeout
Changed in launchpad:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Colin Watson (cjwatson)
Revision history for this message
Colin Watson (cjwatson) wrote :

William's suggestion in https://code.launchpad.net/~cjwatson/launchpad/db-index-bpph-datecreated/+merge/255539/comments/635978/+download requires some lazr.restful work to make it possible to slot in StormRangeFactory.

Changed in lazr.restful:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Colin Watson (cjwatson)
Revision history for this message
Colin Watson (cjwatson) wrote :

12:08 <cjwatson> wgrant: I'm giving up on the StormRangeFactory bit for now. I do actually now have something that appears to work - haven't tried with launchpadlib, but the next_collection_link entries have elaborate-looking memos with full dates in them. The
                 tricky bit is that it breaks if you pass the "wrong" parameters, because e.g. BPN isn't in the resultset so SRF can't cope with a resultset ordered by BPN, and at the moment I'm inserting the ...
12:08 <cjwatson> ... range factory in the interface so it doesn't get to be parameter-sensitive.
12:08 <cjwatson> Maybe this is fixable with a DecoratedResultSet or something, but I've run past the end of my timebox for this.
12:10 <cjwatson> I'll just push up something today that implements order_by_date without that.
12:11 <wgrant> cjwatson: I think launchpadlib is a bit magical, since it supports slicing.
12:12 <wgrant> But yeah, SRF for the default sort order is awkward.
12:12 <wgrant> (and pointless, since it can't be sorted by an index)

Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r17425 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17425>.

tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson)
tags: added: qa-ok
removed: qa-needstesting
Colin Watson (cjwatson)
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.