fetching ppa stats with a ppa.getPublishedBinaries() loop is too slow

Bug #774259 reported by Fabien Tassin
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Triaged
High
Unassigned

Bug Description

When I get the stats for the chromium PPAs with this code, it's way too slow..

        for bin in ppa.getPublishedBinaries():
            d = bin.getDailyDownloadTotals()

Polled 89445 rows for chromium-daily/ppa (20361, 22% empty) in 3:57:31.858862
Polled 10545 rows for chromium-daily/dev (3722, 35% empty) in 0:17:39.911383
Polled 9233 rows for chromium-daily/beta (1399, 15% empty) in 0:14:39.380236
Polled 4543 rows for chromium-daily/stable (1082, 23% empty) in 0:07:12.078046

So basically, it takes 4h30m to get the stats for the 4 PPAs, and that's increasing.

Somehow skipping those empty rows would spare 24% of the requests.

Also, to do an incremental update of the stats, it would be nice to be able to pass a date filter to getPublishedBinaries(). Today, the only filter is a package name and there's no API to get the binary names beforehand (allowing me to fake my own date filter).
The current scheme reports the binary packages one by one, but iterates over all dates before moving to the next package.

Tags: api easy
Changed in launchpad:
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Fabien Tassin (fta) wrote :

to backup my request, here is an example of what i'm doing with those stats:

http://people.ubuntu.com/~fta/chromium/stats/stats.html

See also http://ftagada.wordpress.com/2011/04/29/how-many-chromium-users-in-ubuntu/

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Given this has a performance impact on the rest of LP, and it's pretty easy to fix, I'm going to mark it high.

Changed in launchpad:
importance: Low → High
tags: added: easy
tags: added: api
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.