/people/*/+translations times out for prolific translators

Bug #2497 reported by Stuart Bishop
4
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Unassigned

Bug Description

The queries being done to render /people/*/+translations cannot be done
quickly for users who have done a lot of translations. For example,
https://launchpad.net/people/menthos/+translations

The first one that is attempted is:

SELECT COUNT(DISTINCT POFile.id) FROM POSubmission, POMsgSet, POFile WHERE
POSubmission.person = 5778 AND POSubmission.pomsgset = POMsgSet.id AND POMsgSet.pofile = POFile.id;

This query takes 3 minutes even when rewritten as the more efficient:

SELECT COUNT(DISTINCT pofile) FROM POSubmission, POMsgSet WHERE
POSubmission.person = 5778 AND POSubmission.pomsgset = POMsgSet.id;

(and then it issues it a second time!)

By forcing a particular query plan I've managed to get it down to 30
seconds, but this is still more than an order of magnitude too slow.

Other queries on the page may also be problematic - I haven't profiled them.

OOPS-118C134

Changed in rosetta:
assignee: nobody → carlos
Changed in rosetta:
status: New → Accepted
Revision history for this message
Dafydd Harries (daf) wrote :

The URL mentioned is obsolete. I assume this is the updated one:

https://launchpad.net/people/menthos/+translations

That times out (today's OOPS-A208). Here's one that works:

https://launchpad.net/people/zkrynicki/+translations

Is this just a matter of batching or just truncating the results?

It would be good to sort by last activity and list it on the page.

Revision history for this message
Dafydd Harries (daf) wrote :

Perhaps we could even make this use the Karma table, as this page and the Karma log both come from an "activity log" sort of angle.

Revision history for this message
Dafydd Harries (daf) wrote :

It's not clear what the use cases for this page are. The ones I can think of are:

 * I'm looking at my own translation activity page to satisfy my ego
 * I'm looking at somebody else's translation activity page to see if somebody else is still active or has disappeared

description: updated
Revision history for this message
Christian Reis (kiko) wrote :

I want this to stop oopsing.

Changed in rosetta:
assignee: carlos → kiko
Revision history for this message
Christian Reis (kiko) wrote :

But I can't seem to find a way out of this rathole. The problem is that POSubmission is just too massive, and using it to find out who has contributed to a template is too expensive. I think that a fix would be to cache this information in a TranslationContribution table, which would link people to pofiles (and which could contain some interesting information such as first/last_contribution_time. How do people feel about this? What are the gotchas?

description: updated
Christian Reis (kiko)
Changed in rosetta:
importance: Medium → Critical
Revision history for this message
Diogo Matsubara (matsubara) wrote :

A solution was discussed here:
https://lists.ubuntu.com/mailman/private/launchpad/2006-August/010391.html

The next step is now to Stuart create a cache table based on trigger mentioned in that thread.

Changed in rosetta:
assignee: kiko → stub
Revision history for this message
Stuart Bishop (stub) wrote :

The materialized view is done with tests and available at sftp://devpad.canonical.com/home/warthogs/archives/stub/launchpad/bug-2497

Changed in rosetta:
assignee: stub → nobody
Changed in rosetta:
assignee: nobody → kiko
Christian Reis (kiko)
Changed in rosetta:
status: Confirmed → In Progress
Revision history for this message
Christian Reis (kiko) wrote :

Patch that uses POFileTranslator is now up for review.

Revision history for this message
Christian Reis (kiko) wrote :

The patch is missing code to handle merging of the POFileTranslator table, which is necessary for merging.

Changed in rosetta:
assignee: kiko → stub
Revision history for this message
Stuart Bishop (stub) wrote :

bzr+ssh://devpad.canonical.com/code/stub/launchpad/bug-2497

*punt*

Changed in rosetta:
assignee: stub → kiko
Revision history for this message
Christian Reis (kiko) wrote :

Landed in RF 4232.

Changed in rosetta:
status: In Progress → Fix Committed
Revision history for this message
Christian Reis (kiko) wrote :

The hard timeouts we were seeing are now gone. There are a few soft timeouts, but Matsubara has agreed to prioritize them when they become an issue again. Enjoy the new UI!

Changed in rosetta:
status: Fix Committed → Fix Released
Revision history for this message
Carlos Perelló Marín (carlos) wrote :

This is still an issue:

https://translations.beta.launchpad.net/~yannick-marchegay/+translations

After trying to get it three or four times, I got the page loaded so it's not as bad as before, but it's something we should fix

Changed in rosetta:
importance: Critical → High
status: Fix Released → Confirmed
Revision history for this message
Данило Шеган (danilo) wrote :

This seems to not time-out anymore, even if it takes a while to render the page (with Rosetta DB changes landed).

Revision history for this message
Christian Reis (kiko) wrote :

This has grown to time out again, and is increasingly timing out, in fact.

Changed in rosetta:
assignee: kiko → nobody
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

The Phase-2 schema optimization made a vast difference here, reducing timeouts by almost 90%.

More things we can do here:
 * Keep "translations to English" out of POFileTranslator, and remove the check for those from the slow query.
 * Upgrade the database server. There's a new optimization we'd be likely to benefit from with the above changes.
 * Index on (person, date_last_touched).

Revision history for this message
Данило Шеган (danilo) wrote :

We now seem to be seeing this time out only on staging, thus reducing the importance to 'Medium'.

Changed in rosetta:
importance: High → Medium
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

This doesn't seem to be a problem anymore.

Changed in rosetta:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.