supermirror_rewritemap.py optimization

Raising importance to critical. Needed for low supermirror latency.

I did a few tests against the launchpad_demo database. I did 20 runs of the supermirror_rewritemap.py script, which gave the following average timings:

    4.62 user, 0.30 system, 6.57 elapsed

Hacking the script to prejoin the branch product and owner gave the following average timings:
    4.59 user, 0.29 system, 6.51 elapsed

And removing the call that actually writes the map file (while leaving all the initialisation) gives the following:
    3.21 user, 0.24 system, 6.46 elapsed

So we are talking about a very marginal improvement here. Of course, these numbers don't tell the story of the work done by the database server: it would be doing a lot less work in the second two cases.

I've submitted a branch to PQM that performs the prejoins, which reduces the script to a single DB query for the run. This should make Stuart happy, but doesn't address the overall performance (it is low enough that it could be run more than once an hour though).

I can confirm jamesh's findings - this query is surpisingly slow!

I'll switch on the existing implementation twice per minute, so no need for a cherry pick.

I mean once per minute, not twice.

Thank you Stuart.

Do you mean surprising slow, really, Stuart?

The change is committed to rocketfuel now as r3826.

I've also got a branch available to merge to production at jamesh/launchpad/rewritemap-prejoins-for-1.80 if you think it is worth cherry picking.

Christian Reis wrote:
> Do you mean surprising slow, really, Stuart?

Yes. I was surprised how slow an apparently simple query ran and that
optimizations did not seem possible. Surprisingly slow.

--
Stuart Bishop <email address hidden> http://www.canonical.com/
Canonical Ltd. http://www.ubuntu.com/