launchpad-buildd

Downloading files from librarian should be asynchronous

Bug #241646 reported by Celso Providelo
2
Affects Status Importance Assigned to Milestone
launchpad-buildd
Triaged
Low
Unassigned

Bug Description

The current code uses urllib2.urlopen() which is blocking and become a problem for huge original tarball (OO, for instance). It sometimes takes longer than 120s (current timeout) in our gigabit network.

The only safe way to solve this problem is to use a asynchronous approach to download the files, similarly to what we have done in bug 211974.

The code surrounding BuildDSlave.ensurePresent() needs to be modified to support this and the recently made fix to access http-authed files in private PPA will also be affected.

Once it's done we will be able to reduce the timeout value drastically, since the lp-buildd will operate almost entirely asynchronously.

See original description
Tags: tech-debt
Diogo Matsubara (matsubara)
description: updated
Revision history for this message
Celso Providelo (cprov) wrote :

There is less pressure on this aspect now that the master side is asynchronous, but it is still desirable.

Changed in launchpad-buildd:
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → Triaged
Revision history for this message
William Grant (wgrant) wrote :

The master side of ensurePresent isn't async; I've observed it blocking all operations on other slaves.

Revision history for this message
Celso Providelo (cprov) wrote :

Willian,

Check lib/lp/buildmaster/manager.py and you will see that all master xmlrpc calls are executed asynchronously.

IBuilder.slave, which originally is a synchronous python xmlrpc wrapper is overridden with a `RecordingSlave` which later is have its calls dispatched via a twisted xmlrpc.Proxy.

Checking builder availability and collecting builder results are still synchronous.

However this specific bug is about the slave side, which is still performing downloads from librarian synchronously. This is more of issue for chroots, which are 50MB files.

Robert Collins (lifeless)
Changed in launchpad-buildd:
importance: Medium → High
Revision history for this message
Robert Collins (lifeless) wrote :

Julian says this still is a problem in theory but not observed in practice.

Changed in launchpad-buildd:
importance: High → Low
Revision history for this message
William Grant (wgrant) wrote :

This no longer blocks scans of other builders in buildd-manager, so it's very low priority.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.