mass importer doesn't handle hung-inactive processes

Bug #589523 reported by Robert Collins
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Invalid
Medium
Unassigned
Ubuntu Distributed Development
Fix Released
Critical
Vincent Ladeuil

Bug Description

the mass importer doesn't handle hung, inactive processes. There is a job system in launchpad itself that can run many jobs, perhaps we should just delete the mass-importer and run things from that job system?

The hung jobs looked like:
1000 8754 4.9 0.1 294968 56244 ? Sl May04 2190:13 /usr/bin/python /srv/package-import.canonical.com/new/scripts/mass_import.py
1000 29086 0.0 0.0 3944 564 ? S May21 0:00 \_ /bin/sh -c import_package.py python-unit
1000 29087 0.0 0.0 142920 25104 ? S May21 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py python-unit
1000 26624 0.0 0.0 3944 564 ? S May30 0:00 \_ /bin/sh -c import_package.py exactimage
1000 26625 0.0 0.1 185336 43380 ? S May30 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py exactimage
1000 26685 0.0 0.0 3944 564 ? S May30 0:00 \_ /bin/sh -c import_package.py otags
1000 26686 0.0 0.1 184852 42940 ? S May30 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py otags
1000 26688 0.0 0.0 3944 560 ? S May30 0:00 \_ /bin/sh -c import_package.py tuareg-mode
1000 26689 0.0 0.1 184844 42988 ? S May30 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py tuareg-mode
1000 26797 0.0 0.0 3944 564 ? S May30 0:00 \_ /bin/sh -c import_package.py exiv2
1000 26798 0.0 0.1 184852 42956 ? S May30 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py exiv2
1000 26804 0.0 0.0 3944 564 ? S May30 0:00 \_ /bin/sh -c import_package.py libunistring
1000 26805 0.0 0.1 184720 42804 ? S May30 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py libunistring
1000 26807 0.0 0.0 3944 564 ? S May30 0:00 \_ /bin/sh -c import_package.py gtk2-engines-qtcurve
1000 26808 0.0 0.1 185364 43268 ? S May30 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py gtk2-engines-qtcurve
1000 26810 0.0 0.0 3944 564 ? S May30 0:00 \_ /bin/sh -c import_package.py ptlib
1000 26811 0.0 0.1 184860 42960 ? S May30 0:00 \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py ptlib
pkg_import@jubany:~$ strace -p 29087
Process 29087 attached - interrupt to quit
write(2, " fixed=\"getA"..., 20480 <unfinished ...>

Tags: lp-code

Related branches

Revision history for this message
Tim Penhey (thumper) wrote :

Robert, why add a task for launchpad-code?

Changed in launchpad-code:
status: New → Incomplete
Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 589523] Re: mass importer doesn't handle hung-inactive processes

Because it seemed the most relevant place to make it visible. I or the
bzr team, or whomever fixes this issue, is going to need some time and
input from the lp-code team about reusing your wonderful import
scheduling system, and ripping out the partial reimplementation James
has done in the package import system: they are not different enough
to justify the parallel existence.

Tim Penhey (thumper)
Changed in launchpad-code:
status: Incomplete → Triaged
importance: Undecided → Medium
Revision history for this message
James Westby (james-w) wrote :

The short fix will be to add a timeout to the supervisor loop, set to a day/week/something.

Further we should move to a more robust system as Rob suggests.

Thanks,

James

Revision history for this message
James Westby (james-w) wrote :

(while adding the timeout it would be good to see if we can record and graph running time)

Changed in udd:
assignee: nobody → canonical-bazaar (canonical-bazaar)
Revision history for this message
John A Meinel (jameinel) wrote :

vila mentioned today that he is working on this. so reassigning and updating status there.

Changed in udd:
assignee: canonical-bazaar (canonical-bazaar) → Vincent Ladeuil (vila)
status: Triaged → In Progress
Vincent Ladeuil (vila)
Changed in udd:
status: In Progress → Fix Released
Changed in launchpad:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.