pycompile hangs when there is more than one python installed

Here's a reproducible test case:

from subprocess import *

p1 = Popen('python2.6 -m py_compile -', bufsize=1, shell=True, stdin=PIPE)
p2 = Popen('python2.7 -m py_compile -', bufsize=1, shell=True, stdin=PIPE)

print 'p1.communicate()'
p1.communicate()
print 'p2.communicate()'
p2.communicate()

## print 'p1.terminate()'
## p1.terminate()
## print 'p2.terminate()'
## p2.terminate()

If you have both Python 2.6 and 2.7 installed, run the test script and watch
as it hangs in p1.communicate(). If you trace this through, it's actually
hanging in a call to os.waitpid() on p1 after its stdin has been closed. If
you comment out p2 (i.e. run only one process), you get no hang. If you use
.terminate() calls, you get no hangs. If you serialize the calls instead of
parallelizing them, it does not hang.

You'll notice that the test doesn't actually write anything to subprocess.
From observation, this happens when the python-pkg-resources package gets
installed.

So what it looks like to me is that if we have more than one subprocess, the
first one doesn't recognize its stdin getting closed and so it never gets an
EOF to break out of the 'while True' loop in py_compile. I'm sure it's caused
fundamentally by some blocking IO, but I haven't quite worked out in my mind
where that's happening. I'll continue to debug what I can today.

I've considered other approaches, including putting each subprocess
communication in its own thread, but I didn't want to go down that radical a
rewrite and the switch to terminate() *seems* to work. I've tested this on a
Maverick system with the Python 2.7 stack enabled and as far as I can tell,
all packages are getting installed and byte-compiled correctly.

Doko tells me in IRC that we still need to support Python 2.5 in Debian, so
ywe can't use .terminate(). :(

I marked this as a critical bug and have a package in my PPA with the fix.
Without this, I basically DDoS'd the Launchpad build farm because every
package just freezes the builder. I can reproduce this in my sbuilds,
e.g. when building net-snmp on a Python 2.7 enabled system.

jwilk says in irc that calling 'cat' in the subprocesses exhibits the same hang:

<jwilk> barry: http://paste.debian.net/82314/ [13:17]

and the problem is:

<jwilk> http://paste.debian.net/82315/ - the 2nd process in keeping the first
        pipe open. [13:23]

A workaround is to add close_fds=True to the Popen constructors.

Branch updated with close_fds=True workaround. Will test in PPA now.

Also: POX will upload a new version to Debian soon, so we should just merge that when it's available.

(For the reference, since paste.debian.net links will expire in a few
days:)

A minimal example to trigger the buggy behaviour is:

from subprocess import *
p1 = Popen('cat', stdin=PIPE)
p2 = Popen('cat', stdin=PIPE)
print 'p1.communicate()'
p1.communicate()
print 'p2.communicate()'
p2.communicate()

A simple ls in /proc shows that p2 inherits fd of the other side of p1's
stdin:

$ ls -l /proc/965[45]/fd/
/proc/9654/fd/:
total 0
lr-x------ 1 jwilk users 64 Aug 5 23:54 0 -> pipe:[213041]
lrwx------ 1 jwilk users 64 Aug 5 23:54 1 -> /dev/pts/3
lrwx------ 1 jwilk users 64 Aug 5 23:54 2 -> /dev/pts/3

/proc/9655/fd/:
total 0
lr-x------ 1 jwilk users 64 Aug 5 23:54 0 -> pipe:[213043]
lrwx------ 1 jwilk users 64 Aug 5 23:54 1 -> /dev/pts/3
lrwx------ 1 jwilk users 64 Aug 5 23:54 2 -> /dev/pts/3
l-wx------ 1 jwilk users 64 Aug 5 23:54 4 -> pipe:[213041]

--
Jakub Wilk

It makes sense then that close_fds=True fixes the problem, and I think that's the right fix.

This bug was fixed in the package python-defaults - 2.6.5-12ubuntu1

---------------
python-defaults (2.6.5-12ubuntu1) maverick; urgency=low

  * Merge from debian unstable. Remaining changes:
    - Package priorities and status.
    - Python-2.6 is the only supported python2.x version.
    - Adjust version numbers in maintainer scripts.

python-defaults (2.6.5-12) unstable; urgency=low

  * pycompile: do not hang if interpreters are not started in the same order
    as they're used (LP: #613675)
  * Add manpage for dh_python2, pycompile and pyclean
  * Standards-Version bumped to 3.9.1 (no other changes required)

python-defaults (2.6.5-11) unstable; urgency=medium

  * Add README.PyDist to python-doc package
  * pycompile: compile public module for the right Python version (really
    closes: 590224)
  * dh_python2: rename --depend to --depends, --recommend to --recommends and
    --suggest to --suggests
 -- Matthias Klose <email address hidden> Sat, 07 Aug 2010 19:46:20 +0200