When running the Glance tests repeatedly it looks like glance-api processes are leaking. After a bit of debugging it looks like test_multiprocessing:TestMultiprocessing.test_interrupt_avoids_respawn_storm might be the culprit:
# killall python
python: no process found
# ./run_tests.sh glance.tests.functional.v1.test_multiprocessing:TestMultiprocessing.test_interrupt_avoids_respawn_storm
TestMultiprocessing
test_interrupt_avoids_respawn_storm OK
----------------------------------------------------------------------
Ran 1 test in 4.151s
OK
# ps auwx | grep python
root 7548 0.1 1.2 247788 26480 ? S 18:49 0:00 python /root/glance/bin/glance-api --config-file /tmp/test.9014/etc/api.conf
root 7551 0.0 1.2 247788 26340 ? S 18:49 0:00 python /root/glance/bin/glance-api --config-file /tmp/test.9014/etc/api.conf
root 7694 0.0 0.0 109244 856 pts/0 S+ 18:49 0:00 grep --color=auto python
----
Okay. This is tricky. It looks like the process tree has the following in it when the test executes:
root 9291 1 72 19:03 ? 00:00:00 python /root/glance/
root 9292 9291 0 19:03 ? 00:00:00 logger -t glance-api[9291]
root 9307 9291 0 19:03 ? 00:00:00 python /root/glance/
root 9309 9291 0 19:03 ? 00:00:00 python /root/glance/
root 9316 9281 0 19:03 pts/0 00:00:00 /bin/sh -c ps -fu $USER | grep glance-api
root 9318 9316 0 19:03 pts/0 00:00:00 grep glance-api
The test code creates an array of pids sorted numerically sorted in reverse (largest to smallest) meaning the child pids should be at the top of the list.
The old code stripped off the last two processes leaving us with this: --> ['9309', '9307', '9292']
It should be this (just the child pids): -->['9309', '9307']
The *fix* should be to just take the first two pids in the array. Given the test has 2 workers we should be able to rely on this not changing.