Ironic

Ironic has undocumented depedency on 'fuser' / psmisc package

Bug #1358820 reported by Mike Bayer on 2014-08-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ironic	Fix Released	Medium	Jay Faulkner	Ironic 2015.1.0 "kilo"
	ironic (Ubuntu)	Fix Released	Undecided	Unassigned

Bug Description

This is a fedora 20 VM running on VMWare, a plain checkout at edc9bb038addfc27ffe75ecb81from git.openstack.org/ironic/openstack. The environment has a MySQL database running with "openstack_citest", so MySQL opportunistic tests are also taking place. I also installed "devstack" on this environment separately, not sure if that affects things.

Two tests fail with a "Disk partitioning failed on device" error and another one with something regarding a web service call, it looks like:

$ tox -e py27
py27 create: /opt/stack/ironic/.tox/py27
py27 installdeps: -r/opt/stack/ironic/requirements.txt, -r/opt/stack/ironic/test-requirements.txt
py27 develop-inst: /opt/stack/ironic
py27 runtests: commands[0] | bash -c TESTS_DIR=./ironic/tests/ python setup.py testr --slowest --testr-args=''
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/}
======================================================================
FAIL: ironic.tests.drivers.test_deploy_utils.RealFilePartitioningTestCase.test_different_sizes
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "/opt/stack/ironic/.tox/py27/lib/python2.7/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "ironic/tests/drivers/test_deploy_utils.py", line 709, in test_whole_disk
    image_path='path')
  File "ironic/tests/drivers/test_deploy_utils.py", line 684, in _run_without_root
    return func(*args, **kwargs)
  File "ironic/drivers/modules/deploy_utils.py", line 295, in work_on_disk
    commit=commit)
  File "ironic/drivers/modules/deploy_utils.py", line 129, in make_partitions
    dp.commit()
  File "ironic/common/disk_partitioner.py", line 173, in commit
    % {'device': self._device, 'fuser_err': fuser_err[0]})
InstanceDeployFailure: Disk partitioning failed on device /tmp/tmpPvon12/tmpz8sVkG. Fuser exited with "Cannot stat file /proc/13758/fd/0: Permission denied
Cannot stat file /proc/13758/fd/1: Permission denied
Cannot stat file /proc/13758/fd/2: Permission denied
Cannot stat file /proc/13758/fd/3: Permission denied
Cannot stat file /proc/13758/fd/4: Permission denied
Cannot stat file /proc/13758/fd/5: Permission denied
Cannot stat file /proc/13758/fd/6: Permission denied
Cannot stat file /proc/13758/fd/7: Permission denied
Cannot stat file /proc/13758/fd/8: Permission denied
Cannot stat file /proc/13758/fd/9: Permission denied
Cannot stat file /proc/13758/fd/10: Permission denied
Cannot stat file /proc/13758/fd/11: Permission denied
Cannot stat file /proc/13758/fd/12: Permission denied
Cannot stat file /proc/13758/fd/13: Permission denied
Cannot stat file /proc/13758/fd/14: Permission denied
Cannot stat file /proc/13758/fd/15: Permission denied
Cannot stat file /proc/13758/fd/16: Permission denied
Cannot stat file /proc/13758/fd/17: Permission denied
Cannot stat file /proc/13758/fd/18: Permission denied
". Time out waiting for completion.
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 1123 (+517) tests in 56.250s (+27.409s)
FAILED (id=10, failures=3 (+1), skips=3)
error: testr failed (1)
ERROR: InvocationError: "/usr/bin/bash -c TESTS_DIR=./ironic/tests/ python setup.py testr --slowest --testr-args=''"
py27 runtests: commands[1] | bash -c TESTS_DIR=./ironic/nova/tests/ python setup.py testr --slowest --testr-args=''
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/}
Ran 103 (-1020) tests in 1.045s (-55.140s)
PASSED (id=11)
Slowest Tests
Test id Runtime (s)
--------------------------------------------------------------------------------------------------------------------------------------- -----------
ironic.nova.tests.scheduler.test_ironic_host_manager.IronicHostManagerChangedNodesTestCase.test_consume_identical_instance_from_compute 0.098
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test__start_firewall 0.091
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_spawn_node_trigger_deploy_fail3 0.013
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_spawn_sets_default_ephemeral_device 0.011
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_destroy_unassociate_fail 0.011
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_spawn_destroyed_after_failure 0.011
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_spawn_node_trigger_deploy_fail2 0.010
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_spawn 0.009
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_spawn_node_trigger_deploy_fail 0.009
ironic.nova.tests.virt.ironic.test_driver.IronicDriverTestCase.test_refresh_instance_security_rules 0.009
py27 runtests: commands[2] | bash -c cat .testrepository/1 >>.testrepository/0
______________________________________________________________________________ summary ______________________________________________________________________________
ERROR: py27: commands failed
[classic@localhost ironic]$

Related branches

lp:ubuntu/vivid-proposed/ironic

Revision history for this message

Jay Faulkner (jason-oldos) wrote on 2015-01-13:

Download full text (10.4 KiB)

Failure confirm on Fedora 20; even without the mysql DB setup:

[fedora@jay-fedora20-test-ironicbug ironic]$ tox
py27 develop-inst-noop: /home/fedora/ironic
py27 runtests: PYTHONHASHSEED='3906926702'
py27 runtests: commands[0] | bash -c TESTS_DIR=./ironic/tests/ python setup.py testr --slowest --testr-args=''
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/} --load-list /tmp/tmpv2RHCN
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/} --load-list /tmp/tmpoyfKzR
======================================================================
FAIL: ironic.tests.drivers.test_deploy_utils.RealFilePartitioningTestCase.test_different_sizes
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "ironic/tests/drivers/test_deploy_utils.py", line 761, in test_different_sizes
    image_path='path', **kwargs)
  File "ironic/tests/drivers/test_deploy_utils.py", line 751, in _run_without_root
    return func(*args, **kwargs)
  File "ironic/drivers/modules/deploy_utils.py", line 312, in work_on_disk
    commit=commit)
  File "ironic/drivers/modules/deploy_utils.py", line 135, in make_partitions
    dp.commit()
  File "ironic/common/disk_partitioner.py", line 161, in commit
    timer.start(interval=interval).wait()
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "ironic/openstack/common/loopingcall.py", line 81, in _inner
    self.f(*self.args, **self.kw)
  File "ironic/common/disk_partitioner.py", line 122, in _wait_for_disk_to_become_available
    check_exit_code=[0, 1], run_as_root=True)
  File "ironic/tests/drivers/test_deploy_utils.py", line 748, in fake_execute
    return real_execute(*cmd, **kwargs)
  File "ironic/common/utils.py", line 82, in execute
    result = processutils.execute(*cmd, **kwargs)
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 218, in execute
    env=env_variables)
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 47, in __init__
    subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/pyth...

Failure confirm on Fedora 20; even without the mysql DB setup:

[fedora@jay-fedora20-test-ironicbug ironic]$ tox
py27 develop-inst-noop: /home/fedora/ironic
py27 runtests: PYTHONHASHSEED='3906926702'
py27 runtests: commands[0] | bash -c TESTS_DIR=./ironic/tests/ python setup.py testr --slowest --testr-args=''
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/}  --load-list /tmp/tmpv2RHCN
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ${TESTS_DIR:-./ironic/tests/}  --load-list /tmp/tmpoyfKzR
======================================================================
FAIL: ironic.tests.drivers.test_deploy_utils.RealFilePartitioningTestCase.test_different_sizes
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "ironic/tests/drivers/test_deploy_utils.py", line 776, in test_whole_disk
    image_path='path')
  File "ironic/tests/drivers/test_deploy_utils.py", line 751, in _run_without_root
    return func(*args, **kwargs)
  File "ironic/drivers/modules/deploy_utils.py", line 312, in work_on_disk
    commit=commit)
  File "ironic/drivers/modules/deploy_utils.py", line 135, in make_partitions
    dp.commit()
  File "ironic/common/disk_partitioner.py", line 161, in commit
    timer.start(interval=interval).wait()
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "ironic/openstack/common/loopingcall.py", line 81, in _inner
    self.f(*self.args, **self.kw)
  File "ironic/common/disk_partitioner.py", line 122, in _wait_for_disk_to_become_available
    check_exit_code=[0, 1], run_as_root=True)
  File "ironic/tests/drivers/test_deploy_utils.py", line 748, in fake_execute
    return real_execute(*cmd, **kwargs)
  File "ironic/common/utils.py", line 82, in execute
    result = processutils.execute(*cmd, **kwargs)
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 218, in execute
    env=env_variables)
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 47, in __init__
    subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "ironic/tests/drivers/test_deploy_utils.py", line 776, in test_whole_disk
    image_path='path')
  File "ironic/tests/drivers/test_deploy_utils.py", line 751, in _run_without_root
    return func(*args, **kwargs)
  File "ironic/drivers/modules/deploy_utils.py", line 312, in work_on_disk
    commit=commit)
  File "ironic/drivers/modules/deploy_utils.py", line 135, in make_partitions
    dp.commit()
  File "ironic/common/disk_partitioner.py", line 161, in commit
    timer.start(interval=interval).wait()
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "ironic/openstack/common/loopingcall.py", line 81, in _inner
    self.f(*self.args, **self.kw)
  File "ironic/common/disk_partitioner.py", line 122, in _wait_for_disk_to_become_available
    check_exit_code=[0, 1], run_as_root=True)
  File "ironic/tests/drivers/test_deploy_utils.py", line 748, in fake_execute
    return real_execute(*cmd, **kwargs)
  File "ironic/common/utils.py", line 82, in execute
    result = processutils.execute(*cmd, **kwargs)
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 218, in execute
    env=env_variables)
  File "/home/fedora/ironic/.tox/py27/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 47, in __init__
    subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Ran 1542 tests in 13.299s (-0.539s)
FAILED (id=1, failures=2, skips=14)
error: testr failed (1)
ERROR: InvocationError: "/bin/bash -c TESTS_DIR=./ironic/tests/ python setup.py testr --slowest --testr-args=''"
pep8 develop-inst-noop: /home/fedora/ironic
pep8 runtests: PYTHONHASHSEED='3906926702'
pep8 runtests: commands[0] | flake8
pep8 runtests: commands[1] | bash -c find ironic -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null
______________________________________________ summary ______________________________________________
ERROR:   py27: commands failed
  pep8: commands succeeded

A cursory look shows this function:  https://github.com/openstack/ironic/blob/master/ironic/tests/drivers/test_deploy_utils.py#L746 being the culprit, but IDK why yet.

Changed in ironic:
status:	New → Confirmed
importance:	Undecided → Medium

Revision history for this message

Jay Faulkner (jason-oldos) wrote on 2015-01-13:

As an additional note; this is failing on a shell out to the system. I'm pretty sure we should not be shelling out to the system in unit tests.

summary:	- fuser/permissioning issue occurring during vanilla unit test run + Unit Tests fail due to permissions errors as normal user on Fedora 20
summary:	- Unit Tests fail due to permissions errors as normal user on Fedora 20 + Unit Tests fail on Fedora 20

Revision history for this message

Jay Faulkner (jason-oldos) wrote on 2015-01-13: Re: Unit Tests fail on Fedora 20

This is due to an undocumented dependency on the `psmisc` package for the binary `fuser`.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-13: Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/146948

Changed in ironic:
assignee:	nobody → Jay Faulkner (jason-oldos)
status:	Confirmed → In Progress

Jay Faulkner (jason-oldos) on 2015-01-13

summary:

- Unit Tests fail on Fedora 20
+ Ironic has undocumented depedency on 'fuser' / psmisc package

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-13: Fix merged to ironic (master)

Reviewed: https://review.openstack.org/146948
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=d1f772a8c57b303c2300a22d08361f1006078abf
Submitter: Jenkins
Branch: master

commit d1f772a8c57b303c2300a22d08361f1006078abf
Author: Jay Faulkner <email address hidden>
Date: Tue Jan 13 10:44:13 2015 -0800

Document dependency on `fuser` for pxe driver

This is provided by the package `psmisc` on all three documented
distributions.

Change-Id: Ie0e5b7c20e2ba701d75ed2853190b318c63c0413
Closes-bug: 1358820

Changed in ironic:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2015-02-05

Changed in ironic:
milestone:	none → kilo-2
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-03-31:

This bug was fixed in the package ironic - 2015.1~b3-0ubuntu1

---------------
ironic (2015.1~b3-0ubuntu1) vivid; urgency=medium

  * New upstream release.
    + d/control:
      - Align with upstream dependencies.
      - Add dh-python to build-dependencies.
      - Add psmisc as a dependency. (LP: #1358820)
    + d/p/fix-requirements.patch: Rediffed.
    + d/ironic-conductor.init.in: Fixed typos in LSB headers,
      thanks to JJ Asghar. (LP: #1429962)
-- Chuck Short <email address hidden> Mon, 30 Mar 2015 11:14:57 +0100

Changed in ironic (Ubuntu):
status:	New → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in ironic:
milestone:	kilo-2 → 2015.1.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.