tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern fails sporadically

Bug #1236524 reported by Peter Portante
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Undecided
Soren Hansen

Bug Description

See: http://logs.openstack.org/83/49583/2/check/check-tempest-devstack-vm-full/6939f07/console.html

2013-10-07 15:32:58.121 | ======================================================================
2013-10-07 15:32:58.121 | FAIL: tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,image,volume]
2013-10-07 15:32:58.121 | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,image,volume]
2013-10-07 15:32:58.121 | ----------------------------------------------------------------------
2013-10-07 15:32:58.122 | _StringException: Empty attachments:
2013-10-07 15:32:58.122 | stderr
2013-10-07 15:32:58.122 | stdout
2013-10-07 15:32:58.122 |
2013-10-07 15:32:58.122 | pythonlogging:'': {{{
2013-10-07 15:32:58.122 | 2013-10-07 15:23:56,063 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:23:56,248 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:23:57,477 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:23:58,176 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:24:00,780 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:24:01,875 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:24:03,818 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.123 | 2013-10-07 15:24:05,150 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.124 | 2013-10-07 15:24:06,484 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.124 | 2013-10-07 15:24:07,659 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.124 | 2013-10-07 15:24:08,805 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.124 | 2013-10-07 15:24:09,893 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.124 | 2013-10-07 15:24:10,984 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.124 | 2013-10-07 15:25:12,106 Connected (version 2.0, client dropbear_2012.55)
2013-10-07 15:32:58.125 | 2013-10-07 15:25:12,479 Authentication (publickey) successful!
2013-10-07 15:32:58.125 | 2013-10-07 15:25:12,631 Connected (version 2.0, client dropbear_2012.55)
2013-10-07 15:32:58.125 | 2013-10-07 15:25:13,002 Authentication (publickey) successful!
2013-10-07 15:32:58.125 | 2013-10-07 15:25:13,019 Secsh channel 1 opened.
2013-10-07 15:32:58.125 | 2013-10-07 15:25:13,321 Connected (version 2.0, client dropbear_2012.55)
2013-10-07 15:32:58.125 | 2013-10-07 15:25:13,728 Authentication (publickey) successful!
2013-10-07 15:32:58.126 | 2013-10-07 15:25:13,742 Secsh channel 1 opened.
2013-10-07 15:32:58.126 | 2013-10-07 15:26:20,200 Connected (version 2.0, client dropbear_2012.55)
2013-10-07 15:32:58.128 | 2013-10-07 15:26:20,560 Authentication (publickey) successful!
2013-10-07 15:32:58.128 | 2013-10-07 15:26:20,696 Connected (version 2.0, client dropbear_2012.55)
2013-10-07 15:32:58.128 | 2013-10-07 15:26:21,029 Authentication (publickey) successful!
2013-10-07 15:32:58.128 | 2013-10-07 15:26:21,067 Secsh channel 1 opened.
2013-10-07 15:32:58.129 | 2013-10-07 15:26:21,309 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.130 | 2013-10-07 15:26:21,404 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.130 | 2013-10-07 15:26:22,453 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.130 | 2013-10-07 15:26:22,496 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.131 | 2013-10-07 15:26:22,651 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.132 | 2013-10-07 15:26:23,750 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.132 | 2013-10-07 15:26:24,838 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.132 | 2013-10-07 15:26:25,934 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.133 | 2013-10-07 15:26:27,020 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.133 | 2013-10-07 15:26:31,250 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.133 | 2013-10-07 15:26:32,403 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.133 | 2013-10-07 15:26:33,496 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.133 | 2013-10-07 15:26:34,588 Starting new HTTP connection (1): 127.0.0.1
2013-10-07 15:32:58.133 | }}}
2013-10-07 15:32:58.133 |
2013-10-07 15:32:58.134 | Traceback (most recent call last):
2013-10-07 15:32:58.134 | File "tempest/scenario/test_volume_boot_pattern.py", line 156, in test_volume_boot_pattern
2013-10-07 15:32:58.134 | ssh_client = self._ssh_to_server(instance_from_snapshot, keypair)
2013-10-07 15:32:58.134 | File "tempest/scenario/test_volume_boot_pattern.py", line 100, in _ssh_to_server
2013-10-07 15:32:58.134 | private_key=keypair.private_key)
2013-10-07 15:32:58.134 | File "tempest/scenario/manager.py", line 455, in get_remote_client
2013-10-07 15:32:58.135 | return RemoteClient(ip, username, pkey=private_key)
2013-10-07 15:32:58.135 | File "tempest/common/utils/linux/remote_client.py", line 47, in __init__
2013-10-07 15:32:58.135 | if not self.ssh_client.test_connection_auth():
2013-10-07 15:32:58.135 | File "tempest/common/ssh.py", line 148, in test_connection_auth
2013-10-07 15:32:58.135 | connection = self._get_ssh_connection()
2013-10-07 15:32:58.135 | File "tempest/common/ssh.py", line 70, in _get_ssh_connection
2013-10-07 15:32:58.136 | time.sleep(bsleep)
2013-10-07 15:32:58.136 | File "/usr/local/lib/python2.7/dist-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
2013-10-07 15:32:58.136 | raise TimeoutException()
2013-10-07 15:32:58.136 | TimeoutException

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/58765

Changed in tempest:
assignee: nobody → Soren Hansen (soren)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.openstack.org/58765
Committed: http://github.com/openstack/tempest/commit/b20cf3a30d42ed2ce0c34e338edf498258dfd721
Submitter: Jenkins
Branch: master

commit b20cf3a30d42ed2ce0c34e338edf498258dfd721
Author: Soren Hansen <email address hidden>
Date: Wed Nov 27 14:39:28 2013 +0100

    Use channel_timeout for SSH connection timeout

    Occasionally, SSH will get wedged so that a connection attempt is stuck
    forever. When this happens, we need Tempest to abort the attempt and try
    again. Currently, the individual connection timeout is set to the
    overall timeout, so there will only ever be one attempt if this happens.
    Using the channel_timeout instead will ensure that multiple connection
    attempts are made even when the connection is wedged.

    Fixes bug 1236524

    Change-Id: Ie8dff41780bbf004cff5c880db202a8ae23a85c1

Changed in tempest:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/72468

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (stable/havana)

Reviewed: https://review.openstack.org/72468
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=50eaa8c80189d0c938f2dbced70df4d14dc2cdfa
Submitter: Jenkins
Branch: stable/havana

commit 50eaa8c80189d0c938f2dbced70df4d14dc2cdfa
Author: Soren Hansen <email address hidden>
Date: Wed Nov 27 14:39:28 2013 +0100

    Use channel_timeout for SSH connection timeout

    Occasionally, SSH will get wedged so that a connection attempt is stuck
    forever. When this happens, we need Tempest to abort the attempt and try
    again. Currently, the individual connection timeout is set to the
    overall timeout, so there will only ever be one attempt if this happens.
    Using the channel_timeout instead will ensure that multiple connection
    attempts are made even when the connection is wedged.

    Fixes bug 1236524

    Change-Id: Ie8dff41780bbf004cff5c880db202a8ae23a85c1
    (cherry picked from commit b20cf3a30d42ed2ce0c34e338edf498258dfd721)

tags: added: in-stable-havana
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.