tripleo

CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found

Bug #1729253 reported by Cédric Jeanneret deactivated
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Martin André
tripleo queens-2

Bug Description

Dear Stackers,

Apparently, there's an issue for the aforementioned scenario in the CI. It fails with the following stacktrace:

2017-11-01 07:54:17 | ==============================
2017-11-01 07:54:17 | Failed 1 tests - output below:
2017-11-01 07:54:17 | ==============================
2017-11-01 07:54:17 |
2017-11-01 07:54:17 | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,volume]
2017-11-01 07:54:17 | ------------------------------------------------------------------------------------------------------------------------------------------------------
2017-11-01 07:54:17 |
2017-11-01 07:54:17 | Captured traceback:
2017-11-01 07:54:17 | ~~~~~~~~~~~~~~~~~~~
2017-11-01 07:54:17 | Traceback (most recent call last):
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
2017-11-01 07:54:17 | return f(self, *func_args, **func_kwargs)
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 100, in test_volume_boot_pattern
2017-11-01 07:54:17 | volume_origin = self._create_volume_from_image()
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 43, in _create_volume_from_image
2017-11-01 07:54:17 | return self.create_volume(name=vol_name, imageRef=img_uuid)
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 234, in create_volume
2017-11-01 07:54:17 | volume['id'], 'available')
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 204, in wait_for_volume_resource_status
2017-11-01 07:54:17 | resource_name=resource_name, resource_id=resource_id)
2017-11-01 07:54:17 | tempest.exceptions.VolumeResourceBuildErrorException: volume d59a4620-f7dc-476c-bb62-6279f4e24b5c failed to build and is in ERROR status

An example:
http://logs.openstack.org/74/513274/11/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/4d603a9/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-01_07_54_17

(original review raising that issue: https://review.openstack.org/#/c/513274/)

Thank you!

Revision history for this message
Alex Schultz (alex-schultz) wrote :

It looks like cinder is missing the backends.

http://logs.openstack.org/18/516418/1/gate/legacy-tripleo-ci-centos-7-containers-multinode/bd8a18e/logs/subnode-2/var/log/containers/cinder/cinder-scheduler.log.txt.gz#_2017-11-01_12_45_44_266

2017-11-01 12:45:44.266 7 ERROR cinder.scheduler.flows.create_volume [req-87efaadc-0de0-472c-9459-246af22c12c2 e93848cd3a6b4eda808b5d6eddc247e9 70bc340995e14c97917ea43ec0757958 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available

Changed in tripleo:
status: New → Triaged
milestone: none → queens-2
importance: Undecided → Critical
tags: added: alert ci containers
Emilien Macchi (emilienm)
summary: - CI issue: legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-
- container fails with volume
+ CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid
+ backend was found
Revision history for this message
Emilien Macchi (emilienm) wrote :

Note that it also affects Cinder's gate:

http://logs.openstack.org/52/514952/6/check/legacy-tempest-dsvm-full-sheepdog/ed1e451/logs/screen-c-sch.txt.gz?level=ERROR

See this logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20%20%20tempest.exceptions.VolumeResourceBuildErrorException%3A%20volume%5C%22

Emilien Macchi (emilienm)
Changed in tripleo:
assignee: nobody → Alan Bishop (alan-bishop)
OpenStack Infra (hudson-openstack)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Alan Bishop (alan-bishop) wrote :

cinder-volume is down:

http://logs.openstack.org/18/516418/1/gate/legacy-tripleo-ci-centos-7-containers-multinode/bd8a18e/logs/subnode-2/var/log/cinder/volume.log.txt.gz#_2017-11-01_12_42_49_271

CI jobs are not containerizing cinder-volume (and cinder-backup) when run under pacemaker, and this may be causing a race condition in db sync.

Fix proposed in https://review.openstack.org/517038

OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Alan Bishop (alan-bishop) → Martin André (mandre)
Revision history for this message
wes hayutin (weshayutin) wrote :

adding promotion blocker because the job is failing in the upstream gates jobs.

tags: added: promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/517038
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=17d34888958c892cbf8fda6ef89e323ed1075076
Submitter: Zuul
Branch: master

commit 17d34888958c892cbf8fda6ef89e323ed1075076
Author: Alan Bishop <email address hidden>
Date: Wed Nov 1 12:14:57 2017 -0400

    Containerize cinder services under pacemaker in CI

    This patch fixes a timing issue with cinder's db sync when
    cinder-manage is containerized but cinder-volume is not.

    Cinder services were containerized in [1], and this patch updates
    the CI jobs so cinder-volume and cinder-backup with pacemaker are
    also containerized.

    [1] https://review.openstack.org/479001

    Change-Id: Ic20af8a9bb24c4d21d1fd71bc65b001aa9c09c7c
    Closes-Bug: #1729253
    Closes-Bug: #1729339

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/518856

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.0.0b2

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/pike)

Change abandoned by Martin André (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/518856

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.