Cinder

ceph: volume build failures in scheduler because "volume service is down"

Bug #1654762 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Invalid
Undecided
Unassigned

Bug Description

Seeing this in the ceph CI job:

http://logs.openstack.org/19/403419/9/gate/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/6eb9afb/logs/screen-c-sch.txt.gz#_2017-01-07_07_19_05_162

The c-vol service is dead at this point:

2017-01-07 07:19:05.162 WARNING cinder.scheduler.host_manager [req-04c81631-6bc1-4db2-a7ed-284fe433838f tempest-TestVolumeBootPattern-672217700] volume service is down. (host: ubuntu-xenial-osic-cloud1-s3700-6526215@ceph)
2017-01-07 07:19:05.162 DEBUG cinder.scheduler.base_filter [req-04c81631-6bc1-4db2-a7ed-284fe433838f tempest-TestVolumeBootPattern-672217700] Starting with 0 host(s) get_filtered_objects /opt/stack/new/cinder/cinder/scheduler/base_filter.py:102

This is hitting quite a bit, check and gate, all failures where this shows up.

Looks like this started happening around 1/2/2017.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22volume%20service%20is%20down%5C%22%20AND%20message%3A%5C%22ceph%5C%22%20AND%20tags%3A%5C%22screen-c-sch.txt%5C%22&from=7d

Tags: ceph rbd
Revision history for this message
Matt Riedemann (mriedem) wrote :

I don't see anything looking suspect around 1/2 in the cinder or devstack-plugin-ceph repos, so it might be something from an upstream packaging update released on 1/2?

Changed in cinder:
status: New → Confirmed
Revision history for this message
Matt Riedemann (mriedem) wrote :

ii librbd1 10.2.3-0ubuntu0.16.04.2 amd64 RADOS block device client library

ii ceph 10.2.3-0ubuntu0.16.04.2 amd64 distributed storage and file system

Revision history for this message
Matt Riedemann (mriedem) wrote :

https://launchpad.net/ubuntu/+source/ceph/10.2.3-0ubuntu0.16.04.2 was published on 2016-11-30 so an upstream package is probably not the issue.

Revision history for this message
Matt Riedemann (mriedem) wrote :

This doesn't appear to be a problem anymore so closing it.

Changed in cinder:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.