SPDK failing with KeyError: 'storage_protocol'

Bug #1896764 reported by Abdallah Yasin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Eric Harney

Bug Description

Deploying Cinder with SPDK using confs at [1] causes the cinder volumes to have the error at [2] in the cinder volume (c-vol) service, which cause the cinder to fail in creating volumes.

[1]
[DEFAULT]
cinder_internal_tenant_user_id = 5d9054f9d56849a69e12775c9121ceda
cinder_internal_tenant_project_id = 0378334ce9574263b2d3d9a4030e6747
graceful_shutdown_timeout = 5
glance_api_servers = http://10.7.12.119/image
osapi_volume_workers = 3
logging_exception_prefix = ERROR %(name)s %(instance)s
logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}
transport_url = rabbit://stackrabbit:password@10.7.12.119:5672/
enabled_backends = spdk-1,spdk-2
my_ip = 10.7.12.119
state_path = /opt/stack/data/cinder
osapi_volume_listen = 0.0.0.0
osapi_volume_extension = cinder.api.contrib.standard_extensions
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_config = /etc/cinder/api-paste.ini
target_helper = spdk-nvmeof
debug = True

[spdk-1]
spdk_rpc_ip = localhost
spdk_rpc_port = 8000
spdk_rpc_username = user
spdk_rpc_password = password
target_ip_address = 1.1.1.1
target_port = 4260
target_protocol = nvmet_rdma
target_helper = spdk-nvmeof
target_prefix = nqn.2014-08.org.spdk
volume_driver = cinder.volume.drivers.spdk.SPDKDriver
volume_backend_name = SPDK

[2]
Sep 23 09:42:30 cinder-volume[1126869]: DEBUG cinder.volume.drivers.spdk [None req-3363ec01-afc8-40da-b0f4-603389384810 None None] SPDK lvs name: lvs, total space: 260.5703125, free space: 260.5703125 {{(pid=1126869) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/spdk.py:126}}
Sep 23 09:42:30 cinder-volume[1126869]: INFO cinder.volume.manager [None req-3363ec01-afc8-40da-b0f4-603389384810 None None] Abd1: volume_stats: {'pools': [{'volume_backend_name': 'SPDK', 'vendor_name': 'Open Source', 'driver_version': '1.0.0', 'storage_protocol': 'NVMe-oF', 'total_capacity_gb': 260.5703125, 'free_capacity_gb': 260.5703125, 'pool_name': 'lvs', 'allocated_capacity_gb': 0}]}
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service [None req-3363ec01-afc8-40da-b0f4-603389384810 None None] Error starting thread.: KeyError: 'storage_protocol'
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service Traceback (most recent call last):
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service File "/usr/local/lib/python3.8/dist-packages/oslo_service/service.py", line 807, in run_service
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service service.start()
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/service.py", line 228, in start
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service self.manager.init_host(added_to_cluster=self.added_to_cluster,
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/manager.py", line 443, in init_host
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/manager.py", line 574, in _init_host
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service self.publish_service_capabilities(ctxt)
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/manager.py", line 2694, in publish_service_capabilities
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service self._report_driver_status(context)
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/manager.py", line 2631, in _report_driver_status
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service if volume_stats['storage_protocol'] in [
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service KeyError: 'storage_protocol'
Sep 23 09:42:30 cinder-volume[1126869]: ERROR oslo_service.service

Eric Harney (eharney)
Changed in cinder:
assignee: nobody → Eric Harney (eharney)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/753719

Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/753752

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/753719
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=19a128c31b3ea80dde590abf731a54947d34e6ba
Submitter: Zuul
Branch: master

commit 19a128c31b3ea80dde590abf731a54947d34e6ba
Author: Eric Harney <email address hidden>
Date: Wed Sep 23 10:44:15 2020 -0400

    Fix volume_stats storage_protocol usage in vol mgr

    Commit
     afcaf0b9 Add support volume local cache

    added usage of volume_stats['storage_protocol'] in
    the volume manager. Some drivers only supply this in
    volume_stats['pools']['storage_protocol'].

    This patch fixes the volume manager failure but means
    that such drivers won't use the local cache.

    Change-Id: I5329e02cb15babed6b63de9faae3440583b8d7a1
    Closes-Bug: #1896764

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (master)

Reviewed: https://review.opendev.org/753752
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=df575898a92f0cd9035ae10f77c3151b35c7976f
Submitter: Zuul
Branch: master

commit df575898a92f0cd9035ae10f77c3151b35c7976f
Author: Eric Harney <email address hidden>
Date: Wed Sep 23 11:06:30 2020 -0400

    SPDK: Report info in top-level volume_stats

    Report basic driver info in the top-level
    volume_stats instead of only in volume_stats['pools'].

    This aligns SPDK with how most other drivers
    report this information.

    Related-Bug: #1896764
    Change-Id: I1c8b2152c723f512f1013f31661be68b99579c29

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.