cinderclient.quotas.get() fails against Cinder service in a Devstack

Bug #1491495 reported by Timur Sufiev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Ivan Kolodyazhny

Bug Description

Horizon integration tests which are run against devstack deployed as part of this job https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/horizon.yaml#L29 are failing due to this call https://github.com/openstack/horizon/blob/master/openstack_dashboard/api/cinder.py#L418 failing with status 500 and a message u'The server has either erred or is incapable of performing the requested operation.'

Due to the fact that integration tests passed before, most likely something in Cinder quotas machinery changed and quite recently (1-3 days ago).

/etc/cinder/cinder.conf:

[keystone_authtoken]
signing_dir = /var/cache/cinder
cafile = /opt/stack/data/ca-bundle.pem
auth_uri = http://<HOST_IP>:5000
project_domain_id = default
project_name = service
user_domain_id = default
password = <password>
username = cinder
auth_url = http://<HOST_IP>:35357
auth_plugin = password

[DEFAULT]
os_privileged_user_tenant = service
os_privileged_user_password = <password>
os_privileged_user_name = nova
glance_api_servers = http://<HOST_IP>:9292
osapi_volume_workers = 2
logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s ESC[01;35m%(instance)sESC[00m
logging_debug_format_suffix = ESC[00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)dESC[00m
logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [ESC[00;36m-%(color)s] ESC[01;35m%(instance)s%(color)s%(message)sESC[00m
logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [ESC[01;36m%(request_id)s ESC[00;36m%(user_id)s %(project_id)s%(color)s] ESC[01;35m%(instance)s%(color)s%(message)sESC[00m
volume_clear = zero
rpc_backend = rabbit
default_volume_type = lvmdriver-1
enabled_backends = lvmdriver-1
os_region_name = RegionOne
enable_v1_api = true
periodic_interval = 60
state_path = /opt/stack/data/cinder
osapi_volume_listen = 0.0.0.0
osapi_volume_extension = cinder.api.contrib.standard_extensions
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_config = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
verbose = True
debug = True
auth_strategy = keystone
nova_catalog_admin_info = compute:nova:adminURL
nova_catalog_info = compute:nova:publicURL

[database]
connection = mysql+pymysql://root:swordfish@127.0.0.1/cinder?charset=utf8

[oslo_concurrency]
lock_path = /opt/stack/data/cinder

[lvmdriver-1]
lvm_type = default
iscsi_helper = tgtadm
volume_group = stack-volumes-lvmdriver-1
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvmdriver-1

[oslo_messaging_rabbit]
rabbit_userid = stackrabbit
rabbit_password = <password>
rabbit_hosts = <HOST_IP>

Timur Sufiev (tsufiev-x)
description: updated
Timur Sufiev (tsufiev-x)
description: updated
Ivan Kolodyazhny (e0ne)
Changed in cinder:
status: New → Confirmed
assignee: nobody → Ivan Kolodyazhny (e0ne)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

The root cause is: Cinder returns 500 error for non-admin user for quota-show method:

e0ne@dsvm2:~/devstack$ cinder quota-show 72833f28a109465582295eabde21d64f
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-f0445fc3-e3a2-4807-835f-42523c40b61c)

Changed in cinder:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/219231
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=e065e70a69b1b6de1bd383cb471dbab62b93e03a
Submitter: Jenkins
Branch: master

commit e065e70a69b1b6de1bd383cb471dbab62b93e03a
Author: Ivan Kolodyazhny <email address hidden>
Date: Tue Sep 1 14:00:06 2015 +0300

    Remove empty rules from policies for API access

    Empty policy rule means that API method is allowed by anybody. Default rule
    is used only if such rule is not defined in policy.json.

    This patch changes empty rules to admin_api or admin_or_owner.

    Closes-Bug: #1477621
    Closes-Bug: #1491495
    Change-Id: I512e65e62da7dab5834a21ef9fd967ed6e9bb695

Changed in cinder:
status: In Progress → Fix Committed
Mike Perez (thingee)
Changed in cinder:
milestone: none → liberty-rc1
importance: Undecided → Medium
Revision history for this message
Cory Stone (corystone) wrote :

This isn't failing due to the cinder policy file. It's failing because the quota get now calls the identity for something we don't by default have permission to call.

Revision history for this message
Cory Stone (corystone) wrote :

{u'error': {u'message': u'You are not authorized to perform the requested action: identity:get_project (Disable debug mode to suppress these details.)', u'code': 403, u'title': u'Forbidden'}}

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/225891

Ivan Kolodyazhny (e0ne)
Changed in cinder:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/225891
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=97170c9fba69b6570c59ff7008e45fb1a47e3886
Submitter: Jenkins
Branch: master

commit 97170c9fba69b6570c59ff7008e45fb1a47e3886
Author: Ivan Kolodyazhny <email address hidden>
Date: Mon Sep 21 18:15:31 2015 +0300

    Ignore Forbidden error on quotas-get for nested projects

    Keystone API v2 requires admin permissions for project_get method.
    We ignore Forbidden exception for non-admin users.

    Change-Id: Ia0a17a9377219d3bf92dbbfd6e88077209a22a95
    Closes-Bug: #1491495

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: liberty-rc1 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.