Cinder

Cinder-WSGI cannot find api-paste.ini - Ubuntu

Bug #1715024 reported by Jon Davis on 2017-09-04

This bug affects 7 people

	Status	Importance	Assigned to
Cinder	Invalid	Undecided	Unassigned
Ubuntu Cloud Archive	Fix Released	High	James Page
Pike	Fix Released	High	James Page
cinder (Ubuntu)	Fix Released	High	James Page

Bug Description

I upgraded Ocata to Pike and am getting error in apache2 cinder_error.log when trying to list volumes via API or Horizon:

root@cinder-test:/var/log/apache2# tail -f cinder_error.log
2017-09-04 15:30:52.587013 mod_wsgi (pid=1847): Target WSGI script '/usr/bin/cinder-wsgi' cannot be loaded as Python module.
2017-09-04 15:30:52.587099 mod_wsgi (pid=1847): Exception occurred processing WSGI script '/usr/bin/cinder-wsgi'.
2017-09-04 15:30:52.587126 Traceback (most recent call last):
2017-09-04 15:30:52.587147 File "/usr/bin/cinder-wsgi", line 51, in <module>
2017-09-04 15:30:52.587220 application = initialize_application()
2017-09-04 15:30:52.587235 File "/usr/lib/python2.7/dist-packages/cinder/wsgi/wsgi.py", line 46, in initialize_application
2017-09-04 15:30:52.588456 return wsgi.Loader(CONF).load_app(name='osapi_volume')
2017-09-04 15:30:52.588480 File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 340, in __init__
2017-09-04 15:30:52.589984 raise ConfigNotFound(path=config_path)
2017-09-04 15:30:52.590011 ConfigNotFound: Could not find config at api-paste.ini

Also did fresh controller install (Pike) on a test machine to see if I could replicate, using instructions: https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment

I get the same error, following docs closely.

Here is some output of the environment:

Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-93-generic x86_64)

################# PERMISSIONS ########################

root@cinder-test:/etc/cinder# ls -aFl
total 36
drwxr-x--- 3 root cinder 4096 Sep 4 15:10 ./
drwxr-xr-x 105 root root 4096 Sep 4 15:30 ../
-rw-r--r-- 1 root cinder 2711 Aug 30 11:07 api-paste.ini
-rw-r--r-- 1 root cinder 812 Sep 4 15:10 cinder.conf
-rw-r--r-- 1 root cinder 1359 Aug 30 13:11 logging.conf
-rw-r--r-- 1 root cinder 7708 Aug 30 11:07 policy.json
-rw-r--r-- 1 root root 991 Aug 30 11:07 rootwrap.conf
drwxr-xr-x 2 root root 4096 Aug 31 11:49 rootwrap.d/

################# API-PASTE.INI ########################

#############
# OpenStack #
#############

[composite:osapi_volume]
use = call:cinder.api:root_app_factory
/: apiversions
/v1: openstack_volume_api_v1
/v2: openstack_volume_api_v2
/v3: openstack_volume_api_v3

[composite:openstack_volume_api_v1]
use = call:cinder.api.middleware.auth:pipeline_factory
noauth = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler noauth apiv1
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv1
keystone_nolimit = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv1

[composite:openstack_volume_api_v2]
use = call:cinder.api.middleware.auth:pipeline_factory
noauth = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler noauth apiv2
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv2
keystone_nolimit = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv2

[composite:openstack_volume_api_v3]
use = call:cinder.api.middleware.auth:pipeline_factory
noauth = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler noauth apiv3
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv3
keystone_nolimit = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv3

[filter:request_id]
paste.filter_factory = oslo_middleware.request_id:RequestId.factory

[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory

[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = cinder

[filter:faultwrap]
paste.filter_factory = cinder.api.middleware.fault:FaultWrapper.factory

[filter:osprofiler]
paste.filter_factory = osprofiler.web:WsgiMiddleware.factory

[filter:noauth]
paste.filter_factory = cinder.api.middleware.auth:NoAuthMiddleware.factory

[filter:sizelimit]
paste.filter_factory = oslo_middleware.sizelimit:RequestBodySizeLimiter.factory

[app:apiv1]
paste.app_factory = cinder.api.v1.router:APIRouter.factory

[app:apiv2]
paste.app_factory = cinder.api.v2.router:APIRouter.factory

[app:apiv3]
paste.app_factory = cinder.api.v3.router:APIRouter.factory

[pipeline:apiversions]
pipeline = cors http_proxy_to_wsgi faultwrap osvolumeversionapp

[app:osvolumeversionapp]
paste.app_factory = cinder.api.versions:Versions.factory

##########
# Shared #
##########

[filter:keystonecontext]
paste.filter_factory = cinder.api.middleware.auth:CinderKeystoneContext.factory

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory

################# CINDER.CONF ########################

[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
enabled_backends = lvm
transport_url = rabbit://openstack:openstack@controller
my_ip = 10.1.1.22

[database]
connection = mysql+pymysql://cinder:openstack@controller/cinder

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = openstack

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

################# APACHE WSGI ########################

Listen 8776
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D(us)" cinder_combined

<VirtualHost *:8776>
    WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder display-name=%{GROUP}
    WSGIProcessGroup cinder-wsgi
    WSGIScriptAlias / /usr/bin/cinder-wsgi
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>

ErrorLog /var/log/apache2/cinder_error.log
CustomLog /var/log/apache2/cinder.log cinder_combined

    <Directory /usr/bin>
<IfVersion >= 2.4>
            Require all granted
        </IfVersion>
        <IfVersion < 2.4>
            Order allow,deny
            Allow from all
        </IfVersion>
    </Directory>
</VirtualHost>

See original description

Tags:

Jon Davis (aggiering) on 2017-09-04

description:

updated

Revision history for this message

Christoph Fiehe (fiehe) wrote on 2017-09-05:

I can confirm this bug... I made a fresh OpenStack Pike installation on Ubuntu 16.04.3 LTS and receive the same error.

Some version information:
$ pip list | grep cinder
cinder (11.0.0)
python-cinderclient (3.1.0)

Changed in cinder:
status:	New → Confirmed

Revision history for this message

Christoph Fiehe (fiehe) wrote on 2017-09-05:

I got it working, even when it took me some hours.

It's just a permission problem that can be fixed as follows:
(1) Ensure that the cinder configuration folder is owned by the 'cinder' group:
    $ chown root:cinder /etc/cinder
(2) Modify the apache configuration and ensure that the process is executed using the 'cinder' group.
    WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder group=cinder display-name=%{GROUP}
(3) Restart Apache:
    $ service apache2 restart

These steps fixed the problem described above.

Eric Harney (eharney) on 2017-09-05

Changed in cinder:
status:	Confirmed → Invalid

Revision history for this message

David Seelbach (davedastrange) wrote on 2017-09-09:

can confirm
thanks wolle5050

Revision history for this message

Rufus Deponian (deponian) wrote on 2017-09-12:

Christoph Fiehe, thank you so much

Revision history for this message

Thomas Dreibholz (dreibh) wrote on 2017-09-14:

Download full text (4.2 KiB)

I can confirm the bug. On a fresh install of OpenStack Pike:

1. Run "cinder --debug list":
...
DEBUG:keystoneauth:REQ: curl -g -i -X GET http://nisse.simula.nornet:8776/v3/168529e95adc4284a40c28f648d48d6f/volumes/detail -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}fc794cac0686f40616da08b48c22144c58d2745b"
DEBUG:keystoneauth:RESP: [500] Date: Thu, 14 Sep 2017 20:30:08 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 618 Connection: close Content-Type: text/html; charset=iso-8859-1
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.

DEBUG (shell:974) Internal Server Error (HTTP 500)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 969, in main
    for item in sys.argv[1:]])
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 787, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v3/shell.py", line 328, in do_list
    sort_dir=args.sort_dir, sort=args.sort)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 321, in list
    return self._list(url, resource_type, limit=limit)
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 84, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 177, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 168, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 154, in request
    raise exceptions.from_response(resp, body)
ClientException: Internal Server Error (HTTP 500)
DEBUG:cinderclient.shell:Internal Server Error (HTTP 500)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 969, in main
    for item in sys.argv[1:]])
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 787, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v3/shell.py", line 328, in do_list
    sort_dir=args.sort_dir, sort=args.sort)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 321, in list
    return self._list(url, resource_type, limit=limit)
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 84, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 177, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 168, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 154, in request
    raise exceptions.from_response(resp, body)
ClientException: Internal Server Error (HTTP 500)
ERROR: Internal Server Error (HTTP 500)

2. Check /var/log/apache2/cinder_error.log
017-09-14 20:40:59.764418 mod_wsgi (pid=20203): Target WSGI script '/usr/bin/cinder-wsgi' cannot be loade...

I can confirm the bug. On a fresh install of OpenStack Pike:

1. Run "cinder --debug list":
...
DEBUG:keystoneauth:REQ: curl -g -i -X GET http://nisse.simula.nornet:8776/v3/168529e95adc4284a40c28f648d48d6f/volumes/detail -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}fc794cac0686f40616da08b48c22144c58d2745b"
DEBUG:keystoneauth:RESP: [500] Date: Thu, 14 Sep 2017 20:30:08 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 618 Connection: close Content-Type: text/html; charset=iso-8859-1 
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.

2. Check /var/log/apache2/cinder_error.log
017-09-14 20:40:59.764418 mod_wsgi (pid=20203): Target WSGI script '/usr/bin/cinder-wsgi' cannot be loaded as Python module.
2017-09-14 20:40:59.764463 mod_wsgi (pid=20203): Exception occurred processing WSGI script '/usr/bin/cinder-wsgi'.
2017-09-14 20:40:59.764487 Traceback (most recent call last):
2017-09-14 20:40:59.764508   File "/usr/bin/cinder-wsgi", line 51, in <module>
2017-09-14 20:40:59.764574     application = initialize_application()
2017-09-14 20:40:59.764590   File "/usr/lib/python2.7/dist-packages/cinder/wsgi/wsgi.py", line 46, in initialize_application
2017-09-14 20:40:59.764640     return wsgi.Loader(CONF).load_app(name='osapi_volume')
2017-09-14 20:40:59.764655   File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 340, in __init__
2017-09-14 20:40:59.764793     raise ConfigNotFound(path=config_path)
2017-09-14 20:40:59.764819 ConfigNotFound: Could not find config at api-paste.ini

Fix is as in message #2:

/etc/apache2/conf-available/cinder-wsgi.conf contains:

WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder display-name=%{GROUP}

Changing it to:

WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder group=cinder display-name=%{GROUP}

solves the problem.

Revision history for this message

Thomas Dreibholz (dreibh) wrote on 2017-09-14:

/etc/apache2/conf-available/cinder-wsgi.conf is from the package "cinder-api".

Version:
nornetpp@nisse:~$ apt-show-versions | grep cinder-api
cinder-api:all/xenial-updates 2:11.0.0-0ubuntu1~cloud0 uptodate

Repository:
nornetpp@nisse:~$ cat /etc/apt/sources.list.d/cloudarchive-pike.list
deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/pike main
deb-src http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/pike main

Hans Joachim Desserud (hjd) on 2017-09-15

tags:

added: xenial

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-09-15:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cinder (Ubuntu):
status:	New → Confirmed

Revision history for this message

Sourabh Jaiswal (sourabh-jaiswal) wrote on 2017-09-15:

I am also facing the same issue in a fresh pike installation, However the solution "Christoph Fiehe (cfiehe)" Suggested Works Fine.

James Page (james-page) on 2017-09-18

Changed in cinder (Ubuntu):
importance:	Undecided → High
status:	Confirmed → Triaged
assignee:	nobody → James Page (james-page)

James Page (james-page) on 2017-09-18

Changed in cinder (Ubuntu):
status:	Triaged → In Progress

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-09-19:

This bug was fixed in the package cinder - 2:11.0.0-0ubuntu2

---------------
cinder (2:11.0.0-0ubuntu2) artful; urgency=medium

* d/cinder-wsgi.conf: Ensure apache processes run under cinder group,
allowing access to /etc/cinder/* (LP: #1715024).

-- James Page <email address hidden> Mon, 18 Sep 2017 09:24:40 +0100

Changed in cinder (Ubuntu):
status:	In Progress → Fix Released

Revision history for this message

James Page (james-page) wrote on 2017-09-21: Please test proposed package

#10

Hello Jon, or anyone else affected,

Accepted cinder into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

sudo add-apt-repository cloud-archive:pike-proposed
sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-pike-needed

Revision history for this message

James Page (james-page) wrote on 2017-09-21: Update Released

#11

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

James Page (james-page) wrote on 2017-09-21:

#12

This bug was fixed in the package cinder - 2:11.0.0-0ubuntu2~cloud0
---------------

cinder (2:11.0.0-0ubuntu2~cloud0) xenial-pike; urgency=medium
.
   * New update for the Ubuntu Cloud Archive.
.
cinder (2:11.0.0-0ubuntu2) artful; urgency=medium
.
   * d/cinder-wsgi.conf: Ensure apache processes run under cinder group,
     allowing access to /etc/cinder/* (LP: #1715024).

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.