OpenStack Compute (nova)

Special characters are not allowed in connection string

Bug #1673613 reported by Dr. Clemens Hardewig on 2017-03-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Matt Riedemann
	Ocata	Fix Committed	High	Matt Riedemann

Bug Description

I installed ocata on a three node config (one controller, two compute). When executing

openstack server list

on the controller node, I get

Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-7544be6c-3302-4a94-94ac-834a50696ac6)

openstack --debug server list
START with options: [u'--debug', u'server', u'list']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='', auth_url='http://controller:35357/v3', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_alarming_api_version='2', os_beta_command=False, os_clustering_api_version='1', os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='2', os_key_manager_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='RegionOne', roles='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://controller:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'clustering_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'queues_api_version': '2', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://controller:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'clustering_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'queues_api_version': '2', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
messaging API version 2, cmd group openstack.messaging.v2
database API version 1, cmd group openstack.database.v1
clustering API version 1, cmd group openstack.clustering.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
alarming API version 2, cmd group openstack.alarming.v2
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://controller:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'clustering_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'queues_api_version': '2', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://controller:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'clustering_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'queues_api_version': '2', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
command: server list -> openstackclient.compute.v2.server.ListServer
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://controller:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'clustering_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'queues_api_version': '2', 'auth': {'user_domain_name': 'default', 'project_name': 'admin', 'project_domain_name': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'default', 'auth_url': 'http://controller:35357/v3', 'password': '***', 'project_domain_name': 'default'}
Get auth_ref
REQ: curl -g -i -X GET http://controller:35357/v3 -H "Accept: application/json" -H "User-Agent: osc-lib/1.3.0 keystoneauth1/2.18.0 python-requests/2.12.4 CPython/2.7.12"
Starting new HTTP connection (1): controller
http://controller:35357 "GET /v3 HTTP/1.1" 200 250
RESP: [200] Date: Thu, 16 Mar 2017 21:29:58 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-a047fe62-fc19-4b3c-b576-b888ccafcd46 Content-Length: 250 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://controller:35357/v3/", "rel": "self"}]}}

GET call to None for http://controller:35357/v3 used request id req-a047fe62-fc19-4b3c-b576-b888ccafcd46
Making authentication request to http://controller:35357/v3/auth/tokens
http://controller:35357 "POST /v3/auth/tokens HTTP/1.1" 201 7184
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "d2561fbef389426a9dcd38751c59bb0f", "name": "admin"}], "expires_at": "2017-03-16T22:29:58.000000Z", "project": {"domain": {"id": "a5b6d92142924b69b18bbde404da80da", "name": "default"}, "id": "e3fadcd68fad4f99981189c11ae552ee", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "13e0a79246d445d2ba3842094074b60e"}, {"url": "http://controller:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "3bc1f4b7c7fd458382a0d23b1c1d7f67"}, {"url": "http://controller:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "bd838710d18c4fac82284a00200edca2"}], "type": "network", "id": "2e0cce706b98403a989c899ddf15df58", "name": "neutron"}, {"endpoints": [{"url": "http://controller:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "279e42f1891e4cc7a25b5fe993e99006"}, {"url": "http://controller:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "3d962c9226f546f0ab7822def7a6ec92"}, {"url": "http://controller:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ff857c74b47f43a5a4ae1051298aabbc"}], "type": "image", "id": "4636390814c24bb48dc8c34eaf5dbe58", "name": "glance"}, {"endpoints": [{"url": "http://controller:8004/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "068b70eb0616465ab54e19e24cedfa6f"}, {"url": "http://controller:8004/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "2dd84ba0dd454c7386407c02583c6277"}, {"url": "http://controller:8004/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "8d3df08a8b32419d8d48c6825e39cce9"}], "type": "orchestration", "id": "55105385f39f440c873b40c0685043dd", "name": "heat"}, {"endpoints": [{"url": "http://controller:8776/v2/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "616497b5741949a695447c74bd5e7641"}, {"url": "http://controller:8776/v2/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "b4c7946a4bb0426da3c181951803f64b"}, {"url": "http://controller:8776/v2/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "bd3fced8d5354175811b0db55024ad3a"}], "type": "volumev2", "id": "5eb703d2d5e949159d6e8a210d3aa7fc", "name": "cinderv2"}, {"endpoints": [{"url": "http://controller:5000/v3", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "7bcc46fd75184b9d9573150932d19f9d"}, {"url": "http://controller:35357/v3", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "80b8578011384e5f96b7bbe7699f7263"}, {"url": "http://controller:5000/v3", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ad51ffcdd4ad46aeab67992a56be32ff"}], "type": "identity", "id": "8930fe9ee1e04969bcd47bad92126063", "name": "keystone"}, {"endpoints": [{"url": "http://controller:8776/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "baa22b0224a246b4bb62c093da80adfd"}, {"url": "http://controller:8776/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ec2367cd9e81402399c6160da3d81e71"}, {"url": "http://controller:8776/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f6aa26c57d4c484da1cf95fee38d3bca"}], "type": "volume", "id": "8e68e8d63d094e5bb9801e454a14b66c", "name": "cinder"}, {"endpoints": [], "type": "volumev2", "id": "903490fc4c354126a05f5d6a088ef14e", "name": "cinderv2"}, {"endpoints": [{"url": "http://controller:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "c74d516e2fc843e2a22747a2c9bca7e0"}, {"url": "http://controller:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "df0e197197b446e298dc097bb0534b2e"}, {"url": "http://controller:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "fcfed801e92342eab66ae75f75175a1d"}], "type": "cloudformation", "id": "a276247bf7b042b3a6812b3d3a156313", "name": "heat-cfn"}, {"endpoints": [{"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0c0459aec0614192b162a3b6146079ea"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "5d345e96f14641bc8d80bd5c166e4b94"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "6b09c93bcbf14c90b95eb83609c46739"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "87568fa717994340abd5f2e66cac6055"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "dd7db4e4d6b544b2b7ff62e6b1edda3f"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "f3d09def99364f3cb47e24a67985f941"}], "type": "compute", "id": "a3ced7c2ca5642408ec61cfb19049f03", "name": "nova"}, {"endpoints": [{"url": "http://controller:8778", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "580c70da7a6f4a7481f37ed2b44c304e"}, {"url": "http://controller:8778", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "850e61d97d3347008f87889773fda47f"}, {"url": "http://controller:8778", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "93ec2d5383c34a589ed676e8ccfd1fe2"}], "type": "placement", "id": "c0063048cfa7463b995821b6315f0311", "name": "placement"}, {"endpoints": [{"url": "http://controller:9511/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "12687797bc5b48de8c2e3209f2f9f241"}, {"url": "http://controller:9511/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "a18e51c35b9b469486d5660b26285451"}, {"url": "http://controller:9511/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e9b868e234414ca4be1144a7311cb3eb"}], "type": "container-infra", "id": "eab4ca6a68094a1bbb58a7383edb2eee", "name": "magnum"}], "user": {"domain": {"id": "a5b6d92142924b69b18bbde404da80da", "name": "default"}, "password_expires_at": null, "name": "admin", "id": "85e68b86be514c3aa30d8b65844aa9fb"}, "audit_ids": ["yMMe9bIoQ5eyhn6jUIcKfQ"], "issued_at": "2017-03-16T21:29:58.000000Z"}}
run(Namespace(all_projects=False, changes_since=None, columns=[], deleted=False, flavor=None, formatter='table', host=None, image=None, instance_name=None, ip=None, ip6=None, limit=None, long=False, marker=None, max_width=0, name=None, noindent=False, print_empty=False, project=None, project_domain=None, quote_mode='nonnumeric', reservation_id=None, status=None, user=None, user_domain=None))
Instantiating compute client for API Version Major: 2, Minor: 0
Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
Instantiating image client: <class 'glanceclient.v2.client.Client'>
Making authentication request to http://controller:35357/v3/auth/tokens
http://controller:35357 "POST /v3/auth/tokens HTTP/1.1" 201 7184
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "d2561fbef389426a9dcd38751c59bb0f", "name": "admin"}], "expires_at": "2017-03-16T22:29:58.000000Z", "project": {"domain": {"id": "a5b6d92142924b69b18bbde404da80da", "name": "default"}, "id": "e3fadcd68fad4f99981189c11ae552ee", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "13e0a79246d445d2ba3842094074b60e"}, {"url": "http://controller:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "3bc1f4b7c7fd458382a0d23b1c1d7f67"}, {"url": "http://controller:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "bd838710d18c4fac82284a00200edca2"}], "type": "network", "id": "2e0cce706b98403a989c899ddf15df58", "name": "neutron"}, {"endpoints": [{"url": "http://controller:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "279e42f1891e4cc7a25b5fe993e99006"}, {"url": "http://controller:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "3d962c9226f546f0ab7822def7a6ec92"}, {"url": "http://controller:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ff857c74b47f43a5a4ae1051298aabbc"}], "type": "image", "id": "4636390814c24bb48dc8c34eaf5dbe58", "name": "glance"}, {"endpoints": [{"url": "http://controller:8004/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "068b70eb0616465ab54e19e24cedfa6f"}, {"url": "http://controller:8004/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "2dd84ba0dd454c7386407c02583c6277"}, {"url": "http://controller:8004/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "8d3df08a8b32419d8d48c6825e39cce9"}], "type": "orchestration", "id": "55105385f39f440c873b40c0685043dd", "name": "heat"}, {"endpoints": [{"url": "http://controller:8776/v2/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "616497b5741949a695447c74bd5e7641"}, {"url": "http://controller:8776/v2/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "b4c7946a4bb0426da3c181951803f64b"}, {"url": "http://controller:8776/v2/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "bd3fced8d5354175811b0db55024ad3a"}], "type": "volumev2", "id": "5eb703d2d5e949159d6e8a210d3aa7fc", "name": "cinderv2"}, {"endpoints": [{"url": "http://controller:5000/v3", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "7bcc46fd75184b9d9573150932d19f9d"}, {"url": "http://controller:35357/v3", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "80b8578011384e5f96b7bbe7699f7263"}, {"url": "http://controller:5000/v3", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ad51ffcdd4ad46aeab67992a56be32ff"}], "type": "identity", "id": "8930fe9ee1e04969bcd47bad92126063", "name": "keystone"}, {"endpoints": [{"url": "http://controller:8776/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "baa22b0224a246b4bb62c093da80adfd"}, {"url": "http://controller:8776/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ec2367cd9e81402399c6160da3d81e71"}, {"url": "http://controller:8776/v1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f6aa26c57d4c484da1cf95fee38d3bca"}], "type": "volume", "id": "8e68e8d63d094e5bb9801e454a14b66c", "name": "cinder"}, {"endpoints": [], "type": "volumev2", "id": "903490fc4c354126a05f5d6a088ef14e", "name": "cinderv2"}, {"endpoints": [{"url": "http://controller:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "c74d516e2fc843e2a22747a2c9bca7e0"}, {"url": "http://controller:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "df0e197197b446e298dc097bb0534b2e"}, {"url": "http://controller:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "fcfed801e92342eab66ae75f75175a1d"}], "type": "cloudformation", "id": "a276247bf7b042b3a6812b3d3a156313", "name": "heat-cfn"}, {"endpoints": [{"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0c0459aec0614192b162a3b6146079ea"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "5d345e96f14641bc8d80bd5c166e4b94"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "6b09c93bcbf14c90b95eb83609c46739"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "87568fa717994340abd5f2e66cac6055"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "dd7db4e4d6b544b2b7ff62e6b1edda3f"}, {"url": "http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "f3d09def99364f3cb47e24a67985f941"}], "type": "compute", "id": "a3ced7c2ca5642408ec61cfb19049f03", "name": "nova"}, {"endpoints": [{"url": "http://controller:8778", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "580c70da7a6f4a7481f37ed2b44c304e"}, {"url": "http://controller:8778", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "850e61d97d3347008f87889773fda47f"}, {"url": "http://controller:8778", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "93ec2d5383c34a589ed676e8ccfd1fe2"}], "type": "placement", "id": "c0063048cfa7463b995821b6315f0311", "name": "placement"}, {"endpoints": [{"url": "http://controller:9511/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "12687797bc5b48de8c2e3209f2f9f241"}, {"url": "http://controller:9511/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "a18e51c35b9b469486d5660b26285451"}, {"url": "http://controller:9511/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e9b868e234414ca4be1144a7311cb3eb"}], "type": "container-infra", "id": "eab4ca6a68094a1bbb58a7383edb2eee", "name": "magnum"}], "user": {"domain": {"id": "a5b6d92142924b69b18bbde404da80da", "name": "default"}, "password_expires_at": null, "name": "admin", "id": "85e68b86be514c3aa30d8b65844aa9fb"}, "audit_ids": ["e-JmAEe8Tga_ocWr3gNmUg"], "issued_at": "2017-03-16T21:29:58.000000Z"}}
Instantiating image api: <class 'openstackclient.api.image_v2.APIv2'>
search options: {'status': None, 'deleted': False, 'ip': None, 'image': None, 'host': None, 'ip6': None, 'all_tenants': False, 'flavor': None, 'reservation_id': None, 'user_id': None, 'name': None, 'tenant_id': None, 'instance_name': None, 'changes_since': None}
REQ: curl -g -i -X GET http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee/servers/detail -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}d0dc20da374ca39d7ba171733e1282da74348051"
Starting new HTTP connection (1): controller
http://controller:8774 "GET /v2.1/e3fadcd68fad4f99981189c11ae552ee/servers/detail HTTP/1.1" 500 205
RESP: [500] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 205 X-Compute-Request-Id: req-18761596-272c-425c-b19b-92037123606f Date: Thu, 16 Mar 2017 21:29:59 GMT Connection: keep-alive
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'sqlalchemy.exc.OperationalError'>", "code": 500}}

GET call to compute for http://controller:8774/v2.1/e3fadcd68fad4f99981189c11ae552ee/servers/detail used request id req-18761596-272c-425c-b19b-92037123606f
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-18761596-272c-425c-b19b-92037123606f)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/compute/v2/server.py", line 947, in take_action
    limit=parsed_args.limit)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 861, in list
    "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 254, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 217, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 80, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-18761596-272c-425c-b19b-92037123606f)
clean_up ListServer: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-18761596-272c-425c-b19b-92037123606f)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 112, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/compute/v2/server.py", line 947, in take_action
    limit=parsed_args.limit)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 861, in list
    "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 254, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 217, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 80, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-18761596-272c-425c-b19b-92037123606f)

Manual check of access to mysql db with user nova and configured password prove to work (both nova and nova_api db)

My nova.conf says:
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
debug=true
#debug=false
#logdir=/var/log/nova
transport_url=rabbit://openstack:rabbitblabla@controller
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
enabled_apis = osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.0.88.11
use_neutron = True
security_group_api = neutron
firewall_driver = nova.virt.firewall.NoopFirewallDriver
block_device_allocate_retries = 300
block_device_allocate_retries_interval = 10
block_device_creation_timeout = 300
#vnc_enabled=false
#web=/usr/share/spice-html5
web=/opt/novnc
# SCHEDULER
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_available_filters = nova.scheduler.filters.all_filters
scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter

[api_database]
connection = mysql+pymysql://nova:novablabla@controller/nova_api

[database]
connection = mysql+pymysql://nova:novablabla@controller/nova

[api]
# ...
auth_strategy = keystone

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = novablabla

[placement]
os_region_name = RegionOne
project_domain_name = Default
project_name = service
user_domain_name = Default
username = placement
password = placementblabla
auth_url = http://controller:35357/v3
auth_type = password

[placement_database]
connection = mysql+pymysql://nova:novablabla@controller/nova_api

As a consequence, also no information about server/instances in horizon are displayed.

Why is the access to the nova(_api) db refused/not authorized?

Tags:

Revision history for this message

Andrey Volkov (avolkov) wrote on 2017-03-17:

'openstack --debug server list' outputs a lot )

You definitely got some DB related error. I'd happy to see attached nova api log as your error message suggested.

Also I'm not sure about 'placement_database'. Did you see it somewhere in docs? I believe it was removed (https://review.openstack.org/#/c/442762/).

Changed in nova:
status:	New → Incomplete

Revision history for this message

Dr. Clemens Hardewig (bringha1) wrote on 2017-03-17:

Download full text (14.0 KiB)

Apologies - here my nova-api.log

2017-03-17 17:34:47.622 19345 DEBUG nova.compute.api [req-7478b76c-8ee9-442d-829b-cdb8af4960dc 85e68b86be514c3aa30d8b65844aa9fb e3fadcd68fad4f99981189c11ae552ee - a5b6d92142924b69b18bbde404da80da a5b6d92142924b69b18bbde404da80da] Searching by: {'deleted': False, 'project_id': u'e3fadcd68fad4f99981189c11ae552ee'} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2381
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions [req-7478b76c-8ee9-442d-829b-cdb8af4960dc 85e68b86be514c3aa30d8b65844aa9fb e3fadcd68fad4f99981189c11ae552ee - a5b6d92142924b69b18bbde404da80da a5b6d92142924b69b18bbde404da80da] Unexpected exception in API method
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 181, in wrapper
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 181, in wrapper
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 214, in detail
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 357, in _get_servers
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2466, in get_all
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     sort_dirs=sort_dirs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2606, in _get_instances_by_filters
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1220, in get_by_filters
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     with reader_mode.using(context):
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return self.gen.next()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 944, in _transaction_scope
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     allow_async=self._allow_async) as resource:
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return self.gen.next()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 558, in _session
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     bind=self.connection, mode=self.mode)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _create_session
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     self._start()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _start
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     engine_args, maker_args)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 427, in _setup_for_connection
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     sql_connection=sql_connection, **engine_kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     test_conn = _test_connection(engine, max_retries, retry_interval)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return engine.connect()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2018, in connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return self._connection_cls(self, **kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 72, in __init__
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     if connection is not None else engine.raw_connection()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     self.pool.unique_connection, _connection)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     e, dialect, self)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     util.raise_from_cause(newraise, exc_info)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return fn()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 318, in unique_connection
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     rec = pool._do_get()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     self._dec_overflow()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     compat.reraise(exc_type, exc_value, exc_tb)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return self._create_connection()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return _ConnectionRecord(self)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     self.connection = self.__connect()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     connection = self.__pool._invoke_creator(self)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return dialect.connect(*cargs, **cparams)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 688, in __init__
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     self.connect()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 906, in connect
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     self._request_authentication()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1114, in _request_authentication
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     auth_packet = self._read_packet()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 981, in _read_packet
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     packet.check_error()
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'nova'@'controller' (using password: YES)")
2017-03-17 17:34:47.628 19345 ERROR nova.api.openstack.extensions
2017-03-17 17:34:47.629 19345 INFO nova.api.openstack.wsgi [req-7478b76c-8ee9-442d-829b-cdb8af4960dc 85e68b86be514c3aa30d8b65844aa9fb e3fadcd68fad4f99981189c11ae552ee - a5b6d92142924b69b18bbde404da80da a5b6d92142924b69b18bbde404da80da] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>
2017-03-17 17:34:47.629 19345 DEBUG nova.api.openstack.wsgi [req-7478b76c-8ee9-442d-829b-cdb8af4960dc 85e68b86be514c3aa30d8b65844aa9fb e3fadcd68fad4f99981189c11ae552ee - a5b6d92142924b69b18bbde404da80da a5b6d92142924b69b18bbde404da80da] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1039
2017-03-17 17:34:47.629 19345 INFO nova.osapi_compute.wsgi.server [req-7478b76c-8ee9-442d-829b-cdb8af4960dc 85e68b86be514c3aa30d8b65844aa9fb e3fadcd68fad4f99981189c11ae552ee - a5b6d92142924b69b18bbde404da80da a5b6d92142924b69b18bbde404da80da] 10.0.88.11 "GET /v2.1/e3fadcd68fad4f99981189c11ae552ee/servers/detail HTTP/1.1" status: 500 len: 572 time: 0.1815181

10.0.88.11 is my controller

Yes I agree that I have a DB issue. Point is that I can access the nova_api DB with the credentials configured in nova.conf and the user in the user given by 'openstack user list'. Mysql.log shows me also the output of error code 1045

Revision history for this message

Dr. Clemens Hardewig (bringha1) wrote on 2017-03-17:

And to the placement database: There is some misleading information in the docs wrt to 'database term' for placement in Ocata. Was corrected in another bug. However, extra installation for placement (wsgi) were required with apt install nova-placement-api and subsequent config as an extension for the nova-api.

Revision history for this message

Dr. Clemens Hardewig (bringha1) wrote on 2017-03-18:

Download full text (4.7 KiB)

I believe I found the reason behind:

When Upgrading from Newton to Ocata far going extensions and restructuring of nova/nova_api db is carried out to provide the mandatory cellV2 and placement api extensions. Beside the fact that this is not at all reflected appropriately in the documentation of Ocata so far (sic!), at least in my case, the update of compute completely messed up. Even a complete new installation of the nova from scratch does NOT create the nova_cell0 db NOR is a hint in docs.openstack.org that this step is to be done manually... Anyhow ...

The behavior of the nova-manage command has changed with Ocata: With the far going changes around database structure with cells and placement, it is no longer possible that passwords for the nova, nova_api db contain reserved characters for URL/URI. Creating the cell db with

nova-manage cell_v2 simple_cell_setup

and having a password for nova_api db in /etc/nova.conf

connection = mysql+pymysql://nova:abcd0123?AB@controller/nova_api
connection = mysql+pymysql://nova:abcd0123?AB@controller/nova

leads to

mysql> select * from cell_mappings;
+---------------------+------------+----+--------------------------------------+-------+---------------+---------------------------------------------------------+
| created_at | updated_at | id | uuid | name | transport_url | database_connection |
+---------------------+------------+----+--------------------------------------+-------+---------------+---------------------------------------------------------+
| 2017-03-17 21:53:32 | NULL | 1 | 00000000-0000-0000-0000-000000000000 | cell0 | none:/// | mysql+pymysql://nova:abcd0123/_cell0?AB@controller/nova |
+---------------------+------------+----+--------------------------------------+-------+---------------+---------------------------------------------------------+

before nova-manage stops with the "access denied for 'nova'@'controller' " message

As it could be seen, the URL string in the cell0 db is malformatted. With the incomplete nova-manage command, also 'openstack server list' will fail for the same reasons.

If the password contains indeed special, but no reserved characters, all is fine:
Again /etc/nova.conf contains

connection = mysql+pymysql://nova:abcd0123!AB@controller/nova_api
connection = mysql+pymysql://nova:abcd0123!AB@controller/nova

executing nova-manage cell_v2 simple_cell_setup

counts bravely up to:
(...)
2017-03-18 10:29:14.628 25357 INFO migrate.versioning.api [req-d98c4c1d-1305-4209-8aba-ca798615077b - - - - -] 346 -> 347...
2017-03-18 10:29:14.978 25357 INFO migrate.versioning.api [req-d98c4c1d-1305-4209-8aba-ca798615077b - - - - -] done

and leads to:

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

summary:	- cli command 'openstack server list' fails + Special characters are not allowed in connection string
Changed in nova:
status:	Incomplete → Confirmed
importance:	Undecided → Medium

Changed in nova:
assignee:	nobody → Matt Riedemann (mriedem)
status:	Confirmed → In Progress