OpenStack Compute (nova)

neutronclient in nova doesn't issue new token

Bug #1491905 reported by Timofey Durakov
46
This bug affects 8 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
Undecided
Timofey Durakov

Bug Description

neutronclient in nova uses token_endpoint.Token as auth_plugin, which doesn't have functionality to re-authenticate in case of 401 error. If keystone uses memcached as backend and it was restarted, token that neutronclient use become invalid, and no re-authentication attempts were made.
Trace in nova-api:
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1519, in _build_network_info_model
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] data = client.list_ports(**search_opts)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] ret = self.function(instance, args, *kwargs)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 534, in list_ports
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] **_params)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] for r in self._pagination(collection, path, **params):
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] res = self.get(path, params=params)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] headers=headers, params=params)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] headers=headers, params=params)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 211, in do_request
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] self._handle_fault_response(status_code, replybody)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] exception_handler_v20(status_code, des_error_body)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] message=message)
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] NeutronClientException: Authentication required
2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8]

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/220207

Changed in nova:
assignee: nobody → Timofey Durakov (tdurakov)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Timofey Durakov (<email address hidden>) on branch: master
Review: https://review.openstack.org/220207
Reason: not actual

iwan (igoshin-r)
Changed in nova:
status: In Progress → Fix Released
Revision history for this message
Luo Gangyi (luogangyi) wrote :

@iwan where is the release?

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Not sure the problem is in Nova at all, it should at least be a Neutron bug.

Changed in nova:
status: Fix Released → Opinion
Revision history for this message
Silvan Kaiser (2-silvan) wrote :

Just had this issue pop up on a CentOS 7 / RDO (kilo release) installation when testing migration of a shut down instance in my setup. Shouldn't this be moved to Neutron instead of being marked as 'Opinion'? Furthermore, might this be a configuration issue?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.