OpenStack Compute (nova)

nova-compute error

Bug #1868203 reported by YG Kumar on 2020-03-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Invalid	Undecided	Unassigned

Bug Description

Hi All,

I have a rocky setup. I am unable to launch instances on one of the compute nodes.

I see the following error in the conductor logs and the libvirtd service on the compute. The host is ubuntu 18.04:

------------------

File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last):
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 1237, in schedule_and_build_instances
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager instance_uuids, return_alternates=True)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 750, in _schedule_instances
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return_alternates=return_alternates)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 50, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager retry=self.retry)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager retry=retry)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager call_monitor_timeout, retry=retry)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager raise result
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last):
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return func(*args, **kwargs)
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager raise exception.NoValidHost(reason="")
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
2020-03-20 05:31:09.390 35363 WARNING nova.scheduler.utils [req-cd25c8b1-44a2-40d7-9f0f-0d81bf5c3c08 2db30fa813074deb80e54e76529e2e71 aa73777f89414369924a0794b3eaf99e - default default] Failed to compute_task_build_instances: No valid host was found.
--------------------------------

This is the error on the nova-compute service on that compute node:
-------------
2020-03-20 01:44:24.158 6193 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID ea3af0ff-fb69-449a-b484-5ac722010a1f
------------------

This is the status of the libvirtd service on that compute node:

------------------------
Mar 19 04:32:04 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for probing: /usr/bin/qemu-system-ppc64: relocation error: /usr/
Mar 19 04:32:04 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for
Mar 19 04:32:04 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le for probing: /usr/bin/qemu-system-ppc64le: relocation error: /
Mar 19 04:32:04 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64le: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le
Mar 19 04:32:08 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for probing: /usr/bin/qemu-system-ppc64: relocation error: /usr/
Mar 19 04:32:08 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64 for
Mar 19 04:32:08 b2a-1 libvirtd[5732]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le for probing: /usr/bin/qemu-system-ppc64le: relocation error: /
Mar 19 04:32:08 b2a-1 libvirtd[5732]: Failed to probe capabilities for /usr/bin/qemu-system-ppc64le: internal error: Failed to start QEMU binary /usr/bin/qemu-system-ppc64le
Mar 19 04:32:14 b2a-1 libvirtd[5732]: Failed to open file '/sys/class/net/tap529b886c-d1/operstate': No such file or directory
Mar 19 04:32:14 b2a-1 libvirtd[5732]: unable to read: /sys/class/net/tap529b886c-d1/operstate: No such file or directory
-------------------------------------

Please help me out with this issue.

Thanks
Kumar

Revision history for this message

Balazs Gibizer (balazs-gibizer) wrote on 2020-03-20:

This seems like a QEMU problem that leads to nova not knowing how much resources the hypervisor has.

Revision history for this message

YG Kumar (ygk-kmr) wrote on 2020-03-20:

root@b2a-1:~# /usr/bin/qemu-system-ppc64
/usr/bin/qemu-system-ppc64: relocation error: /usr/bin/qemu-system-ppc64: symbol fdt_check_full version LIBFDT_1.2 not defined in file libfdt.so.1 with link time reference

Revision history for this message

Kashyap Chamarthy (kashyapc) wrote on 2020-03-20:

This looks like your installation of QEMU is completely broken.

As confirmed on IRC:

/usr/bin/qemu-system-ppc64: relocation error: /usr/bin/qemu-system-ppc64: symbol fdt_check_full version LIBFDT_1.2 not defined in file libfdt.so.1 with link time reference

Please consult Ubuntu's guidance to re-install virtualization packages properly.

Changed in nova:
status:	New → Invalid

Revision history for this message

YG Kumar (ygk-kmr) wrote on 2020-03-20:

Download full text (5.4 KiB)

I have removed the qemu and libvirtd packages and reran the nova playbook. Now the libvirtd service is running fine but the nova-compute service is still showing these errors:

I have removed the qemu and libvirtd packages and reran the nova playbook. Now the libvirtd service is running fine but the nova-compute service is still showing these errors:

----------
2020-03-20 07:27:53.807 23547 ERROR nova.scheduler.client.report [req-cf316419-0db4-4916-9fa8-71392eacb052 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f. Got 503: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
.
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager [req-cf316419-0db4-4916-9fa8-71392eacb052 - - - - -] Error updating resources for node b2a-1.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID ea3af0ff-fb69-449a-b484-5ac722010a1f
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager Traceback (most recent call last):
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/compute/manager.py", line 7928, in _update_available_resource_for_node
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 720, in update_available_resource
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     return f(*args, **kwargs)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 797, in _update_available_resource
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     self._update(context, cn)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/retrying.py", line 206, in call
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     return attempt.get(self._wrap_exception)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/retrying.py", line 247, in get
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/retrying.py", line 200, in call
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 959, in _update
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     self._update_to_placement(context, compute_node)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 911, in _update_to_placement
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     parent_provider_uuid=parent_provider_uuid)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 668, in _ensure_resource_provider
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     rps_to_refresh = self._get_providers_in_tree(context, uuid)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 74, in wrapper
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     return f(self, *a, **k)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager   File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 535, in _get_providers_in_tree
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager     raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID ea3af0ff-fb69-449a-b484-5ac722010a1f
2020-03-20 07:27:53.808 23547 ERROR nova.compute.manager
---------------------

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.