tripleo

openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node

Bug #1904936 reported by wes hayutin on 2020-11-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Steve Baker	tripleo wallaby-1

Bug Description

2020-11-19 16:40:08 | "msg": " No nodes completed introspection successfully!"

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/7ac06d9/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

All the baremetal console logs are empty.. not a good sign.
https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/7ac06d9/logs/baremetal_0-console.log
https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/7ac06d9/logs/baremetal_1-console.log

ERRORS found here:

2020-11-19 16:20:11.704 ERROR /var/log/containers/ironic-inspector/ironic-inspector.log: 8 ERROR ironic_inspector.utils [req-f3c85062-7904-4bcf-a685-be0c6671d7df ce5cf8e655bf40dea455ee91426eda67 233cf286c3734182bd8f2aa9cfc3afd2 - default default] Could not find node e2e592cf-e6c5-474a-acf2-22a77b973088 in cache
2020-11-19 16:20:11.812 ERROR /var/log/containers/ironic-inspector/ironic-inspector.log: 8 ERROR ironic_inspector.utils [req-1ab236d7-0f99-4586-9568-08d7c303ed8e ce5cf8e655bf40dea455ee91426eda67 233cf286c3734182bd8f2aa9cfc3afd2 - default default] Could not find node 93699afc-3b8e-4d7c-8d85-55aaf274733d in cache
2020-11-19 16:20:11.865 ERROR /var/log/containers/ironic-inspector/ironic-inspector.log: 8 ERROR ironic_inspector.utils [req-979854a8-868b-42ad-9916-c3d3dee227df ce5cf8e655bf40dea455ee91426eda67 233cf286c3734182bd8f2aa9cfc3afd2 - default default] Could not find node 062a8ec0-4d28-486c-ba87-fb77acf5eed7 in cache
2020-11-19 16:20:11.969 ERROR /var/log/containers/ironic-inspector/ironic-inspector.log: 8 ERROR ironic_inspector.utils [req-12b65f28-a194-4b6e-9568-12ee8d99a325 ce5cf8e655bf40dea455ee91426eda67 233cf286c3734182bd8f2aa9cfc3afd2 - default default] Could not find node 16e4d0ae-37c5-4f5e-8217-e45bd5e37ac8 in cache
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector [req-61ab396c-b762-4af8-804a-09dba05138f3 - - - - -] Unexpected exception while getting inspection status for node e2e592cf-e6c5-474a-acf2-22a77b973088, will retry later: openstack.exceptions.ResourceNotFound: No Introspection found for e2e592cf-e6c5-474a-acf2-22a77b973088: Client Error for url: http://192.168.24.3:5050/v1/introspection/e2e592cf-e6c5-474a-acf2-22a77b973088, Could not find node e2e592cf-e6c5-474a-acf2-22a77b973088 in cache
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector Traceback (most recent call last):
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/inspector.py", line 328, in _check_status
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector status = _get_client(task.context).get_introspection(node.uuid)
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/openstack/baremetal_introspection/v1/_proxy.py", line 87, in get_introspection
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector return self._get(_introspect.Introspection, introspection)
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 47, in check
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector return method(self, expected, actual, *args, **kwargs)
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 509, in _get
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector resource_type=resource_type.__name__, value=value))
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1461, in fetch
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector self._translate_response(response, **kwargs)
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1158, in _translate_response
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector exceptions.raise_from_response(response, error_message=error_message)
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector File "/usr/lib/python3.6/site-packages/openstack/exceptions.py", line 235, in raise_from_response
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector http_status=http_status, request_id=request_id
2020-11-19 16:20:11.721 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.drivers.modules.inspector openstack.exceptions.ResourceNotFound: No Introspection found for e2e592cf-e6c5-474a-acf2-22a77b973088: Client Error for url: http://192.168.24.3:5050/v1/introspection/e2e592cf-e6c5-474a-acf2-22a77b973088, Could not find node e2e592cf-e6c5-474a-acf2-22a77b973088 in cache

# Inspector and conductor logs

Looks like the nodes are not powering on:

# Nodes were registered
Key mac is deprecated, please use ports.
Key mac is deprecated, please use ports.
Key mac is deprecated, please use ports.
Key mac is deprecated, please use ports.
Successfully registered node UUID e2e592cf-e6c5-474a-acf2-22a77b973088
Successfully registered node UUID 93699afc-3b8e-4d7c-8d85-55aaf274733d
Successfully registered node UUID 062a8ec0-4d28-486c-ba87-fb77acf5eed7
Successfully registered node UUID 16e4d0ae-37c5-4f5e-8217-e45bd5e37ac8

Tags:

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-11-19:

ovb env: https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/7ac06d9/logs/undercloud/home/zuul/workspace/ovb/env-93517-base.yaml.txt.gz

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-11-19:

Steve Baker proposed a revert: https://review.opendev.org/763475

Changed in tripleo:
assignee:	nobody → Steve Baker (steve-stevebaker)

Revision history for this message

Marios Andreou (marios-b) wrote on 2020-11-20:

found another example there https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/a7e698c/job-output.txt

noting that this is the integration featureset1 (logs in the description are pointing to the component job periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master

for both however we then had a green run today

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master
SUCCESS @ https://review.rdoproject.org/zuul/build/01a8fdd9b95b491a8a86af2cf40b94ae

and

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master
SUCCESS @ https://review.rdoproject.org/zuul/build/a83ffdff95564d74a3e9a062f9eac86b

so looks like we may not need the revert from comment #2 ?

Revision history for this message

Marios Andreou (marios-b) wrote on 2020-11-20:

ok so perhaps I spoke too soon... I just came across another example of that but in fs35 [1]. There are four failed jobs in that buildset [2] but only this one example of this bug seen.

so is it possible for this to be a non consistent issue? but i cannot see how something like https://review.opendev.org/#/c/763475/1/tripleo_ansible/ansible_plugins/modules/os_tripleo_baremetal_node_introspection.py could lead to a non consistent error... sbaker do you agree?

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/5b69c53/logs/bmc-console.log
[2] https://review.rdoproject.org/zuul/buildset/1d36382481824b22b1d99d35b12c9f2e

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2020-11-22:

Looking at the most recent periodic runs[1], about 50% are failing with this issue.

I think the rewritten module is fine, but this is revealing an issue with triggering inspect via the Ironic API (instead of through the inspector API).

Feel free to approve the revert[2] if you want a better than 50% fail rate while I investigate this issue

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/?C=M;O=A
[2] https://review.opendev.org/c/openstack/tripleo-ansible/+/763475

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-11-23:

See three jobs failing in recent periodic runs:

https://review.rdoproject.org/zuul/buildset/9f50ea89235f4983b68ca25a884c815f
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master FAILURE true 3 hrs 3 mins 47 secs
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master FAILURE true 1 hr 24 mins 23 secs
periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master FAILURE true 1 hr 57 mins 0 secs
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master FAILURE true 2 hrs 8 mins 59 secs