At [1][2][3] the periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussurijob fails during the Controller update with trace like:
* 2020-08-05 11:06:29 | <192.168.24.3> (0, b'\n{"cmd": "CLUSTER_NODE=$(crm_node -n)\\necho \\"Retrieving all the VIPs which are hosted on this node\\"\\nVIPS_TO_MOVE=$(crm_mon --as-xml | xmllint --xpath \'//resource[@resource_agent = \\"ocf::heartbeat:IPaddr2\\" and @role = \\"Started\\" and @managed = \\"true\\" and ./node[@name = \\"\'${CLUSTER_NODE}\'\\"]]/@id\' - | sed -e \'s/id=//g\' -e \'s/\\"//g\')\\nfor v in ${VIPS_TO_MOVE}; do\\n echo \\"Moving VIP $v on another node\\"\\n pcs resource move $v --wait=300\\ndone\\necho \\"Removing the location constraints that were created to move the VIPs\\"\\nfor v in ${VIPS_TO_MOVE}; do\\n echo \\"Removing location ban for VIP $v\\"\\n ban_id=$(cibadmin --query | xmllint --xpath \'string(//rsc_location[@rsc=\\"\'${v}\'\\" and @node=\\"\'${CLUSTER_NODE}\'\\" and @score=\\"-INFINITY\\"]/@id)\' -)\\n if [ -n \\"$ban_id\\" ]; then\\n pcs constraint remove ${ban_id}\\n else\\n echo \\"Could not retrieve and clear location constraint for VIP $v\\" 2>&1\\n fi\\ndone\\n", "stdout": "Retrieving all the VIPs which are hosted on this node\\nMoving VIP ip-192.168.24.16 on another node\\nWarning: Creating location constraint \'cli-ban-ip-192.168.24.16-on-node-0000333831\' with a score of -INFINITY for resource ip-192.168.24.16 on node-0000333831.\\n\\tThis will prevent ip-192.168.24.16 from running on node-0000333831 until the constraint is removed\\n\\tThis will be the case even if node-0000333831 is the last node in the cluster\\nRemoving the location constraints that were created to move the VIPs\\nRemoving location ban for VIP ip-192.168.24.16", "stderr": "Error: resource \'ip-192.168.24.16\' is not running on any node", "rc": 0, "start": "2020-08-05 11:06:25.880027", "end": "2020-08-05 11:06:29.046251", "delta": "0:00:03.166224", "changed": true, "invocation": {"module_args": {"_raw_params": "CLUSTER_NODE=$(crm_node -n)\\necho \\"Retrieving all the VIPs which are hosted on this node\\"\\nVIPS_TO_MOVE=$(crm_mon --as-xml | xmllint --xpath \'//resource[@resource_agent = \\"ocf::heartbeat:IPaddr2\\" and @role = \\"Started\\" and @managed = \\"true\\" and ./node[@name = \\"\'${CLUSTER_NODE}\'\\"]]/@id\' - | sed -e \'s/id=//g\' -e \'s/\\"//g\')\\nfor v in ${VIPS_TO_MOVE}; do\\n echo \\"Moving VIP $v on another node\\"\\n pcs resource move $v --wait=300\\ndone\\necho \\"Removing the location constraints that were created to move the VIPs\\"\\nfor v in ${VIPS_TO_MOVE}; do\\n echo \\"Removing location ban for VIP $v\\"\\n ban_id=$(cibadmin --query | xmllint --xpath \'string(//rsc_location[@rsc=\\"\'${v}\'\\" and @node=\\"\'${CLUSTER_NODE}\'\\" and @score=\\"-INFINITY\\"]/@id)\' -)\\n if [ -n \\"$ban_id\\" ]; then\\n pcs constraint remove ${ban_id}\\n else\\n echo \\"Could not retrieve and clear location constraint for VIP $v\\" 2>&1\\n fi\\ndone\\n", "_uses_shell": true, "warn": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\n', b'')
* 2020-08-05 11:06:29 | "stderr": "Error: resource 'ip-192.168.24.16' is not running on any node",
This does NOT happen in gates luckily appears to be periodic only gates green [4]
[1] https://logserver.rdoproject.org/90/28890/1/check/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/019fa3b/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/dba0e70/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/1442873/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz
[4] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri&pipeline=gate
By looking at the bottom of the logs:https:/
["mysql_
2020-08-05 11:28:12 | [ERROR]: Container(s) which failed to be created by podman_container module:
2020-08-05 11:28:12 | ['mysql_
2020-08-05 11:28:12 | [ERROR]: Container(s) which did not finish after 300 minutes:
2020-08-05 11:28:12 | ['mysql_
2020-08-05 11:28:12 | 2020-08-05 11:28:12.188714 | fa163ec9-
2020-08-05 11:28:12 | "changed": false,
2020-08-05 11:28:12 | "msg": "Failed container(s): ['mysql_
Under https:/
and https:/
Error: Facter: error while resolving custom fact "rabbitmq_