tripleo

standalone deployment victoria failing on AttributeError: 'NoneType' object has no attribute 'items'

Bug #1907006 reported by wes hayutin on 2020-12-06

This bug report is a duplicate of: Bug #1907122: gating repo improperly enabled across releases, enabling the wrong patches to be integrated into build. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Triaged	Critical	Unassigned	tripleo wallaby-2

Bug Description

2020-12-06 20:28:17 |
2020-12-06 20:28:17 | Stack standalone/85a54a77-84a7-4855-88e5-69f5c961bd23 CREATE_COMPLETE
2020-12-06 20:28:17 |
2020-12-06 20:28:17 | 2020-12-06 20:28:17.554 176915 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] ** Downloading standalone ansible.. **[00m
2020-12-06 20:28:20 | 2020-12-06 20:28:20.059 176915 INFO tripleo_common.utils.config.Config [-] Getting deployment data from Heat...[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.776 176915 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] Not cleaning working directory /home/zuul/tripleo-heat-installer-templates[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.813 176915 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] Not cleaning ansible directory /home/zuul/standalone-ansible-_nur4qqv[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.841 176915 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] Install artifact is located at /home/zuul/standalone-install-20201206202825.tar.bzip2[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.842 176915 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] The heat stack standalone virtual state/action is reset to CREATE. Use "--force-stack-update" to set it forcefully to UPDATE[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.842 176915 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] Removing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_standalone[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.843 176915 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-]
2020-12-06 20:28:26 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2020-12-06 20:28:26 |
2020-12-06 20:28:26 | Deployment Failed!
2020-12-06 20:28:26 |
2020-12-06 20:28:26 | ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-2gzw5h3b
2020-12-06 20:28:26 |
2020-12-06 20:28:26 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2020-12-06 20:28:26 | [00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.843 176915 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] Exception: 'NoneType' object has no attribute 'items'[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.848 176915 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] None[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.921 176915 ERROR openstack [-] 'NoneType' object has no attribute 'items': tripleoclient.exceptions.DeploymentError: 'NoneType' object has no attribute 'items'[00m
2020-12-06 20:28:26 | 2020-12-06 20:28:26.925 176915 INFO osc_lib.shell [-] END return value: 1[00m
2020-12-06 20:28:26 | /usr/lib64/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
2020-12-06 20:28:26 | return f(*args, **kwds)
2020-12-06 20:28:26 | Traceback (most recent call last):
2020-12-06 20:28:26 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1421, in take_action
2020-12-06 20:28:26 | self._standalone_deploy(parsed_args)
2020-12-06 20:28:26 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1272, in _standalone_deploy
2020-12-06 20:28:26 | depl_python)
2020-12-06 20:28:26 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 844, in _download_ansible_playbooks
2020-12-06 20:28:26 | self.tmp_ansible_dir)
2020-12-06 20:28:26 | File "/usr/lib/python3.6/site-packages/tripleo_common/utils/config.py", line 562, in write_config
2020-12-06 20:28:26 | self.render_network_config(stack, config_dir, server_roles)
2020-12-06 20:28:26 | File "/usr/lib/python3.6/site-packages/tripleo_common/utils/config.py", line 228, in render_network_config
2020-12-06 20:28:26 | for server, config in network_config.items():
2020-12-06 20:28:26 | AttributeError: 'NoneType' object has no attribute 'items'

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_45d/764090/1/gate/tripleo-ci-centos-8-standalone-upgrade-victoria/45d09a4/logs/undercloud/home/zuul/standalone_upgrade.log

https://a5b95c3ca8f05b423751-efd5ea2c2502e38b3629faa5d1e6d102.ssl.cf1.rackcdn.com/764679/1/gate/tripleo-ci-centos-8-standalone-upgrade-victoria/ae9520e/logs/undercloud/home/zuul/standalone_upgrade.log

2020-12-06 20:22:04,840 p=46633 u=root n=ansible | [WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin
(<ansible.plugins.callback.validation_json.CallbackModule object at
0x7f4ab0449390>): [Errno 2] No such file or directory: '/var/log/validations/fa
163e7c-0fa0-84d1-ce0b-000000000008_deploy_steps_playbook_2020-12-06T19:47:06.77
6885Z.json'

Tags:

Revision history for this message

Marios Andreou (marios-b) wrote on 2020-12-07:

o/ taking over as rover this week so catching up on this new bug...

I cannot see it fail in the periodic/promotion pipeline [1] so I am wondering why it has the promotion-blocker tag I am going to remove it.

Looking at the check/gate fails in [2] this is actually happening during the deployment and the upgrade hasn't started yet [3][4][5][6].

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-victoria
[2] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria
[3] http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6b8/765736/1/check/tripleo-ci-centos-8-standalone-upgrade-victoria/6b8f59c/logs/undercloud/home/zuul/standalone_deploy.log
[4] http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a34/764966/1/check/tripleo-ci-centos-8-standalone-upgrade-victoria/a345247/logs/undercloud/home/zuul/standalone_deploy.log
[5] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e92/765291/1/gate/tripleo-ci-centos-8-standalone-upgrade-victoria/e926d95/logs/undercloud/home/zuul/standalone_deploy.log
[6] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_139/764969/2/check/tripleo-ci-centos-8-standalone-upgrade-victoria/139ca09/logs/undercloud/home/zuul/standalone_deploy.log

tags:

removed: promotion-blocker

wes hayutin (weshayutin) on 2020-12-07

summary:

- standalone-upgrade-victoria failing on AttributeError: 'NoneType' object
- has no attribute 'items'
+ standalone deployment victoria failing on AttributeError: 'NoneType'
+ object has no attribute 'items'

Revision history for this message

Rabi Mishra (rabi) wrote on 2020-12-07:

I also noticed below earlier today for undercloud upgrade:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a72/765736/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/a72b125/logs/undercloud/home/zuul/undercloud_install.log

2020-12-07 04:07:21 | urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/home/zuul/tripleo-heat-installer-templates/environments/undercloud-disable-nova.yaml'>
2020-12-07 04:07:21 | None
2020-12-07 04:07:21 | <urlopen error [Errno 2] No such file or directory: '/home/zuul/tripleo-heat-installer-templates/environments/undercloud-disable-nova.yaml'>
2020-12-07 04:07:22 |

I think it's probably a promotion issue (mismatch between tht/tripleo_common/python-tripleoclient.

The error about the env has been removed from tht but the tripleclient version still needs it for some reason.

https://review.opendev.org/c/openstack/python-tripleoclient/+/749679/2/tripleoclient/v1/undercloud_config.py

Revision history for this message

Marios Andreou (marios-b) wrote on 2020-12-07:

@Rabi I think the issue from comment #2 above might be a different bug?

I see you have merged the removal of that undercloud-disable-nova.yaml file https://review.opendev.org/q/topic:%22bug%252F1891242%22+(status:open%20OR%20status:merged) back in september . SO while it should be in stable/victoria, I think it is not in stable/ussuri (I can't see it having been backported or i missed it).

Revision history for this message

Marios Andreou (marios-b) wrote on 2020-12-07:

so to be clear and as just discussed with weshay and chkumar

the problem @Rabi is commenting on in comment #2 is different (we will file a new bug for that). The problem there is that we are enabling the _victoria_ gating repo during ussuri part of the undercloud upgrade.

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-12-07:

The issue Rabi posted is here: https://bugs.launchpad.net/tripleo/+bug/1907122

wes hayutin (weshayutin) on 2020-12-08

tags:

added: alert

Revision history for this message

Rabi Mishra (rabi) wrote on 2020-12-08:

> @Rabi I think the issue from comment #2 above might be a different bug?

I think it's the same issue i.e. tht vs tripleoclient/tripleo-common versions not in sync. In this case it's old tripleo-common(ussuri) with tht(victoria) as far as I can see from the error.

Revision history for this message

Rabi Mishra (rabi) wrote on 2020-12-08:

To elaborate:

We're trying to get an stack output[1] which is not there is stable/victoria tht overcloud.j2.yaml (would return None) and hence the Attribute error from [2].

[1] https://opendev.org/openstack/tripleo-common/src/branch/stable/ussuri/tripleo_common/utils/config.py#L64

[2] https://opendev.org/openstack/tripleo-common/src/branch/stable/ussuri/tripleo_common/utils/config.py#L228