If you deploy using a separate undercloud with separate overcloud nodes and not using the standalone installer then the ceph deployment fails on the following task:
- name: genereate create_ceph_ansible_remote_tmp playbook
and looking at the logs in /home/stack/config-download/config-download-latest/ceph-ansible we can see that the overcloud nodes tried to change ownership of the temp directory to the stack user which doesn't exist on the overcloud nodes [1]
This bug was introduced by:
https://review.opendev.org/#/c/739521/
We didn't see it in CI because the standalone hides this issue.
[1]
[CentOS-8.2 - stack@undercloud ceph-ansible]$ head -15 create_ceph_ansible_remote_tmp.log
2020-07-15 17:56:49,992 p=399413 u=root n=ansible | [WARNING]: Skipping key (deprecated) in group (overcloud) as it is not a
mapping, it is a <class 'ansible.parsing.yaml.objects.AnsibleUnicode'>
2020-07-15 17:56:50,294 p=399413 u=root n=ansible | PLAY [all] *********************************************************************
2020-07-15 17:56:50,306 p=399413 u=root n=ansible | TASK [create ceph_ansible_remote_tmp on all nodes with necessary ownership] ****
2020-07-15 17:56:50,306 p=399413 u=root n=ansible | Wednesday 15 July 2020 17:56:50 +0000 (0:00:00.026) 0:00:00.026 ********
2020-07-15 17:56:50,643 p=399413 u=root n=ansible | changed: [undercloud]
2020-07-15 17:56:51,089 p=399413 u=root n=ansible | fatal: [oc0-controller-2]: FAILED! => changed=false
gid: 0
group: root
mode: '0755'
msg: 'chown failed: failed to look up user stack'
owner: root
path: /tmp/ceph_ansible_tmp
secontext: unconfined_u:object_r:user_tmp_t:s0
[CentOS-8.2 - stack@undercloud ceph-ansible]$
This is not something we want to run on all overcloud nodes:
[CentOS-8.2 - stack@undercloud ceph-ansible]$ cat create_
- hosts: all
gather_facts: no
tasks:
# Avoiding the following by creating directory owned by user who will
# SSH into nodes (not root). When root needs to write to this directory
# it will not have permission problems by definition. As per ansible:
# """
# Module remote_tmp /tmp/ceph_
# with a mode of 0700, this may cause issues when running as another user.
# To avoid this, create the remote_tmp dir with the correct permissions
# manually.
# """
- name: create ceph_ansible_
become: true
file:
path: "/tmp/ceph_
owner: "stack"
group: "stack"
mode: "700"
state: directory
[CentOS-8.2 - stack@undercloud ceph-ansible]$