deployment fails on task "tripleo_ceph_work_dir : add gnocchi pool"

On the system where I hit this bug here's the output from hiera

$ ansible -i inventory.yaml all -m shell -b -a "hostname; sudo /bin/hiera -c
               /etc/puppet/hiera.yaml enabled_services"

undercloud | CHANGED | rc=0 >>
undercloud.localdomain
["kernel",
 "masquerade_networks",
 "mistral_executor",
 "nova_scheduler",
 "oslo_messaging_notify",
 "snmp",
 "mysql",
 "ironic_api",
 "swift_proxy",
 "swift_ringbuilder",
 "neutron_dhcp",
 "heat_api",
 "container_image_prepare",
 "nova_ironic",
 "iscsid",
 "tempest",
 "ironic_pxe",
 "ironic_neutron_agent",
 "tripleo_firewall",
 "swift_storage",
 "keepalived",
 "rsyslog_baremetal_ansible",
 "certmonger_user",
 "ironic_inspector",
 "mysql_client",
 "mistral_event_engine",
 "glance_api",
 "keystone",
 "tripleo_validations",
 "docker_registry",
 "memcached",
 "oslo_messaging_rpc",
 "logrotate_tmpwatch",
 "neutron_plugin_ml2",
 "keystone_public_api",
 "nova_api",
 "ironic_conductor",
 "mistral_api",
 "keystone_admin_api",
 "heat_engine",
 "neutron_api",
 "openstack_clients",
 "podman",
 "nova_conductor",
 "apache",
 "logrotate_crond",
 "neutron_ovs_agent",
 "haproxy",
 "sshd",
 "placement",
 "neutron_l3",
 "zaqar_api",
 "tuned",
 "chrony",
 "timezone",
 "mistral_engine",
 "ca_certs"]
control-plane-computehci-0 | CHANGED | rc=0 >>
control-plane-computehci-0
["kernel",
 "nova_scheduler",
 "snmp",
 "mysql",
 "cinder_api",
 "ceph_client",
 "ceph_mon",
 "swift_ringbuilder",
 "ceph_osd",
 "heat_api",
 "container_image_prepare",
 "timezone",
 "iscsid",
 "boot_params_service",
 "heat_api_cloudwatch_disabled",
 "tripleo_firewall",
 "swift_storage",
 "nova_metadata",
 "keepalived",
 "certmonger_user",
 "mysql_client",
 "glance_api",
 "keystone",
 "cinder_volume",
 "ovn_dbs",
 "memcached",
 "oslo_messaging_rpc",
 "nova_vnc_proxy",
 "oslo_messaging_notify",
 "keystone_public_api",
 "nova_api",
 "keystone_admin_api",
 "heat_engine",
 "nova_migration_target",
 "cinder_scheduler",
 "tripleo_packages",
 "nova_conductor",
 "nova_compute",
 "nova_libvirt_guests",
 "neutron_plugin_ml2_ovn",
 "logrotate_crond",
 "haproxy",
 "sshd",
 "placement",
 "podman",
 "nova_libvirt",
 "ovn_metadata",
 "tuned",
 "swift_proxy",
 "chrony",
 "horizon",
 "neutron_api",
 "ca_certs",
 "heat_api_cfn",
 "ceph_mgr",
 "ovn_controller"]
control-plane-controller-0 | CHANGED | rc=0 >>
control-plane-controller-0
["kernel",
 "nova_scheduler",
 "snmp",
 "mysql",
 "cinder_api",
 "ceph_client",
 "ceph_mon",
 "swift_ringbuilder",
 "ceph_osd",
 "heat_api",
 "container_image_prepare",
 "timezone",
 "iscsid",
 "boot_params_service",
 "heat_api_cloudwatch_disabled",
 "tripleo_firewall",
 "swift_storage",
 "nova_metadata",
 "keepalived",
 "certmonger_user",
 "mysql_client",
 "glance_api",
 "keystone",
 "cinder_volume",
 "ovn_dbs",
 "memcached",
 "oslo_messaging_rpc",
 "nova_vnc_proxy",
 "oslo_messaging_notify",
 "keystone_public_api",
 "nova_api",
 "keystone_admin_api",
 "heat_engine",
 "nova_migration_target",
 "cinder_scheduler",
 "tripleo_packages",
 "nova_conductor",
 "nova_compute",
 "nova_libvirt_guests",
 "neutron_plugin_ml2_ovn",
 "logrotate_crond",
 "haproxy",
 "sshd",
 "placement",
 "podman",
 "nova_libvirt",
 "ovn_metadata",
 "tuned",
 "swi...

I ran into this bug with ansible-playbook 2.8.6
Our CI is currently using ansible-2.8.8-1.el7.noarch

The problem with the hiera command [1] is that it's returning a string.

Even enabled_services_raw.stdout_lines is a one-item-list containing a large string the characters '[' and '\"'.

u'cmd': [u'hiera', u'-c', u'/etc/puppet/hiera.yaml', u'enabled_services'], u'rc': 0, u'start': u'2020-03-11 18:36:26.245088', u'stderr': u'', u'delta': u'0:00:00.080207',

'stdout_lines': [u'[\"kernel\",', u' \"masquerade_networks\",', u' \"mistral_executor\",', u' \"nova_scheduler\",', u' \"oslo_messaging_notify\",', u' \"snmp\",', u' \"mysql\",', u' \"ironic_api\",', u' \"swift_proxy\",', u' \"swift_ringbuilder\",', u' \"neutron_dhcp\",', u' \"heat_api\",', u' \"container_image_prepare\",', u' \"nova_ironic\",', u' \"iscsid\",', u' \"tempest\",', u' \"ironic_pxe\",', u' \"ironic_neutron_agent\",', u' \"tripleo_firewall\",', u' \"swift_storage\",', u' \"keepalived\",', u' \"rsyslog_baremetal_ansible\",', u' \"certmonger_user\",', u' \"ironic_inspector\",', u' \"mysql_client\",', u' \"mistral_event_engine\",', u' \"glance_api\",', u' \"keystone\",', u' \"tripleo_validations\",', u' \"docker_registry\",', u' \"memcached\",', u' \"oslo_messaging_rpc\",', u' \"logrotate_tmpwatch\",', u' \"neutron_plugin_ml2\",', u' \"keystone_public_api\",', u' \"nova_api\",', u' \"ironic_conductor\",', u' \"mistral_api\",', u' \"keystone_admin_api\",', u' \"heat_engine\",', u' \"neutron_api\",', u' \"openstack_clients\",', u' \"podman\",', u' \"nova_conductor\",', u' \"apache\",', u' \"logrotate_crond\",', u' \"neutron_ovs_agent\",', u' \"haproxy\",', u' \"sshd\",', u' \"placement\",', u' \"neutron_l3\",', u' \"zaqar_api\",', u' \"tuned\",', u' \"chrony\",', u' \"timezone\",', u' \"mistral_engine\",', u' \"ca_certs\"]']}"

[1] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/prepare.yml#L42-L44

- The workdir role is included by external_deploy_steps_tasks only [1]
- The external_deploy_steps_tasks are run against the undercloud node only [2]

So, things I should have asked during the code review:
- Do we want to get running hiera commands on the undercloud?
- Why do we need to this when we already have vars set based on THT, e.g. ceph_pools.glance_pool.enabled?

[1]
(control-plane) [stack@undercloud config-download]$ grep -R work_dir *
external_deploy_steps_tasks.yaml: name: tripleo_ceph_work_dir
external_deploy_steps_tasks.yaml: name: tripleo_ceph_work_dir
(control-plane) [stack@undercloud config-download]$

[2] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/ansible_config_download.html#playbooks

Fix proposed to branch: master
Review: https://review.opendev.org/712589

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.opendev.org/712589
Reason: Fixing the bug through a revert of the original patch.
https://review.opendev.org/#/c/712661/
https://review.opendev.org/#/c/712662/

Fix proposed to branch: master
Review: https://review.opendev.org/712701

Reviewed: https://review.opendev.org/712701
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=02ea16fbf2f631869e9366e757d80c71eb33befd
Submitter: Zuul
Branch: master

commit 02ea16fbf2f631869e9366e757d80c71eb33befd
Author: Francesco Pantano <email address hidden>
Date: Thu Mar 12 15:48:03 2020 +0100

    Get the list of enabled_services on the overcloud

    This patch fixes the way we compute the enabled_services
    list on the overcloud.
    We don't need to use hiera anymore since the overcloud
    services are already provided by config-download.
    This patch also fixes the condition to check if a service
    is present in the list.

    Change-Id: I9952a4df122c00c0163b7e79dab8a821e3feff06
    Closes-Bug: #1867004

Fix proposed to branch: stable/train
Review: https://review.opendev.org/713106

Reviewed: https://review.opendev.org/713106
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=0210ba8596d29d9015b6ae77cf1651bade095e58
Submitter: Zuul
Branch: stable/train

commit 0210ba8596d29d9015b6ae77cf1651bade095e58
Author: Francesco Pantano <email address hidden>
Date: Thu Feb 20 18:05:05 2020 +0100

    Add a new set of tasks to build the openstack pool and key list

    Using this approach we can build the list of the openstack
    pools and keys according to the tht variables generated and
    the list of the enabled services provided by config-download.
    This patch also contains 02ea16 that fixes the way we compute
    the enabled_services list on the overcloud and the condition
    to check if a service is present in the list.

    Closes-Bug: #1864477
    Closes-Bug: #1867004
    Change-Id: I80ac1fc325a2d13be1a1e454eee0389510c090b2
    (cherry picked from commit 144f5aa690eb2c6ea6071dc2732c721e6bde4c77)

    Get the list of enabled_services on the overcloud

    This patch fixes the way we compute the enabled_services
    list on the overcloud.
    We don't need to use hiera anymore since the overcloud
    services are already provided by config-download.
    This patch also fixes the condition to check if a service
    is present in the list.

    Change-Id: I9952a4df122c00c0163b7e79dab8a821e3feff06
    (cherry picked from commit 02ea16fbf2f631869e9366e757d80c71eb33befd)

This issue was fixed in the openstack/tripleo-ansible 0.5.0 release.

This issue was fixed in the openstack/tripleo-ansible 1.3.0 release.