ocata/stable jobs are broken (httpd restart failures)

Mar 15 13:46:52.550010 centos-7-2-node-rax-ord-7889550-474007 python[206653]: Found 'compress' tags in:
Mar 15 13:46:52.550370 centos-7-2-node-rax-ord-7889550-474007 python[206653]: /usr/share/openstack-dashboard/openstack_dashboard/templates/horizon/_conf.html
Mar 15 13:46:52.550629 centos-7-2-node-rax-ord-7889550-474007 python[206653]: /usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html
Mar 15 13:46:52.551032 centos-7-2-node-rax-ord-7889550-474007 python[206653]: /usr/share/openstack-dashboard/openstack_dashboard/templates/horizon/_scripts.html
Mar 15 13:46:52.551287 centos-7-2-node-rax-ord-7889550-474007 python[206653]: Compressing... done
Mar 15 13:46:52.551603 centos-7-2-node-rax-ord-7889550-474007 python[206653]: Compressed 6 block(s) from 3 template(s) for 2 context(s).
Mar 15 13:46:52.669540 centos-7-2-node-rax-ord-7889550-474007 httpd[206837]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:8000
Mar 15 13:46:52.670055 centos-7-2-node-rax-ord-7889550-474007 httpd[206837]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:8000
Mar 15 13:46:52.670275 centos-7-2-node-rax-ord-7889550-474007 httpd[206837]: no listening sockets available, shutting down
Mar 15 13:46:52.670498 centos-7-2-node-rax-ord-7889550-474007 httpd[206837]: AH00015: Unable to open logs
Mar 15 13:46:52.686015 centos-7-2-node-rax-ord-7889550-474007 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Mar 15 13:46:52.715206 centos-7-2-node-rax-ord-7889550-474007 kill[206840]: kill: cannot find process ""
Mar 15 13:46:52.720167 centos-7-2-node-rax-ord-7889550-474007 systemd[1]: httpd.service: control process exited, code=exited status=1
Mar 15 13:46:52.720977 centos-7-2-node-rax-ord-7889550-474007 systemd[1]: Failed to start The Apache HTTP Server.
Mar 15 13:46:52.721219 centos-7-2-node-rax-ord-7889550-474007 systemd[1]: Unit httpd.service entered failed state.
Mar 15 13:46:52.721292 centos-7-2-node-rax-ord-7889550-474007 systemd[1]: httpd.service failed.

Ok so the problem is heat is binding to the * address instead of the internal ip:
etc/httpd/conf.d/15-default.conf:<VirtualHost 192.168.24.6:80>
etc/httpd/conf.d/10-placement_wsgi.conf:<VirtualHost 192.168.24.6:8778>
etc/httpd/conf.d/10-keystone_wsgi_main.conf:<VirtualHost 192.168.24.6:5000>
etc/httpd/conf.d/10-keystone_wsgi_admin.conf:<VirtualHost 192.168.24.6:35357>
etc/httpd/conf.d/10-heat_api_wsgi.conf:<VirtualHost *:8004>
etc/httpd/conf.d/10-heat_api_cfn_wsgi.conf:<VirtualHost *:8000>
etc/httpd/conf.d/10-heat_api_cloudwatch_wsgi.conf:<VirtualHost *:8003>
etc/httpd/conf.d/10-cinder_wsgi.conf:<VirtualHost 192.168.24.6:8776>
etc/httpd/conf.d/10-horizon_vhost.conf:<VirtualHost 192.168.24.6:80>

Spoke to Juan and this seems more of a CI issue because we do not deploy heat under httpd in ocata:
 jaosorior ╡ bandini: then yeah,s eems something is busted on CI
 jaosorior ╡ bandini: no upgrade step in stable/ocata even references
           ╡ heat over httpd

https://review.openstack.org/#/c/446121/ landed

Still seeing this as of 1 hour ago

Still seeing this but on stable/ocata multinode job, http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/undercloud/var/log/postci.txt.gz

It looks like subnode-2 is getting master packages, see http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/subnode-2/rpm-qa.txt.gz vs http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/rpm-qa.txt.gz

it's now passing so maybe it was a timing thing