hook failed: "certificates-relation-changed" for vault:certificates due to SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_XXX' does not exist or is empty

Bug #1822952 reported by Frode Nordahl
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
Fix Released
Critical
Frode Nordahl

Bug Description

Steps to reproduce:
1. Deploy https://jujucharms.com/openstack-base/ modified to use next charms
2. Add Vault and initialize according to https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/rocky/app-vault.html
3. Add certificate lifecycle management according to https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/rocky/app-certificate-management.html

The Keystone charm will never finish the process.

In the logs you will find this:
2019-04-03 07:08:28 DEBUG certificates-relation-changed Enabling site openstack_https_frontend.
2019-04-03 07:08:28 DEBUG certificates-relation-changed To activate the new configuration, you need to run:
2019-04-03 07:08:28 DEBUG certificates-relation-changed systemctl reload apache2
2019-04-03 07:08:28 DEBUG certificates-relation-changed Job for apache2.service failed because the control process exited with error code.
2019-04-03 07:08:28 DEBUG certificates-relation-changed See "systemctl status apache2.service" and "journalctl -xe" for details.

journalctl shows this:
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit apache2.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit apache2.service has begun starting up.
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 apachectl[56435]: AH00526: Syntax error on line 8 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 apachectl[56435]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_172.16.122.10' does not exist or is empty
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 apachectl[56435]: Action 'start' failed.
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 apachectl[56435]: The Apache error log may have more information.
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 systemd[1]: apache2.service: Control process exited, code=exited status=1
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 systemd[1]: apache2.service: Failed with result 'exit-code'.
Apr 03 07:08:28 juju-7aa29b-3-lxd-1 systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit apache2.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit apache2.service has failed.
--
-- The result is RESULT.

Manually starting apache2 will resolve the issue.

This appears to be a race condition where we attempt to (re)start the Apache process before the certificate data is written to disk.

Revision history for this message
Frode Nordahl (fnordahl) wrote :
description: updated
Frode Nordahl (fnordahl)
Changed in charm-keystone:
assignee: nobody → Frode Nordahl (fnordahl)
Frode Nordahl (fnordahl)
Changed in charm-keystone:
status: Triaged → In Progress
Revision history for this message
Frode Nordahl (fnordahl) wrote :

This appears to be happening only when the certificates relation is added post deploy.

I still think this is a critical issue though since following the referenced documentation lead you straight into this bug.

The issue can be consistently be reproduced and I'm working on adding a functional test to this effect along with a fix for the bug.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

The ``certificates-relation-changed`` hook fires without the *.processed_requests relation data set.

In this situation the process_certificates() function will correctly not write any certs, but the hook code still configures and restarts Apache.

The process_certificates() function should be updated to return a success/failure status and the charm should act accordingly.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/651180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (master)

Reviewed: https://review.openstack.org/651180
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=0faecdf97ae12af4949f670eb95c30e9a43648e6
Submitter: Zuul
Branch: master

commit 0faecdf97ae12af4949f670eb95c30e9a43648e6
Author: Frode Nordahl <email address hidden>
Date: Tue Apr 9 11:53:00 2019 +0200

    Defer processing of certificates until cert present

    When ``certificates-relation-changed`` hook is called before the
    certificate data is present on the relation do not attempt to
    configure apache.

    Change-Id: If915451d4b0846023355edcf3a49f643e12c7522
    Closes-Bug: #1822952

Changed in charm-keystone:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-keystone:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.