Mirantis OpenStack

Heat deployment in HA mode: different value for auth_encryption_key parameter on each controller node

Bug #1387345 reported by Timur Nurlygayanov on 2014-10-29

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Fix Released	High	Igor Yozhikov	Mirantis OpenStack 6.0
5.0.x	Won't Fix	High	Igor Yozhikov	Mirantis OpenStack 5.0-updates
5.1.x	Fix Released	High	Igor Yozhikov	Mirantis OpenStack 5.1.1
6.0.x	Fix Released	High	Igor Yozhikov	Mirantis OpenStack 6.0

Bug Description

Note: reproduced on Fuel 5.1, Fuel 5.1.1 and Fuel 6.0.

Steps To Reproduce:
1. Deploy OpenStack in HA mode.
2. Check /etc/heat/heat.conf on each controller.

Expected Result:
We have identical configuration files on each controller and we can see that Heat engine works on all controllers.

Observed Result:
Heat engine works only on one controller and e have different values for parameter 'auth_encryption_key' on all OpenStack controllers. As the result - Heat can't deploy stacks if several Heat engines work in parallel and it will fail if active controller with Heat engine will be lost.

See original description

Tags:

Timur Nurlygayanov (tnurlygayanov) on 2014-10-29

description:

updated

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2014-10-29:

The root of this problem in https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/heat/manifests/engine.pp#L152

Pavlo Shchelokovskyy (pshchelo) on 2014-10-29

summary:	- Heat deployment in HA mode: different value for auth_encryption_pass + Heat deployment in HA mode: different value for auth_encryption_key parameter on each controller node
description:	updated

Revision history for this message

Andrey Danin (gcon-monolake) wrote on 2014-10-29:

The problem code was removed from upstream manifests 30 Dec 2013 https://github.com/stackforge/puppet-heat/commit/9a885b068b90e1e9d89a6ceee7b857b06fc090ea
So, first of all, we should merge the upstream manifests.
Second, we need to add a new AttributeGenerator here https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/utils/__init__.py#L66 to generate HEX-strings. An existing 'password' generator isn't fit for this case, because Heat wants a 16-bytes key presented as a HEX-string.
Third, we need add a new generated value into openstack.yaml https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/fixtures/openstack.yaml#L969 and reuse it in Puppet 'osnailyfacter' module to propagate into openstack::heat class.

Revision history for this message

Sergey Kraynev (skraynev) wrote on 2014-10-30:

This bug is related with another one https://bugs.launchpad.net/fuel/+bug/1386318.

I assigned this bug on Anastasia, because both bugs require same deployment architecture.
On the first step we plan to reproduce these bugs and check how it works.
The next step is to assign these bugs on Igor Yozhikov, who will upload necessary fixes.

Revision history for this message

Anastasia Kuznetsova (akuznetsova) wrote on 2014-11-10:

I checked this issue together with https://bugs.launchpad.net/fuel/+bug/1386318 .
So for successful Heat HA work first of all need to fix mentioned earlier bug.

Now if we have wrong auth_encryption_keys then complex templates will fail.

For example I verified this bug using heat_autoscaling OSTF test (firstly I unskipped this test ).
Expected result:
1) heat-engines on all controllers are running and have the same auth_encryption_key
2) during autoscaling_test one of engines (on primary controller) completed stack creation, autoscaling and transfer control to another engine that execute stack deletion

Observed result:
1) heat-engines on all controllers are running(after a manual start ) and have the different auth_encryption_keys
2) during autoscaling_test one of engines (on primary controller) completed stack creation, autoscaling and stop its execution and transfer control to another engine that can't delete stack that was created by first engine because of database access problem (different encryption keys).

So need to fix mechanism of keys generation to avoid such problems.

Revision history for this message

Igor Yozhikov (iyozhikov) wrote on 2014-11-11:

6.0
* fuel-web: https://review.openstack.org/#/c/133455/
* fuel-library: https://review.openstack.org/#/c/133263/