tripleo

Deployment fails when MongoDB runs on a different node than controller with missing bootstrap_nodeid hiera

Bug #1620485 reported by Marius Cornea
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Emilien Macchi
tripleo newton-rc1 "newton-rc1"

Bug Description

Deployment fails when MongoDB runs on a different node than controller with missing bootstrap_nodeid hiera:

[stack@undercloud ~]$ cat templates/deploy.command
source ~/stackrc
#export THT=/usr/share/openstack-tripleo-heat-templates
export THT=~/templates/tripleo-heat-templates/
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e ~/templates/services.yaml \
--control-scale 3 \
--control-flavor controller \
--compute-scale 1 \
--compute-flavor compute \
--block-storage-scale 3 \
--block-storage-flavor block \
--ntp-server clock.redhat.com
[stack@undercloud ~]$ cat templates/services.yaml
parameter_defaults:
  ControllerServices:
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall

  BlockStorageServices:
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncproxy
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall

Deployment fails with:

vercloud.BlockStorageNodesPostDeployment.VolumeDeployment_Step2.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 197086ca-dfcb-440d-816f-b1d20a2985c1
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |

  deploy_stderr: |
    exception: connect failed
    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/database/mongodb.pp:35 on node overcloud-blockstorage-1.localdomain
    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/database/mongodb.pp:35 on node overcloud-blockstorage-1.localdomain
overcloud.BlockStorageNodesPostDeployment.VolumeDeployment_Step2.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 6fd9730c-25be-46a0-912f-de360b7462e4
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |

  deploy_stderr: |
    exception: connect failed
    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/database/mongodb.pp:35 on node overcloud-blockstorage-0.localdomain
    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/database/mongodb.pp:35 on node overcloud-blockstorage-0.localdomain
overcloud.BlockStorageNodesPostDeployment.VolumeDeployment_Step2.2:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: c26abdc5-3730-4364-856e-df99989839f6
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |

  deploy_stderr: |
    exception: connect failed
    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/database/mongodb.pp:35 on node overcloud-blockstorage-2.localdomain
    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/database/mongodb.pp:35 on node overcloud-blockstorage-2.localdomain
overcloud.ComputeNodesPostDeployment.ComputeOvercloudServicesDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 4d051716-740d-4f68-9d71-baf37f2bedd7
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None

Revision history for this message
Marius Cornea (mcornea) wrote :

The same error shows up for Heat:

    Error: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/heat.pp:43 on node overcloud-blockstorage-2.localdomain

Revision history for this message
Steven Hardy (shardy) wrote :

Ok, something must be wrong with the hieradata or hierarchy, as the bootstrap node data should be set for all roles since https://review.openstack.org/#/c/337346/

Can you "grep bootstrap /etc/puppet/hieradata/* ?

Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → newton-rc1
tags: added: compposable-roles
tags: added: composable-roles
removed: compposable-roles
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/366049

Changed in tripleo:
assignee: nobody → Steven Hardy (shardy)
status: Triaged → In Progress
OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Steven Hardy (shardy) → Juan Antonio Osorio Robles (juan-osorio-robles)
OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Juan Antonio Osorio Robles (juan-osorio-robles) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/366049
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=87f1f7d26abbc65f3095faa28b9e1f62d96ff0ee
Submitter: Jenkins
Branch: master

commit 87f1f7d26abbc65f3095faa28b9e1f62d96ff0ee
Author: Steven Hardy <email address hidden>
Date: Tue Sep 6 11:45:28 2016 +0100

    Add bootstrap_node and vip_data to hierarchy for all roles

    We recently made changes that add data to allNodesConfig, but
    we didn't wire the files into the hierarchy on all roles.

    Change-Id: I8e838b02bd982e600af54b14350106322244890a
    Closes-Bug: #1620485

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.