Juju Charms Collection
percona-cluster package

sstpassword often set to wrong value in cluster and ha relations

Bug #1454317 reported by Liam Young
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Percona Cluster Charm
Fix Released
Wishlist
James Page
OpenStack Percona Cluster Charm 17.08
percona-cluster (Juju Charms Collection)
Invalid
Wishlist
Unassigned

Bug Description

If the sst-password charm config option is not set then the charm generates a password. Unfortunatly in a clustered environment all units race to do this independantly of one another which can result in the wrong password being shared.

For example in this three node deployment, the password is consistent accross the units:

$ juju run --unit percona/0 "relation-get -r cluster:10 mysql-sstuser.passwd percona/1; relation-get -r cluster:10 mysql-sstuser.passwd percona/2;"
P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M
P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M
$ juju run --unit percona/1 "relation-get -r cluster:10 mysql-sstuser.passwd percona/0; relation-get -r cluster:10 mysql-sstuser.passwd percona/2;"
P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M
P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M
$ juju run --unit percona/2 "relation-get -r cluster:10 mysql-sstuser.passwd percona/0; relation-get -r cluster:10 mysql-sstuser.passwd percona/1;"
P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M
P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M

but it's wrong:

$ juju run --service percona "mysql -h localhost -u sstuser --password=P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M"
- MachineId: "10"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: percona/0
- MachineId: "11"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: percona/1
- MachineId: "12"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: percona/2

I added some logging to the charm to see what happened to the password:

$ juju run --service percona "grep -E 'sstuser' /var/log/juju/unit-percona-*.log"
- MachineId: "10"
  Stdout: |
    2015-05-12 15:18:40 DEBUG unit.percona/0.juju-log cmd.go:247 Generating new password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:18:40 DEBUG unit.percona/0.juju-log cmd.go:247 Writing password '2py5Y9JCVtW7XgBf8M2hhmVMpwPybpWd' to '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:18:40 INFO unit.percona/0.juju-log cmd.go:247 Writing file /var/lib/charm/percona/mysql-sstuser.passwd root:root 660
    2015-05-12 15:18:40 DEBUG unit.percona/0.juju-log cmd.go:247 Migrating mysql-sstuser.passwd 2py5Y9JCVtW7XgBf8M2hhmVMpwPybpWd to peer relation
    2015-05-12 15:20:00 DEBUG unit.percona/0.juju-log cmd.go:247 Migrating mysql-sstuser.passwd 2py5Y9JCVtW7XgBf8M2hhmVMpwPybpWd to peer relation
    2015-05-12 15:20:00 DEBUG unit.percona/0.juju-log cmd.go:247 Using existing password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:20:00 DEBUG unit.percona/0.juju-log cmd.go:247 Migrating mysql-sstuser.passwd 2py5Y9JCVtW7XgBf8M2hhmVMpwPybpWd to peer relation
    2015-05-12 15:20:02 DEBUG unit.percona/0.juju-log cmd.go:247 cluster:10: Using existing password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:20:02 DEBUG unit.percona/0.juju-log cmd.go:247 cluster:10: Migrating mysql-sstuser.passwd 2py5Y9JCVtW7XgBf8M2hhmVMpwPybpWd to peer relation
  UnitId: percona/0
- MachineId: "11"
  Stdout: |
    2015-05-12 15:18:39 DEBUG unit.percona/1.juju-log cmd.go:247 Generating new password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:18:39 DEBUG unit.percona/1.juju-log cmd.go:247 Writing password 'P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M' to '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:18:39 INFO unit.percona/1.juju-log cmd.go:247 Writing file /var/lib/charm/percona/mysql-sstuser.passwd root:root 660
    2015-05-12 15:18:39 DEBUG unit.percona/1.juju-log cmd.go:247 Migrating mysql-sstuser.passwd P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M to peer relation
    2015-05-12 15:19:59 DEBUG unit.percona/1.juju-log cmd.go:247 Migrating mysql-sstuser.passwd P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M to peer relation
    2015-05-12 15:19:59 DEBUG unit.percona/1.juju-log cmd.go:247 Using existing password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:19:59 DEBUG unit.percona/1.juju-log cmd.go:247 Migrating mysql-sstuser.passwd P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M to peer relation
    2015-05-12 15:20:06 DEBUG unit.percona/1.juju-log cmd.go:247 cluster:10: Migrating mysql-sstuser.passwd P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M to peer relation
  UnitId: percona/1
- MachineId: "12"
  Stdout: |
    2015-05-12 15:18:44 DEBUG unit.percona/2.juju-log cmd.go:247 Generating new password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:18:44 DEBUG unit.percona/2.juju-log cmd.go:247 Writing password 'PMkbqmzm8Jt5jKK8KWHrRpdF3mg2RCPp' to '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:18:44 INFO unit.percona/2.juju-log cmd.go:247 Writing file /var/lib/charm/percona/mysql-sstuser.passwd root:root 660
    2015-05-12 15:18:44 DEBUG unit.percona/2.juju-log cmd.go:247 Migrating mysql-sstuser.passwd PMkbqmzm8Jt5jKK8KWHrRpdF3mg2RCPp to peer relation
    2015-05-12 15:20:06 DEBUG unit.percona/2.juju-log cmd.go:247 Migrating mysql-sstuser.passwd PMkbqmzm8Jt5jKK8KWHrRpdF3mg2RCPp to peer relation
    2015-05-12 15:20:06 DEBUG unit.percona/2.juju-log cmd.go:247 Using existing password file '/var/lib/charm/percona/mysql-sstuser.passwd'
    2015-05-12 15:20:06 DEBUG unit.percona/2.juju-log cmd.go:247 Migrating mysql-sstuser.passwd PMkbqmzm8Jt5jKK8KWHrRpdF3mg2RCPp to peer relation
    2015-05-12 15:20:08 DEBUG unit.percona/2.juju-log cmd.go:247 cluster:10: Migrating mysql-sstuser.passwd PMkbqmzm8Jt5jKK8KWHrRpdF3mg2RCPp to peer relation
  UnitId: percona/2

Each unit generated it's own password and percona/0 actually had the correct password before the peer stored value was overwritten by percona/1:

juju run --service percona "mysql -h localhost -u sstuser --password=2py5Y9JCVtW7XgBf8M2hhmVMpwPybpWd -e 'select now() from dual;'"
- MachineId: "10"
  Stdout: |
    now()
    2015-05-12 15:45:53
  UnitId: percona/0
- MachineId: "11"
  Stdout: |
    now()
    2015-05-12 15:45:53
  UnitId: percona/1
- MachineId: "12"
  Stdout: |
    now()
    2015-05-12 15:45:53
  UnitId: percona/2

Unsuprisingly the password is incorrect in the ha relation as well:
$ juju run --unit mysql-hacluster/0 "relation-get -r ha:11 - percona/0 | grep -Eoh 'password.*'"
password="P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M"
$ juju run --unit mysql-hacluster/1 "relation-get -r ha:11 - percona/1 | grep -Eoh 'password.*'"
password="P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M"
$ juju run --unit mysql-hacluster/2 "relation-get -r ha:11 - percona/2 | grep -Eoh 'password.*'"
password="P2cqSpzb7Jk6fhwj8cJ6Sdydj3TPcj7M"

See original description
Tags: openstack oil

Related branches

lp:~gnuoy/charms/trusty/percona-cluster/1454317
Edward Hope-Morley: Needs Resubmitting
Mario Splivalo (community): Approve
OpenStack Charmers: Pending requested
Diff: 501 lines (+210/-40)
11 files modified
charm-helpers-hooks.yaml (+1/-1)
charmhelpers/contrib/database/mysql.py (+21/-6)
charmhelpers/contrib/network/ip.py (+5/-3)
charmhelpers/core/hookenv.py (+32/-0)
charmhelpers/core/hugepage.py (+8/-1)
charmhelpers/core/strutils.py (+30/-0)
hooks/percona_hooks.py (+21/-3)
hooks/percona_utils.py (+12/-9)
tests/charmhelpers/contrib/amulet/utils.py (+47/-16)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+1/-1)
tests/charmhelpers/core/hookenv.py (+32/-0)
lp:~gnuoy/charm-helpers/no-create-if-none
Liam Young (gnuoy)
description: updated
Changed in percona-cluster (Juju Charms Collection):
importance: Undecided → Critical
assignee: nobody → Liam Young (gnuoy)
importance: Critical → High
status: New → Confirmed
Revision history for this message
Liam Young (gnuoy) wrote :

Another side effect is that adding a new unit updates the peer relation with the incorrest password:

$ juju run --unit percona/0 "cluster_id=\$(relation-ids cluster); relation-get -r \$cluster_id mysql-sstuser.passwd percona/1; relation-get -r \$cluster_id mysql-sstuser.p
asswd percona/2;"
swZBF8PcwfnjgNrhZkbwYY3rTqKGF3YN
swZBF8PcwfnjgNrhZkbwYY3rTqKGF3YN
$ juju add-unit percona
$ juju run --unit percona/0 "cluster_id=\$(relation-ids cluster); relation-get -r \$cluster_id mysql-sstuser.passwd percona/1; relation-get -r \$cluster_id mysql-sstuser.passwd percona/2;"

yS2h4ywcrY6TPR3xHgdk2m7RYSzpbd2x
yS2h4ywcrY6TPR3xHgdk2m7RYSzpbd2x
$ juju run --unit percona/0,percona/1,percona/2 "mysql -h localhost -u sstuser --password=yS2h4ywcrY6TPR3xHgdk2m7RYSzpbd2x -e 'select now() from dual;'" - MachineId: "15"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: percona/0
- MachineId: "16"
  ReturnCode: 1
  Stderr: "Warning: Permanently added '10.5.32.89' (ECDSA) to the list of known hosts.\r\nERROR
    1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)\n"
  Stdout: ""
  UnitId: percona/1
- MachineId: "17"
  ReturnCode: 1
  Stderr: "Warning: Permanently added '10.5.32.90' (ECDSA) to the list of known hosts.\r\nERROR
    1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)\n"
  Stdout: ""
  UnitId: percona/2

Edward Hope-Morley (hopem)
Changed in percona-cluster (Juju Charms Collection):
milestone: none → 15.07
tags: added: openstack
Changed in percona-cluster (Juju Charms Collection):
status: Confirmed → In Progress
James Page (james-page)
Changed in percona-cluster (Juju Charms Collection):
milestone: 15.07 → 15.10
Revision history for this message
Liam Young (gnuoy) wrote :

Unsurprisingly the same bug is present when leadership election is available:

$ juju run --service mysql "leader-get mysql.passwd"
- MachineId: "1"
  Stdout: |
    P4sKrbhXb6KkLJF7C9dmSWSxY5WS2rn6
  UnitId: mysql/0
- MachineId: "2"
  Stdout: |
    P4sKrbhXb6KkLJF7C9dmSWSxY5WS2rn6
  UnitId: mysql/1
- MachineId: "3"
  Stdout: |
    P4sKrbhXb6KkLJF7C9dmSWSxY5WS2rn6
  UnitId: mysql/2

$ juju run --service mysql "mysql -h localhost -u sstuser --password=P4sKrbhXb6KkLJF7C9dmSWSxY5WS2rn6"
- MachineId: "1"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: mysql/0
- MachineId: "2"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: mysql/1
- MachineId: "3"
  ReturnCode: 1
  Stderr: |
    ERROR 1045 (28000): Access denied for user 'sstuser'@'localhost' (using password: YES)
  Stdout: ""
  UnitId: mysql/2

Revision history for this message
Liam Young (gnuoy) wrote :

The requirement to explicitly set an sst-password is explained in the README. It would be nice to have the sst password auto generated in the future.

Changed in percona-cluster (Juju Charms Collection):
importance: High → Wishlist
status: In Progress → Confirmed
milestone: 15.10 → none
Larry Michel (lmic)
tags: added: oil
James Page (james-page)
Changed in percona-cluster (Juju Charms Collection):
status: Confirmed → Triaged
James Page (james-page)
Changed in charm-percona-cluster:
assignee: nobody → Liam Young (gnuoy)
importance: Undecided → Wishlist
status: New → Triaged
Changed in percona-cluster (Juju Charms Collection):
status: Triaged → Invalid
Revision history for this message
James Page (james-page) wrote :

I have a review that will resolve this be introducing automatic leader led password generation for both the sst and root passwords.

Changed in charm-percona-cluster:
assignee: Liam Young (gnuoy) → nobody
Changed in percona-cluster (Juju Charms Collection):
assignee: Liam Young (gnuoy) → nobody
Changed in charm-percona-cluster:
assignee: nobody → James Page (james-page)
Changed in percona-cluster (Juju Charms Collection):
assignee: nobody → James Page (james-page)
assignee: James Page (james-page) → nobody
Changed in charm-percona-cluster:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-percona-cluster (master)

Fix proposed to branch: master
Review: https://review.openstack.org/440535

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-percona-cluster (master)

Reviewed: https://review.openstack.org/440535
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=60e58482f6a5887e178e3e4712f7c04d70ff7d00
Submitter: Jenkins
Branch: master

commit 60e58482f6a5887e178e3e4712f7c04d70ff7d00
Author: James Page <email address hidden>
Date: Fri Feb 24 14:34:42 2017 -0500

    Improve password management for clustered deploys

    In the past, its mandatory to provide the sst and root password
    configuration options for clustered deployments to ensure consistent
    use of passwords across the cluster from install onwards.

    Rework password management and install process to seed passwords
    from the lead unit if not supplied via configuration options.

    Following units will defer installation until the leader has
    stored this information in leader storage for retrieval by
    followers.

    Closes-Bug: 1454317

    Change-Id: I5ab70cae78ed35322bf60048af841de071a69704

Changed in charm-percona-cluster:
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-percona-cluster:
milestone: none → 17.08
James Page (james-page)
Changed in charm-percona-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.