OpenStack Percona Cluster Charm

Install fails with TypeError: connect() argument 3 must be string, not Non

Bug #1744961 reported by Liam Young
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Percona Cluster Charm
Fix Released
Critical
David Ames
OpenStack Percona Cluster Charm 18.02

Bug Description

When bootstrapping a 6 node cluster the last 4 nodes all failed a TypeError (see below for full stack trace). This seems to be reliably reproducible with the following bundle:

series: xenial
applications:
  mysql:
    charm: cs:~openstack-charmers-next/xenial/percona-cluster
    num_units: 6
    options:
      source: cloud:xenial-pike
      innodb-buffer-pool-size: 512M
      vip: 10.5.100.1
      wait-timeout: 180
      min-cluster-size: 6
      enable-binlogs: True
      performance-schema: True
      max-connections: 2000
  hacluster-pxc:
    charm: cs:~openstack-charmers-next/xenial/hacluster
    options:
      cluster_count: 6
relations:
  - [ mysql, hacluster-pxc ]

2018-01-23 15:33:28 DEBUG leader-settings-changed Traceback (most recent call last):
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-settings-changed", line 780, in <module>
2018-01-23 15:33:28 DEBUG leader-settings-changed main()
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-settings-changed", line 770, in main
2018-01-23 15:33:28 DEBUG leader-settings-changed hooks.execute(sys.argv)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-01-23 15:33:28 DEBUG leader-settings-changed self._hooks[hook_name]()
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-settings-changed", line 734, in leader_settings_changed
2018-01-23 15:33:28 DEBUG leader-settings-changed config_changed()
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-01-23 15:33:28 DEBUG leader-settings-changed return f(*args, **kwargs)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-settings-changed", line 353, in config_changed
2018-01-23 15:33:28 DEBUG leader-settings-changed update_root_password()
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/percona_utils.py", line 845, in update_root_password
2018-01-23 15:33:28 DEBUG leader-settings-changed m_helper.set_mysql_root_password(new_root_passwd)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/contrib/database/mysql.py", line 344, in set_mysql_root_password
2018-01-23 15:33:28 DEBUG leader-settings-changed self.set_mysql_password('root', password)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/contrib/database/mysql.py", line 325, in set_mysql_password
2018-01-23 15:33:28 DEBUG leader-settings-changed self.connect(user=username, password=new_passwd)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/contrib/database/mysql.py", line 91, in connect
2018-01-23 15:33:28 DEBUG leader-settings-changed passwd=password)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
2018-01-23 15:33:28 DEBUG leader-settings-changed return Connection(*args, **kwargs)
2018-01-23 15:33:28 DEBUG leader-settings-changed File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 204, in __init__
2018-01-23 15:33:28 DEBUG leader-settings-changed super(Connection, self).__init__(*args, **kwargs2)
2018-01-23 15:33:28 DEBUG leader-settings-changed TypeError: connect() argument 3 must be string, not None
2018-01-23 15:33:28 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-settings-changed" failed: exit status 1

Revision history for this message
Liam Young (gnuoy) wrote :

Model Controller Cloud/Region Version SLA
otherbug gnuoy-serverstack serverstack/serverstack 2.2.6 unsupported

App Version Status Scale Charm Store Rev OS Notes
hacluster-pxc maintenance 4 hacluster jujucharms 51 ubuntu
mysql 5.6.37-26.21 error 6 percona-cluster jujucharms 280 ubuntu

Unit Workload Agent Machine Public address Ports Message
mysql/0* blocked idle 0 10.5.0.4 3306/tcp Insufficient peers to bootstrap cluster
  hacluster-pxc/2 maintenance executing 10.5.0.4 (config-changed) Setting up corosync
mysql/1 blocked executing 1 10.5.0.65 3306/tcp Insufficient peers to bootstrap cluster
  hacluster-pxc/0* active executing 10.5.0.65 (start) Unit is ready and clustered
mysql/2 blocked executing 2 10.5.0.38 3306/tcp (leader-settings-changed) Insufficient peers to bootstrap cluster
  hacluster-pxc/1 active executing 10.5.0.38 (start) Unit is ready and clustered
mysql/3 blocked idle 3 10.5.0.41 3306/tcp Insufficient peers to bootstrap cluster
  hacluster-pxc/3 maintenance executing 10.5.0.41 (install) Installing apt packages
mysql/4 error idle 4 10.5.0.71 hook failed: "leader-settings-changed"
mysql/5 error idle 5 10.5.0.66 hook failed: "leader-settings-changed"

Machine State DNS Inst id Series AZ Message
0 started 10.5.0.4 8f0dceed-ecde-46a3-89a8-be5c7e54a3fd xenial nova ACTIVE
1 started 10.5.0.65 bc67b368-88ca-4a16-ac77-b3fca652d4ed xenial nova ACTIVE
2 started 10.5.0.38 3b0d7002-315b-4567-913a-b49e0454de8e xenial nova ACTIVE
3 started 10.5.0.41 8ae23b62-2fd6-45b6-aa18-446f88faa580 xenial nova ACTIVE
4 started 10.5.0.71 d21e0747-76d9-4f75-8098-3cfb2dc5daa4 xenial nova ACTIVE
5 started 10.5.0.66 285f123f-ce7e-4781-a013-f7dea67c5c32 xenial nova ACTIVE

Relation provider Requirer Interface Type
hacluster-pxc:ha mysql:ha hacluster subordinate
hacluster-pxc:hanode hacluster-pxc:hanode hacluster peer
mysql:cluster mysql:cluster percona-cluster peer

David Ames (thedac)
Changed in charm-percona-cluster:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → David Ames (thedac)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-percona-cluster (master)

Fix proposed to branch: master
Review: https://review.openstack.org/537020

Changed in charm-percona-cluster:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-percona-cluster (master)

Reviewed: https://review.openstack.org/537020
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=dc19ecb4a3e36999e0dda98a4ff040c8509152a2
Submitter: Zuul
Branch: master

commit dc19ecb4a3e36999e0dda98a4ff040c8509152a2
Author: David Ames <email address hidden>
Date: Tue Jan 23 18:01:21 2018 +0000

    Guarantee timing of installation and render

    Previous attempts to solve Bug #1738896 missed the root cause. The
    root cause problem is when the configuration file is rendered before
    percona is installed. The rendering includes clustering configuration
    which causes percona-cluster to automatically do a single bootstrap
    when percona-cluster packages are installed leading to the UUID
    mismatch.

    The timing and ordering of installation, rendering of the
    configuration and restart of mysql is critical across all hook
    executions.

    This change is a partial reversion of Change ID
    I95e56bd28152c934f413025a22dd6821b2ad8e94. The change primarily
    guarantees percona-cluster is not installed on non-leader nodes
    before the leader is bootstrapped and makes sure the configuration
    does not get rendered prior to installation.

    is_leader_bootstrapped is introduced and guarantees all data expected
    from the leader is available to guard on various tasks.

    Closes-Bug: #1744961
    Closes-Bug: #1738896
    Change-Id: Ifeb1520dba3b14fc1b51a586141905a385f2b2c1

Changed in charm-percona-cluster:
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-percona-cluster:
milestone: none → 18.02
Ryan Beisner (1chb1n)
Changed in charm-percona-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.