unable to sync database when access-network is configured in percona-cluster charm

Alex, I see that you are using a 3-node Percona cluster. Could you please confirm that the cluster formed properly by doing:

juju ssh percona/0 "mysql -uroot -pagoodpassword -e \"show status like 'wsrep_cluster%';\""

and verifying that cluster_size is 3

Also, did you manage to recover from this issue? e.g. by doing 'juju resolved -r glance/0'

I also see that the database access grants are being created using maas dns names as opposed to ip addresses. Could you please provide the output of:

juju ssh percona/0 pt-show-grants -uroot -p<root-pass> | grep glance

NOTE: you'll need to install percona-toolkit on your db node to do this

Edward, this is the result as you mentioned

mysql> show status like 'wsrep_cluster%';
+--------------------------+--------------------------------------+
| Variable_name | Value |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 57c2a09d-a2ab-11e4-addc-979d5ab3b3ac |
| wsrep_cluster_status | Primary |
+--------------------------+--------------------------------------+

ubuntu@db01:~$ pt-show-grants -uroot -pubuntu | grep glance
-- Grants for 'glance'@'10.100.1.170'
GRANT USAGE ON *.* TO 'glance'@'10.100.1.170' IDENTIFIED BY PASSWORD '*3F5426F710A7D88B2337302C169A453136248B15';
GRANT ALL PRIVILEGES ON `glance`.* TO 'glance'@'10.100.1.170';
-- Grants for 'glance'@'10.100.1.171'
GRANT USAGE ON *.* TO 'glance'@'10.100.1.171' IDENTIFIED BY PASSWORD '*3F5426F710A7D88B2337302C169A453136248B15';
GRANT ALL PRIVILEGES ON `glance`.* TO 'glance'@'10.100.1.171';
-- Grants for 'glance'@'172.100.2.99'
GRANT USAGE ON *.* TO 'glance'@'172.100.2.99' IDENTIFIED BY PASSWORD '*3F5426F710A7D88B2337302C169A453136248B15';
GRANT ALL PRIVILEGES ON `glance`.* TO 'glance'@'172.100.2.99';

mysql> select host,user from user;
+---------------+------------------+
| host | user |
+---------------+------------------+
| 10.100.1.161 | keystone |
| 10.100.1.162 | keystone |
| 10.100.1.164 | neutron |
| 10.100.1.164 | nova |
| 10.100.1.165 | neutron |
| 10.100.1.165 | nova |
| 10.100.1.170 | glance |
| 10.100.1.171 | glance |
| 10.100.1.172 | cinder |
| 10.100.1.173 | cinder |
| 127.0.0.1 | root |
| 172.100.2.103 | neutron |
| 172.100.2.103 | nova |
| 172.100.2.105 | cinder |
| 172.100.2.106 | neutron |
| 172.100.2.106 | nova |
| 172.100.2.95 | cinder |
| 172.100.2.96 | keystone |
| 172.100.2.97 | keystone |
| 172.100.2.99 | glance |
| ::1 | root |
| db01 | root |
| ip6-localhost | sstuser |
| localhost | debian-sys-maint |
| localhost | root |
| localhost | sstuser |
+---------------+------------------+
26 rows in set (0.00 sec)

as you see this result, only one glance node didn't get grant thorugh public network in db.
ohter services are all fine to sync databases. (even another glance node got the grant )

juju resolve still failed to resolve because of grant

2015-01-23 04:59:36 INFO shared-db-relation-changed 2015-01-23 04:59:36.703 18407 CRITICAL glance [-] OperationalError: (OperationalError) (1130, "Host '172-100-2-98.maas' is not allowed to connect to this MySQL server") None None

If you look at grant tables, you'll notice that .98 is not in it. My questions would be:

1) if .98 is VIP, why is glance accessing DB with VIP
2) if .98 is not VIP, why isn't charm creating proper grant rights in DB

In both cases, bug would be in the charm, cause glance binds to *any* address on the host due to bind_host = 0.0.0.0.

here is mysql query result. one of glance node has a grant but anothere doesn't have it.
.98 is nova glance ha vip.

mysql> select host,user from user;
+---------------+------------------+
| host | user |
+---------------+------------------+
[...]

| 10.100.1.170 | glance |
| 10.100.1.171 | glance |

| 172.100.2.99 | glance |
| ::1 | root |
| db01 | root |

+---------------+------------------+

Additional Information of my env is that
Machines have 2 network interfaces which one is 10.100.1.x, another is 172.100.2.x
As above comment 1 glance node has successful grant right, and another glance has only grant right with 10.100.1.x network

here is my openstack bundle file
https://pastebin.canonical.com/124513/

I think I see the issue here; the SharedDB helper context deals with switching from private-address to an address on the 'access-network' if the percona-cluster charm presents that data over the relation.

The code that does this simply returns {} during this process:

  http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/contrib/openstack/context.py#L177

however 'context_complete' thinks this is still a complete context, as it does not contain any keys which are either None or '':

  http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/contrib/openstack/context.py#L91

which results in:

  http://bazaar.launchpad.net/~openstack-charmers/charms/precise/glance/trunk/view/head:/hooks/glance_relations.py#L145

return true, even though the context is not actually complete yet.

Returning None instead of {} for the SharedDB context in this case is the right thing todo; I'm reticent to change context_complete to deal with empty dicts....

Charmhelper resync pushed to all openstack charms - Marking Fix Committed for Glance.

Now doing the stable branches.