Juju Charms Collection
rabbitmq-server package

regression: rabbimq cluster is broken with Juju 1.22

Bug #1483949 reported by Nobuto Murata on 2015-08-12

This bug report is a duplicate of: Bug #1486177: 3-node native rabbitmq cluster race. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	rabbitmq-server (Juju Charms Collection)	New	Undecided	Unassigned

Bug Description

[the latest revision charm-store.33]
$ juju deploy -n2 cs:trusty/rabbitmq-server
Added charm "cs:trusty/rabbitmq-server-33" to the environment.

$ juju run --service rabbitmq-server 'sudo rabbitmqctl cluster_status'
- MachineId: "1"
  Stdout: |
    Cluster status of node 'rabbit@juju-nobuto-machine-1' ...
    [{nodes,[{disc,['rabbit@juju-nobuto-machine-1']}]},
     {running_nodes,['rabbit@juju-nobuto-machine-1']},
     {partitions,[]}]
    ...done.
  UnitId: rabbitmq-server/0
- MachineId: "2"
  Stdout: |
    Cluster status of node 'rabbit@juju-nobuto-machine-2' ...
    [{nodes,[{disc,['rabbit@juju-nobuto-machine-2']}]},
     {running_nodes,['rabbit@juju-nobuto-machine-2']},
     {partitions,[]}]
    ...done.
  UnitId: rabbitmq-server/1

^^^ not clustered properly. they don't know each other.

[the previous revision charm-store.32]
$ juju deploy -n2 cs:trusty/rabbitmq-server-32 rabbitmq-server-previous
Added charm "cs:trusty/rabbitmq-server-32" to the environment.

$ juju run --service rabbitmq-server-previous 'sudo rabbitmqctl cluster_status'
- MachineId: "3"
  Stdout: |
    Cluster status of node 'rabbit@juju-nobuto-machine-3' ...
    [{nodes,[{disc,['rabbit@juju-nobuto-machine-3',
                    'rabbit@juju-nobuto-machine-4']}]},
     {running_nodes,['rabbit@juju-nobuto-machine-4',
                     'rabbit@juju-nobuto-machine-3']},
     {partitions,[]}]
    ...done.
  UnitId: rabbitmq-server-previous/0
- MachineId: "4"
  Stdout: |
    Cluster status of node 'rabbit@juju-nobuto-machine-4' ...
    [{nodes,[{disc,['rabbit@juju-nobuto-machine-3',
                    'rabbit@juju-nobuto-machine-4']}]},
     {running_nodes,['rabbit@juju-nobuto-machine-3',
                     'rabbit@juju-nobuto-machine-4']},
     {partitions,[]}]
    ...done.
  UnitId: rabbitmq-server-previous/1

^^^ properly clustered.

Tags:

Revision history for this message

Nobuto Murata (nobuto) wrote on 2015-08-12:

all-machines.log.gz Edit (180.3 KiB, application/octet-stream)

attached all-machines.log.gz.

I can reproduce this issue both in openstack provider and maas provider with Juju 1.22.6-0ubuntu1~14.04.1.

Nobuto Murata (nobuto) on 2015-08-17

tags:

added: cpec

Revision history for this message

Ante Karamatić (ivoks) wrote on 2015-08-17:

Have you configured min-cluster-size?

Revision history for this message

Nobuto Murata (nobuto) wrote on 2015-08-17:

> Have you configured min-cluster-size?

nope in the example above. but yes for my previous deployment affected.

My gut feeling is that some logics are wrong without leader election. 1.24 has leader election, 1.22 not. From all-machines.log with charm store 33, cookie never be synced between nodes.

[hooks/rabbitmq_server_relations.py]
    306 if is_elected_leader('res_rabbitmq_vip'):
    307 cookie = open(rabbit.COOKIE_PATH, 'r').read().strip()
    308 peer_store('cookie', cookie)

I did not test it intensively, but the logic above might be related.

Revision history for this message

Nobuto Murata (nobuto) wrote on 2015-08-17: