Fuel for OpenStack

rabbit-fence.py process doesn't start after controller reboot.

Bug #1456791 reported by Alexander Nevenchannyy on 2015-05-19

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	High	Bogdan Dobrelya	Fuel for OpenStack 6.1
	6.0.x	Invalid	Undecided	Unassigned

Bug Description

Folks, i'm found that rabbit-fence.py process doesn't start after controller reboot.
node-1 was rebooted after deployment, node-2 was not rebooted.

root@node-1:/usr/bin# ps -ef | grep fence
root 17121 13920 0 19:38 pts/4 00:00:00 grep --color=auto fence
root@node-1:/usr/bin# dmesg | grep fence
[ 10.259292] init: fuel-rabbit-fence main process (2354) terminated with status 1
root@node-1:~# uptime
20:02:15 up 1:08, 1 user, load average: 0.18, 0.39, 0.47

root@node-2:~# ps -ef | grep fence
rabbitmq 19896 1 0 19:49 ? 00:00:00 /usr/bin/python /usr/bin/rabbit-fence.py
root 25404 25241 0 20:04 pts/4 00:00:00 grep --color=auto fence
root@node-2:~# dmesg | grep fence
[ 10.333070] init: fuel-rabbit-fence main process (2353) terminated with status 1
root@node-2:~# uptime
20:05:03 up 1 day, 6:10, 2 users, load average: 0.32, 0.45, 0.57

Environment available at 172.16.59.114
Environment was deployed with 432 build, Ubuntu underlying OS and HA configuration.

Tags:

Vladimir Kuklin (vkuklin) on 2015-05-19

Changed in fuel:
milestone:	none → 6.1
importance:	Undecided → High
status:	New → Confirmed

Bogdan Dobrelya (bogdando) on 2015-05-20

Changed in fuel:
status:	Confirmed → New
status:	New → Confirmed
tags:	added: rabbitmq

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-20: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/184465

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-05-20:

How to test example for the patch https://review.openstack.org/184465:

root@node-8:~# initctl status fuel-rabbit-fence
fuel-rabbit-fence start/running, process 12903
root@node-8:~# kill 12903
root@node-8:~# <30>May 20 11:45:49 node-8 rabbit-fence 2015-05-20 11:45:49,980 INFO Caught SIGTERM, terminating...
<30>May 20 11:45:50 node-8 rabbit-fence 2015-05-20 11:45:50,085 INFO Starting rabbit fence script main loop

OpenStack Infra (hudson-openstack) on 2015-05-21

Changed in fuel:
assignee:	Bogdan Dobrelya (bogdando) → Vladimir Kuklin (vkuklin)

OpenStack Infra (hudson-openstack) on 2015-05-22

Changed in fuel:
assignee:	Vladimir Kuklin (vkuklin) → Bartlomiej Piotrowski (bpiotrowski)

Bartłomiej Piotrowski (bpiotrowski) on 2015-05-22

Changed in fuel:
assignee:	Bartlomiej Piotrowski (bpiotrowski) → Bogdan Dobrelya (bogdando)

OpenStack Infra (hudson-openstack) on 2015-05-23

Changed in fuel:
assignee:	Bogdan Dobrelya (bogdando) → Vladimir Kuklin (vkuklin)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-25: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/184465
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6e6d1cb23c89f5159a0112b4c496adeb30462111
Submitter: Jenkins
Branch: master

commit 6e6d1cb23c89f5159a0112b4c496adeb30462111
Author: Vladimir Kuklin <email address hidden>
Date: Thu May 21 20:21:42 2015 +0300

Ensure the rabbit-fence daemon started and respawned

    W/o this fix, if the daemon crashed, it
    would remain stopped. Also, the rabbit fence
    daemon will crash on start, when there is no
    /var/run/rabbitmq piddir exist.

    The solution is
    * add the respawn option to
    the daemon's upstart (TODO for Centos inittab)
    * and ensure the piddir
    created by upstart/init.d before starting the daemon.

This also requires to add the rabbitmq and
fuel-rabbit-fence packages as a dependency.

Closes-bug: #1456791

Change-Id: I4e71eb9e4aa4ff3b877aa89a37d82215740aaeab
Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Bogdan Dobrelya (bogdando) on 2015-05-25

Changed in fuel:
assignee:	Vladimir Kuklin (vkuklin) → Bogdan Dobrelya (bogdando)

Veronica Krayneva (vkrayneva) on 2015-05-26

tags:

added: on-verivication

Revision history for this message

Veronica Krayneva (vkrayneva) wrote on 2015-05-28:

Checked in fuel-6.1-471-2015-05-26_20-59-56.iso.
Bug was fixed

tags:	removed: on-verivication
Changed in fuel:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.