Ubuntu
eucalyptus package

stopping eucalyptus may lose the database

Bug #411692 reported by Chris Jones
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
Fix Released
High
Daniel Nurmi
Jaunty
Won't Fix
High
Unassigned
Karmic
Fix Released
High
Daniel Nurmi

Bug Description

I created a new user and in the course of changing other configuration items restarted both the -cc and -cloud init scripts a number of times. At one point the cloud came back with my non-admin user gone.
nekro on #eucalyptus suggested this may be down to the init script having a 5 second timeout after which it kill -9's the process, leading to potential data loss.

Tags: eucalyptus
Revision history for this message
Chris Jones (cmsj) wrote :

Perhaps the kill -9 isn't directly responsible...

[00:45] <@nekro_> Ng: eucalyptus.script is read in initially. eucalyptus.log is the current,
                  rolling state. It will be dumped to eucalyptus.script on shutdown.

cf:

root@curium:/var/lib/eucalyptus/db# lsof -n | grep eucalyptus.log
java 6298 eucalyptus 147w REG 105,1 861398 114894 /var/lib/eucalyptus/db/eucalyptus.log (deleted)
java 30086 eucalyptus 147w REG 105,1 647 114889 /var/lib/eucalyptus/db/eucalyptus.log (deleted)

Revision history for this message
Chris Jones (cmsj) wrote :

FWIW, I restarted our cloud after it had been running for a week and it didn't dump the (by that stage really very large) .log file to the .script, but that did happen when I started it all back up again.

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Marking 'critical' since there's a potential for data loss here.

Chris, I don't understand your last comment. It looks like you tried to reproduce the problem sometime later and were not able to? Do you believe this is still a problem?

:-Dustin

Changed in eucalyptus (Ubuntu):
status: New → Incomplete
Changed in eucalyptus (Ubuntu Jaunty):
status: New → Incomplete
importance: Undecided → Critical
Changed in eucalyptus (Ubuntu):
importance: Undecided → Critical
Revision history for this message
Chris Jones (cmsj) wrote :

Dustin: Yeah it's not totally reproducible. I do have a slight concern that it seems like the database isn't stored on disk until the daemon shuts down gracefully, but I could be wrong about that.

Revision history for this message
Matt Zimmerman (mdz) wrote :

I don't know if this is related at all, but Thierry mentioned this in a test report:

Then the power outtage left some state that apparently makes it
impossible to recover from, unless you run "/etc/init.d/eucalyptus-cc
cleanstop".

Revision history for this message
Matt Zimmerman (mdz) wrote :

Dan indicates that Thierry's problem was unrelated

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Lowering to High from Critical, since this doesn't always happen.

Adding a task for Karmic, leaving incomplete since we haven't yet seen this here.

Assigning the Karmic task to Dan, as he's going to try and kill -9 the database and see if he sees data loss.

Dan, please update this bug status accordingly.

Milestoning the bug against 9.10 GA.

:-Dustin

Changed in eucalyptus (Ubuntu):
status: Incomplete → Confirmed
Changed in eucalyptus (Ubuntu Jaunty):
status: Incomplete → Confirmed
importance: Critical → High
Changed in eucalyptus (Ubuntu Karmic):
importance: Critical → High
status: Confirmed → Incomplete
assignee: nobody → Daniel Nurmi (nurmi)
John Pugh (jpugh)
tags: added: eucalyptus
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Closing this bug for now, per phone conversation with Chris.

He found the bug against 1.5 in Jaunty. We've never seen this in 1.6 in Karmic. So we're marking fix-released in Karmic, and won't-fix in Jaunty.

Please reopen if you see this problem again in Karmic.

Thanks,
:-Dustin

Changed in eucalyptus (Ubuntu Karmic):
status: Incomplete → Fix Released
Changed in eucalyptus (Ubuntu Jaunty):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.