[SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space

Bug #565101 reported by C de-Avillez
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
Fix Released
High
Unassigned
eucalyptus (Ubuntu)
Fix Released
High
Dustin Kirkland 
Lucid
Fix Released
High
Dustin Kirkland 

Bug Description

stress-testing, 400 instances to be started in total. After a while the walrus starts reporting out-of-memory, and all new instances fail.

Ubuntu-server 2010-04-16, UEC 2010-04-16.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: eucalyptus-walrus 1.6.2-0ubuntu30
ProcVersionSignature: Ubuntu 2.6.32-21.31-server 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-server x86_64
Architecture: amd64
Date: Fri Apr 16 19:18:49 2010
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: eucalyptus

========
IMPACT:
 * This bug affects UEC installs, with the CLC and Walrus on separate machines. After running a few hundred (or thousands) of instances, the Walrus will eventually run out of memory.

ADDRESSED:
 * The bug is addressed in two ways. First, a configuration parameter has been added to /etc/eucalyptus/eucalyptus.conf, JVM_MEM, defaulting to 512M. Admins can tailor this value to the needs of their cloud (more memory mean more instances in a shorter period of time, before the garbage collector has to run). Secondly, we have cherry picked several commits from the upstream stable source branch that address various memory leaks in the Walrus Java code. Both of these are necessary to fully address the problem.

PATCH:
 * The patch isn't very minimal. Sorry. But it's what's required to solve the problem. And it's already in the upstream stable branch.

REPRODUCE:
 * Reproducing this bug is not easy. We most frequently see this happen when the CLC and Walrus are not co-located on the same machine. And then we run several thousand instances. Eventually, Walrus falls over with OOM.

REGRESSION POTENTIAL:
 * There's certainly regression potential in the Walrus itself, as that's the code that's changed. However, we have put this code through a very heavy barrage of scalability testing (which also exercises many of the base features). We believe that we have proactively tested this code thoroughly.
========

Revision history for this message
C de-Avillez (hggdh2) wrote :
Changed in eucalyptus (Ubuntu):
status: New → In Progress
importance: Undecided → High
milestone: none → ubuntu-10.04
assignee: nobody → Dustin Kirkland (kirkland)
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

The suggested fix is to up the jvm's memory to 384M in /etc/init/eucalyptus.conf:

=== modified file 'debian/eucalyptus-common.eucalyptus.upstart'
--- debian/eucalyptus-common.eucalyptus.upstart 2010-02-04 01:00:15 +0000
+++ debian/eucalyptus-common.eucalyptus.upstart 2010-04-20 16:59:34 +0000
@@ -36,7 +36,7 @@
 script
        . /etc/eucalyptus/eucalyptus.conf

- opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/eucalyptus/eucalyptus.pid -l $LOGLEVEL -L console-log"
+ opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/eucalyptus/eucalyptus.pid -l $LOGLEVEL -L console-log -Xmx384m"
        services=""

        # If the -cloud package is not installed, disable the cloud service

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Awaiting testing feedback from Carlos, before committing.

Revision history for this message
C de-Avillez (hggdh2) wrote :

Got some more out-of-memory errors.

ubuntu@mabolo:/var/log/eucalyptus$ ps aux | grep euca
root 1023 0.0 0.0 16828 1052 ? Ss Apr19 0:00 avahi-publish -s Walrus _eucalyptus._tcp 8773 txtvers=1 protovers=1.5.0 type=walrus ipaddr=10.55.55.5
root 14733 0.0 0.0 16688 928 ? Ss 13:21 0:00 eucalyptus-cloud -h / -u eucalyptus --pidfile /var/run/eucalyptus/eucalyptus.pid -l DEBUG -L console-log -Xmx384m --disable-cloud --disable-storage --remote-dns
105 14857 29.4 5.9 966552 605300 ? Sl 13:21 28:11 eucalyptus-cloud -h / -u eucalyptus --pidfile /var/run/eucalyptus/eucalyptus.pid -l DEBUG -L console-log -Xmx384m --disable-cloud --disable-storage --remote-dns
ubuntu 20444 0.0 0.0 7620 1016 pts/1 S+ 14:57 0:00 grep --color=auto euca
ubuntu@mabolo:/var/log/eucalyptus$

14:48:07 DEBUG [NioMessageReceiver:New I/O server worker #1-16] Message Received from: http://127.0.0.1:8773/internal/BukkitInternal
14:48:38 DEBUG [NioServerHandler:New I/O server worker #1-20] Internal Error.
java.lang.OutOfMemoryError: Java heap space
14:49:04 DEBUG [NioServerHandler:New I/O server worker #1-11] Internal Error.
java.lang.OutOfMemoryError: Java heap space
14:49:32 DEBUG [NioServerHandler:New I/O server worker #1-18] Internal Error.
java.lang.OutOfMemoryError: Java heap space
14:49:57 DEBUG [NioServerHandler:New I/O server worker #1-25] Internal Error.
java.lang.OutOfMemoryError: Java heap space
14:50:20 DEBUG [NioServerHandler:New I/O server worker #1-16] Internal Error.
14:50:47 DEBUG [NioServerHandler:New I/O server worker #1-4] Internal Error.
java.lang.OutOfMemoryError: Java heap space
14:51:43 DEBUG [NioServerHandler:New I/O server worker #1-5] Internal Error.
14:52:33 DEBUG [NioServerHandler:New I/O server worker #1-1] Internal Error.
java.nio.channels.ClosedChannelException

Different from before, though: there was the above cluster of errors, tbut mostly they are spread across.

Revision history for this message
C de-Avillez (hggdh2) wrote :
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Copying this bug upstream, as the simple configuration change didn't solve the problem for us.

Revision history for this message
C de-Avillez (hggdh2) wrote :

running with -Xmx1024m seems to have reached a steady-state at about 930M of RES. At least with our rig I cannot open sessions faster enough to cause more increases.

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 565101] Re: walrus reports java.lang.OutOfMemoryError: Java heap space

Good information, Carlos.

So perhaps upping our default JVM heap to 1024M would alleviate some
of the pain, but perhaps not heal the wound.

Let's see what upstream Eucalyptus has to say...

Changed in eucalyptus (Ubuntu Lucid):
milestone: ubuntu-10.04 → lucid-updates
Revision history for this message
Thierry Carrez (ttx) wrote : Re: walrus reports java.lang.OutOfMemoryError: Java heap space

Java programs have a bad habit of not being able to use "the memory" but rather preselect an amount of memory usage at JVM startup, which makes it difficult to have a sane default in packaging. It's the same for Tomcat, where we explicitly give people the knobs to tune it after the fact and recommend that they do so.

I'm not sure there is a sane default here... I suspect raising the load would require you to bump that value again. It's also about minimal requirements for the Walrus: if we say it can be run on a machine with 1Gb of RAM, then obviously we can't really set the default to more than -Xmx512m.

So it lokks like a documentation issue ("tune the Walrus memory usage !"), adding the knob ("setting this value in that file is the way to do it") and maybe slightly raising the default value so that, by default, it can handle slightly more load ?

Revision history for this message
C de-Avillez (hggdh2) wrote :

Yesterday night I bounced the whole rig (to start clean), and submitted a 2,000 instances run. Run ended at 03:30 UTC. Right now it is about 09:10 UTC, and the Walrus shows 1,166M of resident memory.

I am not sure how the garbage collection works, but it really does not look like it *is* working, or there is indeed a memory leak somewhere.

Revision history for this message
C de-Avillez (hggdh2) wrote :

Sorry, times above are EDT, not UTC.

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 565101] Re: walrus reports java.lang.OutOfMemoryError: Java heap space

Okay, so Thierry, the best way to solve this then is to set this value
to a sane default in a variable in the Upstart script, document it,
and point users to adjust that variable if they hit this barrier?

Revision history for this message
C de-Avillez (hggdh2) wrote : Re: walrus reports java.lang.OutOfMemoryError: Java heap space

It sounds like a sane approach, yes. I have a feeling that this memory creep may be related to errors starting instances, causing the related Java-instantiated classes to be kept. I also do not remember seeing a sequence of failures so big as what I am seeing now at or before Beta1.

Revision history for this message
Thierry Carrez (ttx) wrote :

Yes, my opinion is it *will* use more heap memory when used under load and that needs to be configured, no matter what.

Maybe there is a bug in eucalyptus code that makes it consume much more memory than it should, and it should never be necessary to raise it over the default heap size. Maybe there is a bug in eucalyptus that makes it leak references to objects over time, and any value we put in will always not be enough after some time. But having that knob and documenting its use cannot hurt. Especially if upstream considers the memory usage profile under load to be "working as designed".

Revision history for this message
Thierry Carrez (ttx) wrote :

Important note: when not specified, max heap size is set to:
***
Smaller of 1/4th of the physical memory or 1GB. Before J2SE 5.0, the default maximum heap size was 64MB. You can override this default using the -Xmx command-line option.
***

So it sounds like moving from no value to to -Xmx384m would actually reduce the maximum heap size on most server systems, and that the default value is sanely calculated from available memory. It also makes a case for *not* specifying the value by default, something like:

XMX=""
[ -n ${MAXHEAPSIZE} ] && XMX="-Xmx ${MAXHEAPSIZE}m"
opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/eucalyptus/eucalyptus.pid -l $LOGLEVEL -L console-log $XMX"

Or even

# Extra options passed to JVM call. For exemple you can set a 1Gb maximum heap size by setting
# EXTRAOPTS="-Xmx1024m"
EXTRAOPTS=""
opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/eucalyptus/eucalyptus.pid -l $LOGLEVEL -L console-log $EXTRAOPTS"

In all cases, confirming that the observed memory profile usage is judged sane by upstream (and setting max heap size is their recommended way of handling load) needs to happen before the fix.

Revision history for this message
C de-Avillez (hggdh2) wrote :

Thierry, some comments:

*Heap size*

0. The machines on the test rig all have 10G of main memory;
1. when we started (no -Xmx) we would get OutOfMemory when the eucalyptus (CLC, Walrus, CC) resident memory -- as reported by 'top' was at about 800M;
2. When I tried -Xmx384m I observed OutOfMemory eariler, when the eucalyptus resident memory was at about 650M;
3. When I tried -Xmx1024m I observed OutOfMemory later, when the eucalyptus resident memory was at about 1300M.

So, indeed, setting -Xmx has to be done carefully. But (3) above suggests the default heap allocation is *not* 1G (less of 1/4 of 10G or 1G).

*Heap usage*

I have been unable to observe reduction on memory usage (again, as reported by 'top', even on an idle cloud (i.e., a cloud that has no instances running). I would expect memory increase on usage -- class instantiations, etc --, followed by memory being released on system idling. All I have observed, so far, if that memory usage *only* increases. So either there is a code issue, or a Java issue.

Day before yesterday I bounced *all* systems (so that I could guarantee we were starting from a clean slate), and submitted a 2,000 instance run. This run went from about 22:00 to 03:00 EDT; from about 01:30 onwards, no instance start succeeded, and OutOfMemory errors were being reported on the CLC, Walrus, and CC. The only conclusion I can make, right now, is that failed instance startup does not release memory. Remember we are observing 40~50% failure rates right now.

Additionally, it seems that after an OutOfMemory no new instances can be run. In other words, an OutOfMemory *should* be considered a fatal error, requiring restart on the affected components. At least right now.

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 565101] Re: walrus reports java.lang.OutOfMemoryError: Java heap space

Note from Dan on the weekly call...

He notes that Thierry's analysis of how heap size is determined by the
JVM doesn't apply to Eucalyptus, as they actually set it themselves,
rather than relying on the JVM to decide.

Revision history for this message
Neil Soman (neilsoman) wrote : Re: walrus reports java.lang.OutOfMemoryError: Java heap space

------------------------------------------------------------
revno: 1225
committer: Neil Soman
branch nick: 1.6.2
timestamp: Sat 2010-04-24 12:09:26 -0700
message:
  fixes LP: #565101
------------------------------------------------------------

Changed in eucalyptus:
importance: Undecided → High
status: New → Fix Committed
Revision history for this message
C de-Avillez (hggdh2) wrote :

I just rab a sequence of 400 instances on topo2 (CLC;Walrus;CC;SC;NC-1;NC-2). Topo2 was has been, on my experience, the easiest one to get the OOMs; usually I would see both CLC and Walrus increasing memory in a similar way, and starting to spit out OOMs at about the 750M of resident memory (without using '-Xmx nnnn' to force a specific heap size).

I ran it using today's daily -- 20100426.1 for the base install, 20100426 for the UEC image, plus eucalyptus 1.6.2-0ubuntu31~ppa2 .

This time I see a slower increase in memory on the CLC, and an even slower on the Walrus. At the end of this run the CLC reported (via 'top') ~800M of resident memory, and the Walrus ~700M. The run has been saved on lp:~hggdh2/+junk/uec-qa, under lucid/20100426.1/topo2-0.tar (the output of the test, plus the contents of /var/log/* for each machine).

Revision history for this message
C de-Avillez (hggdh2) wrote :

Ran another 400-instances, without restarting Eucalyptus. Got OOMs on the Walrus, but not on the CLC. For the record, here's the CLC eucalyptus package details:

ubuntu@cempedak:~/uec-testing-scripts$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================-=============================-==========================================================================
ii euca2ools 1.2-0ubuntu10 managing cloud instances for Eucalyptus
un eucalyptus-cc <none> (no description available)
ii eucalyptus-cloud 1.6.2-0ubuntu31~ppa2 Elastic Utility Computing Architecture - Cloud controller
ii eucalyptus-common 1.6.2-0ubuntu31~ppa2 Elastic Utility Computing Architecture - Common files
ii eucalyptus-java-common 1.6.2-0ubuntu31~ppa2 Elastic Utility Computing Architecture - Common Java package
un eucalyptus-javadeps <none> (no description available)
un eucalyptus-sc <none> (no description available)
un eucalyptus-walrus <none> (no description available)
ii uec-component-listener 1.6.2-0ubuntu31~ppa2 Ubuntu Enterprise Cloud - Component listener
ubuntu@cempedak:~/uec-testing-scripts$

And here is the Walrus:

ubuntu@mabolo:/var/log/eucalyptus$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================-=============================-==========================================================================
ii euca2ools 1.2-0ubuntu10 managing cloud instances for Eucalyptus
un eucalyptus-cloud <none> (no description available)
ii eucalyptus-common 1.6.2-0ubuntu31~ppa2 Elastic Utility Computing Architecture - Common files
ii eucalyptus-java-common 1.6.2-0ubuntu31~ppa2 Elastic Utility Computing Architecture - Common Java package
un eucalyptus-javadeps <none> (no description available)
ii eucalyptus-walrus 1.6.2-0ubuntu31~ppa2 Elastic Utility Computing Architecture - Walrus (S3)
No packages found matching uec*.
ubuntu@mabolo:/var/log/eucalyptus$

Revision history for this message
C de-Avillez (hggdh2) wrote :

Logs were pushed to lp:~hggdh2/+junk/uec-qa, under lucid/20100426.1/topo2-1.tar (the output of the test, plus the contents of /var/log/* for each machine).

Revision history for this message
Neil Soman (neilsoman) wrote :

There was another place where memory was being leaked. This was happening after a register requests, so that over time, after a number of register requests, Walrus would OOM. The issue was with the interaction between the CLC and Walrus.

This is fixed in revno 1227.

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 565101] Re: walrus reports java.lang.OutOfMemoryError: Java heap space

Thanks, Neil.

Carlos,

I have uploaded eucalyptus_1.6.2-0ubuntu31~ppa3_source.changes to the
~ubuntu-virt-ppa. Please give this another round of testing when your
GA-candidate tests are done. Once we can conclusively show that this
solves the OOM, we can apply for SRU approval.

Revision history for this message
C de-Avillez (hggdh2) wrote : Re: walrus reports java.lang.OutOfMemoryError: Java heap space
Download full text (5.7 KiB)

starting the setup for topo2 (the most sensitive to the issue, all components in different machines).

for the CLC:
ubuntu@cempedak:~$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================-=============================-==========================================================================
ii euca2ools 1.2-0ubuntu10 managing cloud instances for Eucalyptus
un eucalyptus-cc <none> (no description available)
ii eucalyptus-cloud 1.6.2-0ubuntu31~ppa3 Elastic Utility Computing Architecture - Cloud controller
ii eucalyptus-common 1.6.2-0ubuntu31~ppa3 Elastic Utility Computing Architecture - Common files
ii eucalyptus-java-common 1.6.2-0ubuntu31~ppa3 Elastic Utility Computing Architecture - Common Java package
un eucalyptus-javadeps <none> (no description available)
un eucalyptus-sc <none> (no description available)
un eucalyptus-walrus <none> (no description available)
ii uec-component-listener 1.6.2-0ubuntu31~ppa3 Ubuntu Enterprise Cloud - Component listener
ubuntu@cempedak:~$

for the Walrus:

ubuntu@mabolo:~$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================-=============================-==========================================================================
ii euca2ools 1.2-0ubuntu10 managing cloud instances for Eucalyptus
un eucalyptus-cloud <none> (no description available)
ii eucalyptus-common 1.6.2-0ubuntu31~ppa3 Elastic Utility Computing Architecture - Common files
ii eucalyptus-java-common 1.6.2-0ubuntu31~ppa3 Elastic Utility Computing Architecture - Common Java package
un eucalyptus-javadeps <none> (no description available)
ii eucalyptus-walrus 1.6.2-0ubuntu31~ppa3 Elastic Utility Computing Architecture - Walrus (S3)
No packages found matching uec*.
ubuntu@mabolo:~$

for the CC:

ubuntu@marula:~$ dpkg -l euca\* uec\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================-=============================-==========================================================================
ii euca2ools 1.2-0ubuntu10 managing cloud instances for...

Read more...

Revision history for this message
C de-Avillez (hggdh2) wrote :

First 1,000-instances run just finished. No OOMs, but:

- memory usage on the Walrus seems pretty constant around ~420M
- memory usage on the CLC is still increasing; at the end of the 1,000 runs it was at 870M, suggesting the end is near.

I have saved the logs at lp:~hggdh2/+junk/uec-qa, under the lucid/final/bug565101-20100427 directory. I have also started another 400-instances run.

Revision history for this message
C de-Avillez (hggdh2) wrote :

Hum. Still up. After another 800 runs, CLC is using 438M, Walrus is using 891M. Still trying to bring it down.

Revision history for this message
C de-Avillez (hggdh2) wrote :

Still surviving. Memory usage did not seem to increase above 891M on the Walrus, even after I threw more instances on it.

I still do not really like the memory increase, but it seems we are good enough to go. Great work.

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

<hggdh> kirkland: No OOMs. I ran about 4,000 instances against it, and no OOM. mem usage on the Walrus still sounds high, but it survived

description: updated
summary: - walrus reports java.lang.OutOfMemoryError: Java heap space
+ [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space
summary: - [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space
+ [SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap
+ space
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted eucalyptus into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in eucalyptus (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
C de-Avillez (hggdh2) wrote :

Ran two sequences, one before and one after aplying the packages in -proposed. Both of them are logged in lp:~hggdh2/+junk/uec-qa/lucid/proposed/euca-1.6.2-0ubuntu30.1.

No OOM on the after-maintenance run, on a 2,000 instances run. No new errors. Fix verified.

Revision history for this message
Martin Pitt (pitti) wrote : Re: [Bug 565101] Re: [SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space

C de-Avillez [2010-04-30 1:55 -0000]:
> Ran two sequences, one before and one after aplying the packages in
> -proposed. Both of them are logged in lp:~hggdh2/+junk/uec-
> qa/lucid/proposed/euca-1.6.2-0ubuntu30.1.
>
> No OOM on the after-maintenance run, on a 2,000 instances run. No new
> errors. Fix verified.

Did you use the actual packages from -proposed, or a local build from
your branch? (We need to test the actual .debs in the archive)

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 565101] Re: [SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space

Martin-

His branch is test results.

Carlos used the lucid-proposed repository.

Martin Pitt (pitti)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

Copied lucid-proposed to maverick.

Changed in eucalyptus (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6.2-0ubuntu30.1

---------------
eucalyptus (1.6.2-0ubuntu30.1) lucid-proposed; urgency=low

  Address LP: #565101
  * debian/eucalyptus.conf: set default JVM_MEM option
  * debian/eucalyptus-common.eucalyptus.upstart: use $JVM_MEM
    from eucalyptus.conf, or default to 512m
  * tools/eucalyptus.conf.5: document the JVM_MEM option

  Cherry-pick upstream commit r1223..1227:
  * node/handlers.c, node/handlers_kvm.c: handle situation where NC's
    do not detach pthreads, LP: #567371
  * node/handlers_kvm.c: fix console bug (was only showing first 64K),
    LP: #566793
  * clc/modules/storage-common/src/main/java/edu/ucsb/eucalyptus/storage/StorageManager.java,
    clc/modules/storage-common/src/main/java/edu/ucsb/eucalyptus/storage/fs/FileSystemStorageManager.java,
    clc/modules/walrus/src/main/java/edu/ucsb/eucalyptus/cloud/ws/WalrusImageManager.java,
    clc/modules/walrus/src/main/java/edu/ucsb/eucalyptus/cloud/ws/WalrusManager.java,
    clc/modules/wsstack/src/main/java/com/eucalyptus/ws/handlers/ServiceSinkHandler.java:
    - fix Walrus OOM errors (java heap), LP: #565101
 -- Dustin Kirkland <email address hidden> Wed, 28 Apr 2010 08:43:38 -0500

Changed in eucalyptus (Ubuntu Lucid):
status: Fix Committed → Fix Released
Martin Pitt (pitti)
Changed in eucalyptus (Ubuntu Lucid):
status: Fix Released → Fix Committed
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Carlos-

Could you please test the package in lucid-proposed, note that it fixes the issue here in this bug, such that Martin can move this package from lucid-proposed to lucid-updates? Thanks!

Revision history for this message
C de-Avillez (hggdh2) wrote :

The test rig already had the updated packages from -proposed, so I am now running a test sequence.

Revision history for this message
C de-Avillez (hggdh2) wrote :

Done. Ran 1,400 instances, no OOMs, no unexpected errors.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6.2-0ubuntu30.2

---------------
eucalyptus (1.6.2-0ubuntu30.2) lucid-proposed; urgency=low

  * Revert: node/handlers_kvm.c: fix console bug (was only showing first 64K),
    LP: #566793
  * clc/modules/www/src/main/java/edu/ucsb/eucalyptus/admin/server/EucalyptusWebBackendImpl.java:
    - fix user enumeration and account brute force, LP: #579942
  * debian/eucalyptus-sc.upstart: Bump maximum number of loop devices for
    SC to 512, LP: #586134

eucalyptus (1.6.2-0ubuntu30.1) lucid-proposed; urgency=low

  Address LP: #565101
  * debian/eucalyptus.conf: set default JVM_MEM option
  * debian/eucalyptus-common.eucalyptus.upstart: use $JVM_MEM
    from eucalyptus.conf, or default to 512m
  * tools/eucalyptus.conf.5: document the JVM_MEM option

  Cherry-pick upstream commit r1223..1227:
  * node/handlers.c, node/handlers_kvm.c: handle situation where NC's
    do not detach pthreads, LP: #567371
  * node/handlers_kvm.c: fix console bug (was only showing first 64K),
    LP: #566793
  * clc/modules/storage-common/src/main/java/edu/ucsb/eucalyptus/storage/StorageManager.java,
    clc/modules/storage-common/src/main/java/edu/ucsb/eucalyptus/storage/fs/FileSystemStorageManager.java,
    clc/modules/walrus/src/main/java/edu/ucsb/eucalyptus/cloud/ws/WalrusImageManager.java,
    clc/modules/walrus/src/main/java/edu/ucsb/eucalyptus/cloud/ws/WalrusManager.java,
    clc/modules/wsstack/src/main/java/com/eucalyptus/ws/handlers/ServiceSinkHandler.java:
    - fix Walrus OOM errors (java heap), LP: #565101
 -- Chris Cheney <email address hidden> Fri, 04 Jun 2010 00:39:00 -0500

Changed in eucalyptus (Ubuntu Lucid):
status: Fix Committed → Fix Released
Changed in eucalyptus:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.