You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Todd Hivnor <sp...@yahoo.com> on 2009/05/11 20:11:09 UTC
Preventing OutOfMemoryError: Java heap space
I have a Java application running under Tomcat 6.0.18
on Ubuntu. This is using Sun's 1.6.0_07 JVM. I know
how to set the max heap space by setting -Xmx256m
in CATALINA_OPTS. But with a lot of sessions, I still
have the possibility of running out of heap space. My
application uses a lot of memory per user. I know I
should avoid this, but at this point I can't dramatically
change the software. And most of the time we have
a small number of users anyway.
I would like to proactively avoid running out of heap
space. I would like people get a "Server Too Busy"
message, _before_ the heap is actually exhausted.
I would rather serve 40 users well than 45 users
poorly.
I'm wondering if there are any Tomcat
configuration options, or filters, or whatever,
which might address this.
My current idea, which I think is solid, is to
check the available memory before populating
any new Session instances. My current
test uses the following formula to determine
the amount of space available:
Runtime.getRuntime().maxMemory()
-Runtime.getRuntime().totalMemory()
+Runtime.getRuntime().freeMemory();
In other words, take the amount of memory we COULD
allocate, subtract the amount we HAVE allocated, and
then add back the amount that is allocated but unused.
To phrase it another way, add the amount of free memory to the
amount of unallocated memory. In practice this formula
approaches zero as I get closer and closer to running out
of heap space. So it seems to be correct.
I’m wondering what people think of this idea,
or if there is a better / simpler way to
accomplish my goal.
Opinions? Ideas?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by George Sexton <ge...@mhsoftware.com>.
I think the results are going to be pretty erratic.
The issue that I see is that the garbage collector operation is (to my
knowledge) not deterministic.
IOW, you're not really accounting for memory that could be garbage
collected. So, I think that you'll have a systematic bias showing less
memory than is actually available.
I suppose you could throw in
Runtime.getRuntime().runFinalization()
Runtime.getRuntime().gc()
However, runtime.gc() just "suggests that the Java Virtual Machine
expend effort toward recycling unused objects in order to make the
memory they currently occupy available for quick reuse"
Todd Hivnor wrote:
> I have a Java application running under Tomcat 6.0.18
> on Ubuntu. This is using Sun's 1.6.0_07 JVM. I know
> how to set the max heap space by setting -Xmx256m
> in CATALINA_OPTS. But with a lot of sessions, I still
> have the possibility of running out of heap space. My
> application uses a lot of memory per user. I know I
> should avoid this, but at this point I can't dramatically
> change the software. And most of the time we have
> a small number of users anyway.
>
> I would like to proactively avoid running out of heap
> space. I would like people get a "Server Too Busy"
> message, _before_ the heap is actually exhausted.
>
> I would rather serve 40 users well than 45 users
> poorly.
>
> I'm wondering if there are any Tomcat
> configuration options, or filters, or whatever,
> which might address this.
>
> My current idea, which I think is solid, is to
> check the available memory before populating
> any new Session instances. My current
> test uses the following formula to determine
> the amount of space available:
>
> Runtime.getRuntime().maxMemory()
> -Runtime.getRuntime().totalMemory()
> +Runtime.getRuntime().freeMemory();
>
> In other words, take the amount of memory we COULD
> allocate, subtract the amount we HAVE allocated, and
> then add back the amount that is allocated but unused.
> To phrase it another way, add the amount of free memory to the
> amount of unallocated memory. In practice this formula
> approaches zero as I get closer and closer to running out
> of heap space. So it seems to be correct.
>
> I’m wondering what people think of this idea,
> or if there is a better / simpler way to
> accomplish my goal.
>
> Opinions? Ideas?
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
George Sexton
MH Software, Inc.
Voice: +1 303 438 9585
URL: http://www.mhsoftware.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear Todd,
> I'm not sure I can use session counting, as my
> session size is not consistent. I could try to estimate
> the size of each session, and keep a global counter,
> but that seems like a lot of work.
JMX offers quite a bit of insight in what your GC is doing. You could
implement throttling as a function of the number of full GC's that
your system experiences. While full GC's do happen from time to time,
you could trigger if they happen more than once per so-many seconds.
Actual values would probably be highly application-specific.
I agree with someone else that 256MB is not a whole lot or RAM for a
busy server. With current prices of RAM I would go for a gig or two
and take it from there.
> I understand that garbage collection is, err, whimsical,
> but I think I'm going to give it a go anyway.
> I will keep in mind that the memory results
> from Runtime will probably under-report the
> available memory. I'm going to add an explicit
> request for garbage collection, _before_ the memory
> becomes seriously depleted.
Here is what happens if you do http://java-monitor.com/forum/showthread.php?t=188
It is not pretty, I can tell you that. I add -XX:-DisableExplicitGC
to my JAVA_OPTS for this very reason.
--
Kees Jan
http://java-monitor.com/forum/
kjkoster@kjkoster.org
06-51838192
The secret of success lies in the stability of the goal. -- Benjamin
Disraeli
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Todd Hivnor [mailto:spambox_98103@yahoo.com]
> One challenge with Peter's suggestion of tracking the
> number of sessions myself is that I have a collection
> of webapps. So I can't just set up a shared static counter;
> I need a counter which works across multiple webapps.
> The only way I know to do that is to use a text file,
> and take care about locking the file before updating it.
> Or perhaps I can use ServletContext.
Some classes can be shared across webapps, though the precise approach varies depending on the version. See the "Classloader how-to" for your Tomcat version at http://tomcat.apache.org for details. Pre-6.0, you could load a class through the common or shared classloaders; in 6.0, your options are reduced somewhat. That class could keep a common integer. It doesn't have to be a straight session count; you could weight it based on the relative sizes of the sessions. A heavy session increments the counter by 10, a light one by 1.
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Todd Hivnor [mailto:spambox_98103@yahoo.com]
> Subject: Re: Preventing OutOfMemoryError: Java heap space
>
> In my tests, as the RAM becomes depleted, the server
> response becomes slower and slower.
That's because you're going through garbage collections at an ever increasing frequency, with little benefit.
> Yes of course I can throw more RAM at the problem.
> But I would still like to have a graceful response to the
> "overloaded" scenario. More RAM is just delaying the
> problem.
You need to size the system for the expected load, and refuse requests when you approach some arbitrary threshold. You also need to check very carefully for memory leaks in your webapps, since your symptoms are typical of that kind of problem. If you aren't doing so already, run in a 64-bit environment, allowing a much larger heap; your current setting of -Xmx256m is tiny in today's world. Just make sure you have enough RAM to support the heap, otherwise paging will kill your performance.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by Todd Hivnor <sp...@yahoo.com>.
>> But would it not be easier to catch the OOM exception and then
>> return a "sorry, server overloaded" page to the browser ?
> It's difficult to do that when the OOME may occur in Tomcat code,
> outside of control of the webapp.
Wow I had assumed I could always catch this type of exception.
Thanks for clarifying.
In any case, tho, I specifically don't want the OOME exception to occur,
largely because it takes so much _time_ to occur. In my tests, as the RAM
becomes depleted, the server response becomes slower and slower.
The actual OOME takes over a minute to appear. Apparently the JVM
is making a heroic effort to satisfy the request.
>> (I would mischievously also add a suggestion to buy some more RAM,
>> as an even cheaper alternative).
>That's the only reasonable solution, if the OP is unwilling to
> implement the throttling Peter suggested.
Yes of course I can throw more RAM at the problem.
But I would still like to have a graceful response to the
"overloaded" scenario. More RAM is just delaying the
problem.
I was hoping there was a simple solution I had overlooked.
One challenge with Peter's suggestion of tracking the
number of sessions myself is that I have a collection
of webapps. So I can't just set up a shared static counter;
I need a counter which works across multiple webapps.
The only way I know to do that is to use a text file,
and take care about locking the file before updating it.
Or perhaps I can use ServletContext.
Overall session counting sounds like my best option.
Again, thanks for all the clues & suggestions.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Subject: Re: Preventing OutOfMemoryError: Java heap space
>
> > I'm going to add an explicit request for garbage
> > collection, _before_ the memory becomes seriously
> > depleted.
To the OP: explicitly calling GC is a complete waste of time. You have to do something to free up memory by shutting down sessions or otherwise terminating activity within the JVM. Your calling GC won't accomplish anything that the JVM automatically calling GC doesn't do.
> But would it not be easier to catch the OOM exception and then
> return a "sorry, server overloaded" page to the browser ?
It's difficult to do that when the OOME may occur in Tomcat code, outside of control of the webapp.
> (and maybe then also trigger a GC just for tidiness)
Again, a waste of time - the OOME is not presented until after a major GC has been attempted.
> Then this thread starts listening for OOM exceptions (I just
> imagine that this can be done).
I don't know of any means to do that.
> (I would mischievously also add a suggestion to buy some more RAM,
> as an even cheaper alternative).
That's the only reasonable solution, if the OP is unwilling to implement the throttling Peter suggested.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
Posted by Peter Crowther <Pe...@melandra.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Thanks, Chuck and Peter, for the clarifications on OOM.
> I believe that unconsciously, with my "large object
> reservation" theory,
> I was vaguely remembering something I had read some time in the past.
> So I searched Google for "java +parachute +memory" and this
> is something
> I found :
>
> http://mail-archives.apache.org/mod_mbox/tomcat-dev/200703.mbo
x/%3C20070325171940.34DAE1A9838@eris.apache.org%3E
>
> Does this have any bearing on the OP's issue ?
>
> It is past me capacities to see if it is relevant or not (I don't mean
> the above post per se, but the idea).
Nice.
The idea's relevant... as long as you realise that all it's doing is defending a very few critical areas of Tomcat's code against some (not all) OOMEs. Another thread could allocate heap memory between lines in this code, leading to unexpected failures because the memory freed by the parachute has been used elsewhere; and it doesn't defend all areas of Tomcat's code (I think - Filip will no doubt correct me).
It's a great way of helping to make sure that, most of the time, it's at least possible to log an OOME - assuming the application passes it up the stack, which most will. It's not a general solution without a lot more work; each webapp would have to do something similar at each point that it might allocate heap memory.
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by André Warnier <aw...@ice-sa.com>.
Thanks, Chuck and Peter, for the clarifications on OOM.
I believe that unconsciously, with my "large object reservation" theory,
I was vaguely remembering something I had read some time in the past.
So I searched Google for "java +parachute +memory" and this is something
I found :
http://mail-archives.apache.org/mod_mbox/tomcat-dev/200703.mbox/%3C20070325171940.34DAE1A9838@eris.apache.org%3E
Does this have any bearing on the OP's issue ?
It is past me capacities to see if it is relevant or not (I don't mean
the above post per se, but the idea).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by Rainer Jung <ra...@kippdata.de>.
On 13.05.2009 08:07, Peter Crowther wrote:
>> From: André Warnier [aw@ice-sa.com] would it not be easier to catch
>> the OOM exception and then return a "sorry, server overloaded" page
>> to the browser ?
>
> At that point, it's too late. A thread, somewhere in the system,
> tried to allocate some memory for an object and couldn't. This could
> happen anywhere! It might be while allocating the buffer to read the
> request, or the buffer to send the response, or even the socket
> instance from which to read the request.
>
> There are then no guarantees. You may be unable to send anything
> back to the client, as you may be unable to access the socket because
> the OOM was caused by trying to create that socket. Or the thread
> that got the exception might be one of the threads processing
> incoming requests. Once the system encounters an out-of-memory
> error, you pretty much have to stop the process and start again, as
> you generally have no idea what else has failed. You can't even save
> any outstanding work to file, as there's no guarantee you can
> allocate the memory for the file object - if MS Word runs out of
> memory, you just lost your document!
>
> This isn't unique to Java, by the way - if you want a coding
> nightmare, try to handle *all* malloc() fails in a reasonably sized C
> program. Or kalloc() calls in a kernel :-). When you realise that
> running out of memory means you can no longer guarantee to allocate
> any new memory for any purpose at all, the scale of the problem
> becomes apparent.
+1!
Another example for "no more guarantees": after an OOME in heap, your
business logic might be broken in a very unexpected way! Don't let any
Java process do more work once it had an OOME in heap.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
Posted by Peter Crowther <Pe...@melandra.com>.
> From: André Warnier [aw@ice-sa.com]
> would it not be easier to catch the OOM exception and then return a
> "sorry, server overloaded" page to the browser ?
At that point, it's too late. A thread, somewhere in the system, tried to allocate some memory for an object and couldn't. This could happen anywhere! It might be while allocating the buffer to read the request, or the buffer to send the response, or even the socket instance from which to read the request.
There are then no guarantees. You may be unable to send anything back to the client, as you may be unable to access the socket because the OOM was caused by trying to create that socket. Or the thread that got the exception might be one of the threads processing incoming requests. Once the system encounters an out-of-memory error, you pretty much have to stop the process and start again, as you generally have no idea what else has failed. You can't even save any outstanding work to file, as there's no guarantee you can allocate the memory for the file object - if MS Word runs out of memory, you just lost your document!
This isn't unique to Java, by the way - if you want a coding nightmare, try to handle *all* malloc() fails in a reasonably sized C program. Or kalloc() calls in a kernel :-). When you realise that running out of memory means you can no longer guarantee to allocate any new memory for any purpose at all, the scale of the problem becomes apparent.
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by André Warnier <aw...@ice-sa.com>.
Todd Hivnor wrote:
>>> From: Todd Hivnor [spambox_98103@yahoo.com]
>
>>> I would like to proactively avoid running out of heap
>>> space. I would like people get a "Server Too Busy"
>>> message, _before_ the heap is actually exhausted.
>>> I would rather serve 40 users well than 45 users
>>> poorly.
>
>> Rather than monitor memory, which is subject to GC
>> whims as George points out, I'd suggest monitoring
>> the number of sessions in your application - find a
>> number of sessions you can live with, keep a (suitably
>> synchronized) counter that's incremented and decremented
>> as sessions are created and destroyed, and don't start
>> new sessions if that counter is higher than your threshold.
>
>> - Peter
>
>
>
> Peter / George, thanks for the input.
>
> I'm not sure I can use session counting, as my
> session size is not consistent. I could try to estimate
> the size of each session, and keep a global counter,
> but that seems like a lot of work.
>
> I understand that garbage collection is, err, whimsical,
> but I think I'm going to give it a go anyway.
> I will keep in mind that the memory results
> from Runtime will probably under-report the
> available memory. I'm going to add an explicit
> request for garbage collection, _before_ the memory
> becomes seriously depleted.
>
> Thanks for the suggestions!
>
Sorry for jumping in as a naive and frankly incompetent java and Tomcat
programmer, and do not feel obliged to respond if the question is really
stupid.
But would it not be easier to catch the OOM exception and then return a
"sorry, server overloaded" page to the browser ?
(and maybe then also trigger a GC just for tidiness)
Or a more ambitious scheme : at Tomcat startup, a thread is created
which immediately allocates some largish object and keeps it. Then this
thread starts listening for OOM exceptions (I just imagine that this can
be done). When one occurs, the thread releases the object and triggers
a GC. Then it tries to allocate the object again. Then.. I'm running
out of ideas and technique. It was just a thought, in the direction of
making GC a bit more deterministic.
(I would mischievously also add a suggestion to buy some more RAM, as an
even cheaper alternative).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Preventing OutOfMemoryError: Java heap space
Posted by Todd Hivnor <sp...@yahoo.com>.
>> From: Todd Hivnor [spambox_98103@yahoo.com]
>> I would like to proactively avoid running out of heap
>> space. I would like people get a "Server Too Busy"
>> message, _before_ the heap is actually exhausted.
>
>> I would rather serve 40 users well than 45 users
>> poorly.
> Rather than monitor memory, which is subject to GC
> whims as George points out, I'd suggest monitoring
> the number of sessions in your application - find a
> number of sessions you can live with, keep a (suitably
> synchronized) counter that's incremented and decremented
> as sessions are created and destroyed, and don't start
> new sessions if that counter is higher than your threshold.
> - Peter
Peter / George, thanks for the input.
I'm not sure I can use session counting, as my
session size is not consistent. I could try to estimate
the size of each session, and keep a global counter,
but that seems like a lot of work.
I understand that garbage collection is, err, whimsical,
but I think I'm going to give it a go anyway.
I will keep in mind that the memory results
from Runtime will probably under-report the
available memory. I'm going to add an explicit
request for garbage collection, _before_ the memory
becomes seriously depleted.
Thanks for the suggestions!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Preventing OutOfMemoryError: Java heap space
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Todd Hivnor [spambox_98103@yahoo.com]
> I would like to proactively avoid running out of heap
> space. I would like people get a "Server Too Busy"
> message, _before_ the heap is actually exhausted.
> I would rather serve 40 users well than 45 users
> poorly.
Rather than monitor memory, which is subject to GC whims as George points out, I'd suggest monitoring the number of sessions in your application - find a number of sessions you can live with, keep a (suitably synchronized) counter that's incremented and decremented as sessions are created and destroyed, and don't start new sessions if that counter is higher than your threshold.
- Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org