Questions on tomcat heap usage and garbage collection (avoiding O utOfMemoryError exceptions)

["Bang, Steinar" <St...@tandbergtv.com>]

Platform:	Intel PIII 797.499MHz, 256MB RAM
		Debian Woody GNU/Linux,
		kernel 2.2.19
		Blackdown J2SDK 1.3.1
		apache 1.3.19
		tomcat 3.2.3

When using httperf[1] to stresstest JSPs on the above 
platform, I'm running into the OutOfMemoryError 
exception problem:
	<http://www.jguru.com/faq/view.jsp?EID=64889>

The solution suggested in the URL above, is to increase
the heap size, but this is just pushing the problem
ahead.  The important thing for us, is that tomcat doesn't die.

If this is a problem for high load sites, JSP technology
would be unusable, and obviously it isn't.  So what
do others do?

Possible alternatives:
 1. increase the heap size to something astronomical,
    and just hope for the best? 
 2. tune apache/tomcat to restart more often, so that
    the max heap size aren't reached before the tomcat
    processes are restarted?
 3. catch the exception?
 4. use some kind of watchdog to restart tomcat?

Question 3 raises more questions:
 3a. What do you do when you catch the exception?
     There isn't much you _can_ do if you don't have
     any memory left.  Maybe force a garbage collection
     and then re-run the request?
 3b. Why do all the tomcat processes connected to
     apache processes die, if one of them gets an
     OutOfMemoryError exception

Which brings us to question 2: tuning apache/tomcat:
When reading the documentation, I thought that there
was supposed to be a single tomcat process, serving
all requests.  However top reports a lot of tomcat
processes, when I'm stresstesting the server, which
leads me to belive that there is a separate tomcat
process for each apache server process running.
Is this correct?

In any case; based on this assumption, I reduced
the number of currently running apache processes
from the default 150 to 20, with dramatically
changed behaviour: instead of running with a 20%
load, the machine was running at an 80%-90% load
with less physical memory used (restarting processes
is a lot of work).

However, even with the high load, the end result
was a lot more stable.  Watching the tomcat processes
in top, I could watch them grow up to around 82MB,
before there suddenly was a lot of processes with
around 19MB.

It wasn't completely stable though.  I set up
httperf to do 1 million requests, before I left
last night, and it only got to 51927 before tomcat
died.

I run httperf from a different machine (PIII 800MHz),
using the command:
	httperf --server=no-video2 --port=80 --uri=/tps-sba/Welcome.jsp
--rate 150 --num-conns=1000000 --num-calls=1
(--num-conns varies.  I have mostly been using 
10000).


The memory usage observed in top, also raises some
questions related to my question 1:
 1a. Why does tomcat processes always start at around
     19MB, no matter what I've set the initial heap
     size to be?
 1b. Why does the initial heap size seem to matter
     for my stability if the processes always start
     at 19MB?  Is it all in my head?

I'm interested in all direct experiences, ideas,
guesses, answers, etc.

Thanx!


- Steinar


[1] <http://freshmeat.net/projects/httperf>


 

















This email, its content and any attachments is PRIVATE AND CONFIDENTIAL to
TANDBERG Television. If received in error please notify the sender and
destroy the original message and attachments.

Re: Questions on tomcat heap usage and garbage collection (avoiding O utOfMemoryError exceptions)

["Craig R. McClanahan" <cr...@apache.org>]

The most important task in a situation like this is to find out why memory
is being consumed in the first place.  Generally, this is caused by one of
the following types of factors:

* Creating lots and lots of session attributes in your
  applications (which causes the objects to stay allocated
  until the session times out or is invalidated)

* Creating a new session on every request from your
  stress test tool, which causes a new session object
  to be created each time.

The latter case is pretty insidious for doing stress testings, because JSP
pages are session-aware by default (unless you tell them not to).  This
behavior is not typical of real users, howevever, because they will be
returning the same session id (and therefore accessing the same session)
throughout their interaction with the server.

You should investigate whether httperf (or other tool) can let you
remember the session id that was assigned in the first place and then
reuse it on the subsequent requests, and/or whether your test JSP page can
temporarily be set to not use sessions so that this doesn't happen.

Craig


On Wed, 24 Oct 2001, Bang, Steinar wrote:

> Date: Wed, 24 Oct 2001 09:21:19 +0200
> From: "Bang, Steinar" <St...@tandbergtv.com>
> Reply-To: tomcat-user@jakarta.apache.org
> To: "'tomcat-user@jakarta.apache.org'" <to...@jakarta.apache.org>
> Subject: Questions on tomcat heap usage and garbage collection (avoiding
>     O utOfMemoryError exceptions)
>
> Platform:	Intel PIII 797.499MHz, 256MB RAM
> 		Debian Woody GNU/Linux,
> 		kernel 2.2.19
> 		Blackdown J2SDK 1.3.1
> 		apache 1.3.19
> 		tomcat 3.2.3
>
> When using httperf[1] to stresstest JSPs on the above
> platform, I'm running into the OutOfMemoryError
> exception problem:
> 	<http://www.jguru.com/faq/view.jsp?EID=64889>
>
> The solution suggested in the URL above, is to increase
> the heap size, but this is just pushing the problem
> ahead.  The important thing for us, is that tomcat doesn't die.
>
> If this is a problem for high load sites, JSP technology
> would be unusable, and obviously it isn't.  So what
> do others do?
>
> Possible alternatives:
>  1. increase the heap size to something astronomical,
>     and just hope for the best?
>  2. tune apache/tomcat to restart more often, so that
>     the max heap size aren't reached before the tomcat
>     processes are restarted?
>  3. catch the exception?
>  4. use some kind of watchdog to restart tomcat?
>
> Question 3 raises more questions:
>  3a. What do you do when you catch the exception?
>      There isn't much you _can_ do if you don't have
>      any memory left.  Maybe force a garbage collection
>      and then re-run the request?
>  3b. Why do all the tomcat processes connected to
>      apache processes die, if one of them gets an
>      OutOfMemoryError exception
>
> Which brings us to question 2: tuning apache/tomcat:
> When reading the documentation, I thought that there
> was supposed to be a single tomcat process, serving
> all requests.  However top reports a lot of tomcat
> processes, when I'm stresstesting the server, which
> leads me to belive that there is a separate tomcat
> process for each apache server process running.
> Is this correct?
>
> In any case; based on this assumption, I reduced
> the number of currently running apache processes
> from the default 150 to 20, with dramatically
> changed behaviour: instead of running with a 20%
> load, the machine was running at an 80%-90% load
> with less physical memory used (restarting processes
> is a lot of work).
>
> However, even with the high load, the end result
> was a lot more stable.  Watching the tomcat processes
> in top, I could watch them grow up to around 82MB,
> before there suddenly was a lot of processes with
> around 19MB.
>
> It wasn't completely stable though.  I set up
> httperf to do 1 million requests, before I left
> last night, and it only got to 51927 before tomcat
> died.
>
> I run httperf from a different machine (PIII 800MHz),
> using the command:
> 	httperf --server=no-video2 --port=80 --uri=/tps-sba/Welcome.jsp
> --rate 150 --num-conns=1000000 --num-calls=1
> (--num-conns varies.  I have mostly been using
> 10000).
>
>
> The memory usage observed in top, also raises some
> questions related to my question 1:
>  1a. Why does tomcat processes always start at around
>      19MB, no matter what I've set the initial heap
>      size to be?
>  1b. Why does the initial heap size seem to matter
>      for my stability if the processes always start
>      at 19MB?  Is it all in my head?
>
> I'm interested in all direct experiences, ideas,
> guesses, answers, etc.
>
> Thanx!
>
>
> - Steinar
>
>
> [1] <http://freshmeat.net/projects/httperf>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> This email, its content and any attachments is PRIVATE AND CONFIDENTIAL to
> TANDBERG Television. If received in error please notify the sender and
> destroy the original message and attachments.
>

Re: Questions on tomcat heap usage and garbage collection (avoiding OutOfMemoryError exceptions)

[Pier Fumagalli <pi...@betaversion.org>]

Bang, Steinar at Steinar.bang@tandbergtv.com wrote:

> Which brings us to question 2: tuning apache/tomcat:
> When reading the documentation, I thought that there
> was supposed to be a single tomcat process, serving
> all requests.  However top reports a lot of tomcat
> processes, when I'm stresstesting the server, which
> leads me to belive that there is a separate tomcat
> process for each apache server process running.
> Is this correct?

Nope.. That's the idiotish implementation of Threads under Linux. Under
Linux every thread is seen as a process by "ps" so, if you have an
application using 500 threads, "ps" will report 500 processes (this is
because Linux has just one scheduling table for all, unlike real operating
systems which have two, and differentiate between threads and processes).

Short test to prove it, you know how much ram you have in your machine, do a
"ps" and see the amount of Ram used by each single "java" process, sum it
all up, see how much swap space you're using, and "tadaaaa" You have more
memory than you thought....

Regarding your OutOfMemoryException, when that's thrown in one thread, all
other threads die too, because all threads reside in the same memory space,
so, if one thread doesn't have memory left, so not even the others. (they're
threads, not processes).

    Pier (I hate Linux's threading model)