Re: OutOfMemoryError - The Real Problem?

[Max Hugen <ma...@hugen.com.au>]

I've been plugging along with this problem, and I think that the issue is
with the maximum number of threads set by default in Apache - 256. I
monitored the number of processes running during a reboot and found as
follows:

                Processes    Started | Total

On reboot                         37      37
Start Oracle Db 1             18      55
Start Oracle Db 2             14      69
Start Tomcat App 1          41    110
Start Tomcat App 2          35    145
Start Tomcat App 3          36    181
Start Tomcat App 4          37    218
Start Tomcat App 5          36    254

Well, at 254, and only 256, I can see why it was crashing out. The
"OutOfMemoryError" seems a bit of a poor message - the important issue here
seems that I've run out of available processes. And of course, the above
with 5 tomcat engines running, its going to crash as soon as Oracle gets
requests and spawns more processes.

I'm a bit surprised that Tomcat needs so many processes per instance, and
also that the Linux/Apache config is brought to its knees if you hit the
maximum number of processes? Does anyone who runs a number of Internet
applications using JSP have any suggestions here? Seems crazy that I can
only run about 4 (tiny) apps on a server with 512Mb of memory and heaps of
hard drive space!

Next step, I guess, is to look into recompiling Apache with a higher number
of processes specified. Not much fun for a rookie, especially when some
applications are already in service. Aaarrggh!

Any and all suggestions gratefully devoured!
Max Hugen

Re: OutOfMemoryError - The Real Problem?

[Jacek Prucia <ja...@7bulls.com>]

[...]
> I'm a bit surprised that Tomcat needs so many processes per instance,

This is because of Linux kernel and JVM design, not Tomcat. AFAIK you can't
do fork() in Java, but instead you have threads. However Linux kernel only
recognizes processes, not threads (that is: stable Linux threads packages
are pure user space threads). People that do JVMs decided to implement Java
threads as Linux processes to achieve load balancing on multiCPUs
architectures.

Why? Look: If you would have 2 CPUs, and Java threads implemented as Linux
threads - then you would have only one CPU dealing with your application,
and other CPU beeing idle all the time. Since Java threads are implemented
as Linux processes - they are distributed on two CPUs. This creates also
other problem - fork() is much slower than pthread_create() which is
probably one of reasons behind not-so-good Linux JVM performance.

Some JDKs (AFAIK IBM) had option to switch between green threads (Java
threads mapped to fork()) and native threads (Java threads mapped to Linux
threads). I think that IBM was first to point that Linux kernel doesn't
recognize threads, and also supplied a beta patch for Linux kernel. So far
it doesn't have effect on Linux kernel gurus (which value stability over
functionality, and threads tends to introduce some complexity, which leads
to errors...)

Try to find other JDK (IBM, blackdown.org, kaffe.org) and see if there's a
native threads switch. Be sure to look at
http://www.ibm.com/developerworks/java/ for articles related Java and
Linux. You might be able to get more information there.

> and also that the Linux/Apache config is brought to its knees if you hit
the
> maximum number of processes?

This is quite strange. It might be that you didn't hit maximum number of
processes, but run out of some other resource like file descriptors. Apache
(when reaching MaxClients) usually is locking down clients (makes them
wait) and outputs to ErrorLog warning message about raising MaxClients
directive.
 
> Next step, I guess, is to look into recompiling Apache with a higher
> number
> of processes specified. Not much fun for a rookie

This is easy... just find HARD_SERVER_LIMIT in httpd.h and raise it to
whatever you'll find necesary. Like I said: this just might not be the
problem...

-- 
Jacek Prucia
7bulls.com S.A.