memory usage (longish)

[Simon Juden <Si...@ProQuestAlison.com>]

Got a funny problem with memory usage.  Initially assumed likely to be a
linux issue.  Had the problem on RedHat 6.2, upgraded to RedHat 7.3, still
have the problem.  Not sure now if it's a linux or tomcat issue.

Set-up/environment: Red Hat (now 7.3, with ext3), 1xi686 processor, 1.5Gb
memory, Apache 1.3.26, Tomcat 3.2.4, JVM 1.3.1_01 (for the curious there are
plans to upgrade these latter three elements but not yet).  There are four
of these in a webserver farm (hardware-accelerated SSL and loadbalancing
happens before traffic gets near the farm).  The problem happens on two of
the four boxes occasionally, on one fairly often and on the fourth almost
every other day.

There are two ways the problem manifests itself.  The most usual is as
follows.

Run webapp in tomcat (max heap size set to 500Mb).  After some time Tomcat
falls over with a an OutOfMemoryError.  When it does this it is only using
about 150Mb total (so in particular heap well under 500mb max) reported
under ps, top etc.

However the box reports all but a few megs of the 1.5Gb in use.  It reports
all this as still in use when every single java process (and every process
my clients wrote) is shut down.  There's about 60 Mb-worth of usage from the
CPU processes and that's all I can account for.  There is virtually no swap
space usage though there's plenty available.  The output of free (after
tomcat and all non-canonical processes shut down) is:

[root@prodfarm3 logs]# free
             total       used       free     shared    buffers     cached
Mem:       1548084    1351560     196524          0     106900    1082072
-/+ buffers/cache:     162588    1385496
Swap:       538136          0     538136

Load averages are not high throughout any of this (1.6-1.7 or so).

I know the DMA can take memory (but surely not this much and not for so
long).  I can't for the life of me think what could be using all the rest.  

The other way the problem manifests itself is that Tomcat falls over with an
OutOfMemoryError but there are still several hundred megs still available on
the linux box.  The webapp does not allocate "huge" objects - I can fully
load it on the test box with a bunch of (robot) users and it uses about 200
megs or so.  So how it finds itself out of memory in this instance I don't
know.  I'm not sure if this is the same problem or a different one.

I've written a few robots and bombarded a test box (identical configuration,
except less physical memory, only 250Mb) with a bunch of concurrent user
sessions.  Different phenomena totally: load average soars (to about 8-9)
but java correctly handles its memory and the thing doesn't die.  However
after an extended run turning tomcat off again leaves 248 of 250Mb reported
in use (but this doesn't seem to be a problem when rerunning stuff).  But my
point is this isn't happening because the app's getting overloaded
(threadpool for tomcat has max size of 200 but it's never used more than
120-130 threads).

Not to put too fine a point on it: help ;-)  Are there any FMs I should be
R'ing but haven't found (I've tried my favourite metamanual (Google) and the
red hat and tomcat list archives)?  Are there configuration settings that
could affect/cause this kind of behaviour?  I'm kind of running out of
ideas...


_________________________________________________________________________

ProQuest Alison

Please note Alison Associates is now ProQuest Alison - this is purely a change of name to bring us in line with the rest of the ProQuest group.  As a result our e-mail domain has changed to proquestalison.com.  We will also continue to accept messages addressed to alisonassociates.com until 1st January 2003.  Please can you ensure that you change any address book entries that you have to use the new proquestalison.com addresses.  

The information contained in this e-mail and any attached files is intended only for the use of the person(s) to whom it is addressed and may be privileged, confidential and exempt from disclosure under applicable law. The views of the author may not necessarily reflect the views of the Company. If you are not the intended recipient please do not copy or convey this message or any attached files to any other person but delete this message and any attached files and notify us of incorrect receipt via e-mail to mailadmin@proquestalison.com.


_____________________________________________________________________
This message has been checked for all known viruses by MessageLabs.

--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>