Would like to monitor memory use offline

["Richard S. Huntrods" <hu...@nucleus.com>]

I've been running Tomcat for many versions now, mostly without incident. 
However with the latest set of upgrades rather "forced" upon me all at 
once (instead of managed more properly), my application appears to have 
a severe memory leak.

System Info: OS is Solaris 10-u5 (2008); java 1.6.0_06-b02; 
apache-tomcat-6.0.16; mysql 5.0.51a-solaris10-x86_64. I have fast 
servers and plenty of memory (8 gigs). I'm running 1 gig stack and 
getting at least 2 GC/stack exceptions per day (sometimes more). Yes - 
it's a user/use triggered leak but I can't trace it further yet.

Of course what is odd is that there was NO memory leak using older 
versions of this stuff (Solaris 10 (2006), java 1.5.x, tomcat 5.5.12, 
mysql 5.0.16). I'm sure the memory leak was there, but it was "well 
masked". On the older system I was running 512 meg stack and it never 
gave GC or stack errors.

So, while I am actively trying to fix the memory leak, I still have to 
maintain these production servers at operatonal status (politics - don't 
ask). However, it's difficult as the memory leak is causing repeated GC 
and "out of stack" exceptions.

What I've noticed recently is that when using the manager application, I 
can watch the memory utilization grow and more memory get allocated (via 
refreshing the page), right up until the stack is used up and the main 
application crashes. However, if I'm watching it grow, and then log on 
to the server and reset tomcat (stop and then start tomcat), the memory 
use is back at the start. Thanks to session persistence, no users are 
"harmed" during this exercise.

So for the moment, while I try and debug the application, I can keep 
things running by having a cron job periodically reset tomcat for me. 
But this is really crude. Until I fix the memory leak, I'd like 
something a little bit more elegant.

SO - my question - is there a relatively easy way to create something 
(say a servlet) to watch the stack *just like I can do manually using 
the manager application* but email me when the stack approaches the 
memory limits?

Thanks,

-Richard



---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Would like to monitor memory use offline

[Mark Thomas <ma...@apache.org>]

Richard S. Huntrods wrote:
> SO - my question - is there a relatively easy way to create something 
> (say a servlet) to watch the stack *just like I can do manually using 
> the manager application* but email me when the stack approaches the 
> memory limits?

Richard,

Tomcat is open source so if you want to copy something Tomcat is doing all 
you need to do is look at the Tomcat source and copy it. Granted it can be 
a bit impenetrable in places but I don't think this is one of them.

1. You know from the manager page what URL you are interested in.
2. The web.xml for the manager app will tell you which servlet is serving 
that URL.
3. The Tomcat 6 source is in a single tree under 
http://svn.apache.org/repos/asf/tomcat/tc6.0.x/trunk/java/ so finding the 
class is easy.
4. A quick look at the doGet() method in that class points you towards 
StatusTransformer.writeVMState()
5. And there you have your answer - a simple API call.
http://java.sun.com/javase/6/docs/api/java/lang/Runtime.html#freeMemory()

For the e-mail side, take a look at the JavaMail example in the JNDI docs. 
http://tomcat.apache.org/tomcat-6.0-doc/jndi-resources-howto.html

Mark



---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Would like to monitor memory use offline

[Ben Stringer <be...@burbong.com>]

On Sat, 2008-08-02 at 19:18 -0700, Richard S. Huntrods wrote:
> 
> SO - my question - is there a relatively easy way to create something 
> (say a servlet) to watch the stack *just like I can do manually using 
> the manager application* but email me when the stack approaches the 
> memory limits?

Hi Richard,

I'd suggest running the tomcat JVM with verbose garbage collection
enabled eg. [1], then monitoring the log produced with a shell script.
This should show you when you are nearing your out of memory limit.

To track down the memory leak, try something like Lambda Probe [2] to
give you an overview of resource consumption and monitor it over time.

Cheers, Ben

[1]
-XX:+PrintGCTimeStamps
-Xloggc:/var/log/java/gc.log
-verbosegc
-XX:+PrintGCDetails

[2] http://www.lambdaprobe.org/d/index.htm




---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Would like to monitor memory use offline

[Sameer Acharya <ac...@yahoo.com>]

You can write a simple JSP which will run a freeMemory/totalMemory call in your JVM and possibly send a mail/log when the limits are reached.

You could set a refresh interval and have this page refresh say every 5 minutes in your browser.

Alternatively you can tweak with the manager app code.

If you are an administrator ten from a long term monitoring aspect you may want to explore the usage of Lambdaprobe.

http://www.lambdaprobe.org

-Sameer



--- On Sun, 8/3/08, Richard S. Huntrods <hu...@nucleus.com> wrote:

> From: Richard S. Huntrods <hu...@nucleus.com>
> Subject: Would like to monitor memory use offline
> To: users@tomcat.apache.org
> Date: Sunday, August 3, 2008, 7:48 AM
> I've been running Tomcat for many versions now, mostly
> without incident. 
> However with the latest set of upgrades rather
> "forced" upon me all at 
> once (instead of managed more properly), my application
> appears to have 
> a severe memory leak.
> 
> System Info: OS is Solaris 10-u5 (2008); java 1.6.0_06-b02;
> 
> apache-tomcat-6.0.16; mysql 5.0.51a-solaris10-x86_64. I
> have fast 
> servers and plenty of memory (8 gigs). I'm running 1
> gig stack and 
> getting at least 2 GC/stack exceptions per day (sometimes
> more). Yes - 
> it's a user/use triggered leak but I can't trace it
> further yet.
> 
> Of course what is odd is that there was NO memory leak
> using older 
> versions of this stuff (Solaris 10 (2006), java 1.5.x,
> tomcat 5.5.12, 
> mysql 5.0.16). I'm sure the memory leak was there, but
> it was "well 
> masked". On the older system I was running 512 meg
> stack and it never 
> gave GC or stack errors.
> 
> So, while I am actively trying to fix the memory leak, I
> still have to 
> maintain these production servers at operatonal status
> (politics - don't 
> ask). However, it's difficult as the memory leak is
> causing repeated GC 
> and "out of stack" exceptions.
> 
> What I've noticed recently is that when using the
> manager application, I 
> can watch the memory utilization grow and more memory get
> allocated (via 
> refreshing the page), right up until the stack is used up
> and the main 
> application crashes. However, if I'm watching it grow,
> and then log on 
> to the server and reset tomcat (stop and then start
> tomcat), the memory 
> use is back at the start. Thanks to session persistence, no
> users are 
> "harmed" during this exercise.
> 
> So for the moment, while I try and debug the application, I
> can keep 
> things running by having a cron job periodically reset
> tomcat for me. 
> But this is really crude. Until I fix the memory leak,
> I'd like 
> something a little bit more elegant.
> 
> SO - my question - is there a relatively easy way to create
> something 
> (say a servlet) to watch the stack *just like I can do
> manually using 
> the manager application* but email me when the stack
> approaches the 
> memory limits?
> 
> Thanks,
> 
> -Richard
> 
> 
> 
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail:
> users-help@tomcat.apache.org


      

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org