Re: Tomcat 6.0.35 Crashed again

[Daniel Mikusa <dm...@vmware.com>]

On Apr 11, 2013, at 1:09 AM, saumil shah wrote:

> Hello All,
> We are using Tomcat 6.0.35 for our production system with 64 bit JVM (1.6.33) on Windows 2008 R2 SP1. Our physical memory is 24gb. Load is ~ 100 concurrent sessions.
> The Tomcat crashed again with OutOfMemoryError: Java heap space error. We are using COTS product SAP Business Objects. Upon providing Tomcat logs ,
> SAP came with recommendation of increasing the heap size . Our current Java params are :
> -Xms512m -Xmx1024m
> -XX:MaxPermSize=512m
> SAP's recommendation is to make it ...
> -Xms1024m -Xmx4096m
> We have ample of free memory on the box ...
> 1a. but the question was heap size as Max 4GB , would it cause application delay when java GC is due? It's NOT time-sensitive application.

A bigger heap will generally take longer to clean, but 4GB is not particularly large.  

If you're concerned about the performance of the garbage collector, I would suggest that you enable garbage collection logging.  This adds very little overhead, simply logging an entry for every garbage collection that occurs.  Each entry will indicate memory before and after the GC, plus how much time was spent performing the collection.  You can use that to analyze the performance for your application.


> 1b. Also, when putting MAx as 4GB , does it mean JVM will make base effort to get 4GB MAX memory for heap , or is that guaranteed? Based on my reading, it appeared that only Min, was guaranteed.

The JVM will start at -Xms and, if needed, expand up to -Xmx.


> 1c. Is it then recommended to have Min and Max as 4GB... so that Heap doesn't have to resize and 4GB is guaranteed at the time of initialization?

It depends.  This can be good because it will prevent a few full GC's as the JVM expands from -Xms to -Xmx.  However, it can also waste memory if you allocate too much.  


> 1d. I have Tomcat service installer..... I have provided Min and Max heap size values under Tomcat --> Java tab --> Initial Memory Pool and Maximum memory pool ...... I am assuming that means Min and Max values for Heap.....is this correct?

Yes, I believe so.


> 2. When trying to monitor Tomcat , using JConsole and VisualVM ..... do I need to add JMX agent and port ....  in Java tab of Tomcat and then restart , or is it possible to Monitor tomcat , without rebooting it ? Since, it would not be possible to re-boot tomcat in production for it.

Normally when you run jconsole or jvisualvm on the same machine as your Tomcat instance, you can connect without needing any additional configuration.  This is a special case because the two processes are running on the same machine.  However, I seem to remember needing to configure JMX when running Tomcat as a Windows Service.  Perhaps one of the Windows guys on this list can clarify.


> 3. Once JConsole and/or VisualVM is setup ... I can look at Heap/Permgen , memory , threads etc...... I was wondering how do I see memory leaks and what is causing it , using these tools?

Spotting memory leaks is tricky and I'm not sure that what you're proposing will be much help.  You could certainly look at the live graphs and watch for a trend in memory usage increasing, but if the leak is small it would be very hard to spot.

I would suggest that you add -XX:+HeapDumpOnOutOfMemoryError to your JVM options.  This will create a heap dump when you encounter an OOME.  You can then view that heap dump in jvisualvm or a profiler to see what was consuming the memory.  


> 4. Also, I am seeing that Tomcat is deploying the webapps on start-up .... is there anyway to avoid Tomcat to deploy the application on start-up....as the applications are already deployed and exploded?
> Example is below :Apr 5, 2013 7:17:27 PM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive AnalyticalReporting.war

Tomcat has to deploy your application when it starts or your user's won't be able to access it.  This is normal and expected behavior.


> Apr 5, 2013 7:17:28 PM org.apache.catalina.core.StandardContext addApplicationListenerINFO: The listener "com.sun.faces.config.ConfigureListener" is already configured for this context. The duplicate definition has been ignored.

Looks like you might have a duplicate definition in your application's web.xml.  You should remove it, but you don't have to.  This is just Tomcat telling you that the duplicate was ignored.

Dan


> 
> As you can tell , I am novice to Tomcat. Please let me know if I missed on something.
> Appreciate all your help. 		 	   		  


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat 6.0.35 Crashed again

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Dan,

On 4/11/13 9:52 AM, Daniel Mikusa wrote:
> On Apr 11, 2013, at 1:09 AM, saumil shah wrote:
> 
>> Hello All, We are using Tomcat 6.0.35 for our production system
>> with 64 bit JVM (1.6.33) on Windows 2008 R2 SP1. Our physical
>> memory is 24gb. Load is ~ 100 concurrent sessions. The Tomcat
>> crashed again with OutOfMemoryError: Java heap space error. We
>> are using COTS product SAP Business Objects. Upon providing
>> Tomcat logs , SAP came with recommendation of increasing the heap
>> size . Our current Java params are : -Xms512m -Xmx1024m 
>> -XX:MaxPermSize=512m SAP's recommendation is to make it ... 
>> -Xms1024m -Xmx4096m We have ample of free memory on the box ... 
>> 1a. but the question was heap size as Max 4GB , would it cause 
>> application delay when java GC is due? It's NOT time-sensitive 
>> application.
> 
> A bigger heap will generally take longer to clean, but 4GB is not 
> particularly large.

Yes and no: GC time is dictated by the number of live objects (and how
much space they take up), not by the size of the heap space or the
amount of garbage. Moving a chunk of data from one heap space to
another is fairly quick, but if there are millions of tiny moves (8
bytes here, 8 bytes there), that's what takes the time. If you have a
10MiB byte array, it's going to move faster than 1M 8-byte pointers
randomly-scattered throughout the heap space.

> If you're concerned about the performance of the garbage collector,
>  I would suggest that you enable garbage collection logging. This
> adds very little overhead, simply logging an entry for every
> garbage collection that occurs. Each entry will indicate memory
> before and after the GC, plus how much time was spent performing
> the collection. You can use that to analyze the performance for
> your application.

Again, yes and no: when focusing on performance of the garbage
collector, enabling GC logging is pretty much the first step you want
to take. On the other hand, performance doesn't seem to be an issue,
here: (lack of) stability is the current issue.

>> 1b. Also, when putting MAx as 4GB , does it mean JVM will make 
>> base effort to get 4GB MAX memory for heap , or is that
>> guaranteed? Based on my reading, it appeared that only Min, was
>> guaranteed.
> 
> The JVM will start at -Xms and, if needed, expand up to -Xmx.
> 
>> 1c. Is it then recommended to have Min and Max as 4GB... so that
>>  Heap doesn't have to resize and 4GB is guaranteed at the time
>> of initialization?
> 
> It depends.  This can be good because it will prevent a few full
> GC's as the JVM expands from -Xms to -Xmx.  However, it can also
> waste memory if you allocate too much.

+1

Re-sizing the heap does take time. If you expect to fully-utilize a
4GiB heap, you may as well allocate it all at once to avoid
heap-thrashing just to get up to 4GiB.

- -Xms4096M -Xmx4096M


On the other hand, the OP's problem is likely to be an actual *leak*,
in which case a huge heap only prolongs the inevitable. If I were
debugging this on a test rig, the first thing I would do is *reduce*
the size of the heap to make heap analysis easier.

>> 3. Once JConsole and/or VisualVM is setup ... I can look at 
>> Heap/Permgen , memory , threads etc...... I was wondering how do
>> I see memory leaks and what is causing it , using these tools?
> 
> Spotting memory leaks is tricky and I'm not sure that what you're 
> proposing will be much help.  You could certainly look at the live 
> graphs and watch for a trend in memory usage increasing, but if
> the leak is small it would be very hard to spot.
> 
> I would suggest that you add -XX:+HeapDumpOnOutOfMemoryError to
> your JVM options. This will create a heap dump when you encounter
> an OOME. You can then view that heap dump in jvisualvm or a
> profiler to see what was consuming the memory.

+1

Heap dumps are no fun to analyze, but you might notice that you have a
whole bunch of objects you didn't expect to be sitting-around in the
heap (like java.sql.Connection, or huge byte arrays, or whatever).

I use YourKit (full disclosure: I get a free license as an ASF
committer) and it's got nice utilities for scanning heap snapshots for
some obvious potential problems (heap hogs, duplicate objects, arrays
with nothing but nulls in them, etc.). Most heap analyzers have tools
to let you find out how many objects of a particular type you have,
etc. and that may give you a quick idea of what may be going on. If
this is the first time you've ever looked at a heap dump, don't be
scared when you see the number of byte[], char[], and String objects
are still reachable ;)

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRZy8cAAoJEBzwKT+lPKRYV8wQAJ9fNdj7ARfzro6y3QHd7w/Y
NcnnynPMQbgJK52RBWFDP4TrqxKMeeF+NpTm14i4yS2x+S6qQvjZp3NgMjW+0h7H
nFUHWgYWumN7tHUcS213/Rqe8lQUk1EAKyH2SzlQbpveY/LOYa1tR92i5X7zW/Et
NxQh8RjxqvaaFgsnXb84V7YpD3dlo6OmjLZGYNvKMD/HAV5mvRJny5AHRUGhO7zU
a3SAohelp0dxPx3Sx0DsRK2Ld2MrhyVXn4zgznAjLqjz8jleci61ctUA9786qR28
wPZDI82ihVNYelDMhIjQanYWGE8cHvZ8p9Na/BN3oKyKN4ViwHjt2Qhm4SGd3hID
ini1pLlMcUEMn8NJaYaKvqz2KdpZHwanbHkIKfxSLhvhWQxGO1eZo2rX2bTz2xFE
nsQWd0mNaSvJ/Y1ED4zuTBiMj2WwzLoszaNX4k/qteYzq/1ITJUZmLrGdcHuT7ik
ZZgXRVDoH/uQoIPbylXjzPqIGb2ChEBCljIlnCEsowLNhr/2q59Dquki+wseD60/
Hou5swvu6ineAZ4MgtJ+fjdjExWWjfJhrF4Oa2lJViQTpbijOiTrpvNwtMe3D6zd
OlepBETdwglxLPfk7UGzWM/yDYh9oXWSUaMd5RhgPzuVP7tdhU/J3Z/IywLreCct
/u+Vsig1JLDzn+SJr0kp
=kW+i
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org