Frequent tomcat outages

[Akbar Thanakalacheruvu <Ak...@sumtotalsystems.com>]

Hi

We are using Tomcat 7.0.62 (with JVM version 1.7.0_79-b15) in our production servers. This web server has 16 GB RAM and out of which we allocated 12 GB RAM (xms and xmx) to tomcat (through Tomcat7w.exe).
When I monitor tomcat manager application, I see that "CMS old Gen" is increasing minute to minute and it is reaching to 99% (11 GB) in less than 12 hrs.

I am not sure why this memory is not clearing out. Are there any settings that you can suggest which can help this "CMS old Gen" to be under control?
I see lot many "OutOfMemory" errors in Tomcat logs which were caused by our Java application(which is a 3rd party application). I tried to take heapdump using the below command, but it failed saying "insufficient memory" error.

Command: jmap -dump:format=b,file=snapshot.jmap process-pid

Not sure how to proceed further to get out of these "OutOfMemory" errors. Please help.

-Akbar

Re: Frequent tomcat outages

[Christopher Schultz <ch...@christopherschultz.net>]

Mark,

On 12/9/15 4:51 PM, Mark Thomas wrote:
> On 09/12/2015 21:36, Akbar Thanakalacheruvu wrote:
>> Hi
>>
>> We are using Tomcat 7.0.62 (with JVM version 1.7.0_79-b15) in our production servers. This web server has 16 GB RAM and out of which we allocated 12 GB RAM (xms and xmx) to tomcat (through Tomcat7w.exe).
>> When I monitor tomcat manager application, I see that "CMS old Gen" is increasing minute to minute and it is reaching to 99% (11 GB) in less than 12 hrs.
>>
>> I am not sure why this memory is not clearing out. Are there any settings that you can suggest which can help this "CMS old Gen" to be under control?
>> I see lot many "OutOfMemory" errors in Tomcat logs which were caused by our Java application(which is a 3rd party application). I tried to take heapdump using the below command, but it failed saying "insufficient memory" error.
>>
>> Command: jmap -dump:format=b,file=snapshot.jmap process-pid
>>
>> Not sure how to proceed further to get out of these "OutOfMemory" errors. Please help.
> 
> It depends if a) you really have a memory leak or b) the app just needs
> more memory than you have given it.
> 
> If a) yell at the vendor of application to get their act together and
> fix the memory leaks.
> 
> If b) install some more RAM in the box

It also depends upon the nature of the OOMEs. Lots of applications have
PermGen errors, which is just a matter of adjusting the size of PermGen.

My experience has been that often /reducing/ the heap size ends up being
more informative than just increasing the heap until you stop getting OOMEs.

Akbar, can you post some of the OOMEs you have been getting?

The 99% of old generation used certainly looks like it might be an
actual leak. Obvious places to look include the HttpSession objects (are
you storing lots of data in users sessions?) and application data caches
(are they the right size? are members expiring appropriately?).

A final thought about memory usage. Long ago, a friend of mine
complained about not seeing very much "free memory" available on his
Linux workstation. When we investigated, the OS was using large amounts
of RAM for "caches" which basically means that the OS was using memory
for whatever it wanted -- no specific user process was using that
memory. When he bristled about the OS "wasting" memory, I explained that
unused memory was a waste of money and resources if it wasn't going to
be used. So, back to the Java heap: your heap should be as full as you
can keep it while maintaining a healthy application hosted inside that
JVM. Caches are great, but remember that sometimes you have to let the
users have that memory. But when they're done... refill those caches, baby!

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Frequent tomcat outages

[Mark Thomas <ma...@apache.org>]

On 09/12/2015 21:36, Akbar Thanakalacheruvu wrote:
> Hi
> 
> We are using Tomcat 7.0.62 (with JVM version 1.7.0_79-b15) in our production servers. This web server has 16 GB RAM and out of which we allocated 12 GB RAM (xms and xmx) to tomcat (through Tomcat7w.exe).
> When I monitor tomcat manager application, I see that "CMS old Gen" is increasing minute to minute and it is reaching to 99% (11 GB) in less than 12 hrs.
> 
> I am not sure why this memory is not clearing out. Are there any settings that you can suggest which can help this "CMS old Gen" to be under control?
> I see lot many "OutOfMemory" errors in Tomcat logs which were caused by our Java application(which is a 3rd party application). I tried to take heapdump using the below command, but it failed saying "insufficient memory" error.
> 
> Command: jmap -dump:format=b,file=snapshot.jmap process-pid
> 
> Not sure how to proceed further to get out of these "OutOfMemory" errors. Please help.

It depends if a) you really have a memory leak or b) the app just needs
more memory than you have given it.

If a) yell at the vendor of application to get their act together and
fix the memory leaks.

If b) install some more RAM in the box

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org