You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kiran Badi <ki...@poonam.org> on 2015/07/05 16:24:44 UTC

Re: Tomcat - OOM Perm gen

Sorry for taking time to reply Chris.

Are you periodically hot-re-deploying your application in production?
If so, you may want to stop doing that, as it appears that you have a
ClassLoader-pinning leak in your application or some dependent library

I do not do hot deployment in production.But somehow I feel its leaking
memory.There are some threads which for some reasons remains hanging.
But I think I will focus on it later on. I do have copy of your
presentation.I will go through it and may be post couple of question if I
have.

You should be able to run a reasonable service in half a gig of heap
space. Just be aware that caching information in your application is
going to significantly increase the amount of heap space required by
your application in a steady-state.

Yup I am aware of this limitation. I know cache is often stored in local
heap and probably that's reason it consumes heap space.
Currently I am checking if ehcache or jcs meets my need. Ehcache for some
reasons fills my heap very fast.I tried storing some 100 json strings and
retrieving them.
My OOM's started once I integrated this library in my web inf /lib
folder.After I removed it, frequency of OOM has substantially
decreased.Probably I am not using it correctly somewhere.

Do you know what open source caching people frequently use for java web
apps ?

Fetching a lot of data isn't usually a big deal. Just make sure that
your "fetch size" is set to something reasonable. There are some JDBC
drivers (MySQL Connector/J in particular) that will load the entire
ResultSet into memory before allowing you to traverse it unless you
make arrangements to limit that memory usage.

But bringing-back thousands of records from a db isn't a problem --
unless you don't *have to* in which case you might want to optimize
your queries to avoid pulling data you don't actually need.

But too-many-records would be a performance problem, not a memory proble
m.

haha.Its fetch size which my autocomplete was missing...I am not sure how I
missed it but it was missing.
Thanks Mark.



On Tue, Jun 30, 2015 at 1:17 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Kiran,
>
> On 6/29/15 10:21 AM, Kiran Badi wrote:
> > Christopher Schultz wrote:
> >> The number of users shouldn't impact your PermGen space. Perhaps
> >> only once you get to that stage are you hitting enough of your
> >> features to load classes into PermGen. (Or maybe you are using
> >> String.intern a lot...)
> >
> > I analysed some logs and I could see that users query features
> > which makes DB calls, so those calls do have 1000's of rows in
> > it.But some calls also fetch empty result set and some error out,
> > partly because code for those calls are broken( Some of those dao
> > classes have hard coded DB parameter which I am cleaning it out
> > now). As far as I know I do not do any string cancat, those calls
> > are all simple list fetch calls to views.
> >
> > I am trying to implement some caching using either ehcache or JCS
> > but I think it has to wait for some time, till I gain some
> > understanding on how these works.( I think I need to serialize lot
> > of model classes for that probably will require some code changes
> > again).i  know I have lot of work to do ,maybe I one at a time
> > change :)
> >
> >> PermGen failures will effect the whole JVM. There is no way to
> >> protect App B from App A unless they are in different JVMs.
> >
> > I can understand this. so doing daily restart now to manage  issue
> > till I figure out some solution to it.
> >
> >> What makes you say that? It seems that you have more information
> >> than you are giving us.
> >
> > Its not hardened code so I think it still has some issues with it.
> > Also during development I can see similar errors on local dev box,
> > If I do deploy and redeploy at least 8 to 10 times, I start seeing
> > those  perm gen errors,its just that it references a new class
> > file every time,maybe I can share it with you all once I get it
> > again.
>
> Stop right there.
>
> Are you periodically hot-re-deploying your application in production?
> If so, you may want to stop doing that, as it appears that you have a
> ClassLoader-pinning leak in your application or some dependent library.
>
> See this presentation for more information:
> http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60
> mins.pdf
>
> > Also it's I have written this code and I am not that fantastic
> > coder yet, but I will reach there short span :)
> >
> >
> >> Usually, PermGen doesn't have to be enormous. What's your memory
> >> cap with your hosting provider?
> >
> > I have private tomcat 7x but I remember hosting provider
> > mentioning that 512mb is final,but I will check with them again
> > later this week.
>
> You should be able to run a reasonable service in half a gig of heap
> space. Just be aware that caching information in your application is
> going to significantly increase the amount of heap space required by
> your application in a steady-state.
>
> > Below is what I see in catalina logs when I do restart of tomcat,
> >
> > Picked up _JAVA_OPTIONS: -Xms20m  -Xmx128m -XX:MinHeapFreeRatio=20
> >  -XX:MaxHeapFreeRatio=40 -XX:NewSize=10m  -XX:MaxNewSize=10m
> > -XX:SurvivorRatio=6 -XX:TargetSurvivorRatio=80
> > -XX:+CMSClassUnloadingEnabled -XX:+CMSClassUnloadingEnabled
>
> So you have a 128MiB max heap space, but you aren't specifying a
> PermGen size. For some frameworks (particularly Spring), you'll need
> to increase the size of the PermGen heap space because it really just
> does need to load 3e8 classes or whatever.
>
> Or, you could upgrade to Java 8 which does not have a PermGen space.
> You'll just eventually run out of "regular" heap space in that case,
> if you really do have a memory leak.
>
> > I think was thinking CMSClassUnloadingEnabled should fix my perm
> > gen issues, but I think its not the case.
>
> Class unloading should pretty much always be enabled, so that setting
> likely does nothing for you. It doesn't look like you are enabling the
> CMS GC, so those settings probably don't make a bit of difference.
>
> >> You either need more PermGen space, or you need to locate some
> >> kind of leak in your application and fix it. IIRC, there are
> >> some RMI-related leaks and Proxy-related leaks in PermGen
> >> depending upon your exact circumstances.
> >>
> >> It would be good to know what's in PermGen when it hits its
> >> limit.
> >>
> >> What are your current heap settings, including PermGen? What
> >> JVM?
> >>
> >> Try: $ jmap -heap <pid>
> >>
> >> and
> >>
> >> $ jmap -permstat <pid>
> >
> > I will try to get those dumps but I do not use any RMI or generate
> > some kind of proxies . Mine is simple app with lot of forms in it.
> > Though there are few calls which fetches lot of data from
> > servers.Sometimes few autocomplete calls fetch 1000's of records.I
> > am trying to remove those calls.
>
> Fetching a lot of data isn't usually a big deal. Just make sure that
> your "fetch size" is set to something reasonable. There are some JDBC
> drivers (MySQL Connector/J in particular) that will load the entire
> ResultSet into memory before allowing you to traverse it unless you
> make arrangements to limit that memory usage.
>
> But bringing-back thousands of records from a db isn't a problem --
> unless you don't *have to* in which case you might want to optimize
> your queries to avoid pulling data you don't actually need.
>
> But too-many-records would be a performance problem, not a memory proble
> m.
>
> > Below is my jvm details
> >
> > Apache Tomcat/7.0.50 1.7.0_17-b02 Oracle Corporation Linux
> > 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64 amd64
> >
> > Thanks Chris for reply.
>
> Hope that helps,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVks88AAoJEBzwKT+lPKRYOh8P/2DwjRn0hH/R3Xsa0Sf+FFKh
> 5/dzcnRf7C1PvCVQ8IrBTIsKuMCBY6dbS9PxJe7+hWo/UIaScF62B6S4xwd4WCsa
> 5ojVDKslt/BcNGk/6bXyK2uDFXJRFB3j3654XgsU9/T+GvNOF9F6rNQYnRDZVuuv
> UMbphYOEHfFBplIEJnu5Q6SiLD219aKLa3Xmbl1+WHn1tQyoPivFmdBBkAkUO+92
> WN6vwRKiI0VgKZ9mbH2pgI1BVq3qDxKksPx8nv1ju3nDb5MRCWgbNGeJFhl87B5b
> 78XmiNdmYQA4K1PSk4tGXergTDpgJ4PtYhBSYvZC8xqFQnEaiUWtqFwKCqAyFL+S
> FX1Tef/Rc/20XYnxqUm9Wi5XX2Nq2KCBxs/ob7DdwC52TyB9LQDNP7eUvYPZQv13
> cuEW+eDozfYaao9Io+rparQ3Wz95y/1bwgEP5BjxK0b5rqDuDn84HsC6FrSLBPJc
> qp1q6vs4JYB3jIwdNKN+JbGoPTPYfwcI2ESuviVbAAKO+nYQKj0lIgERQBp2UDCN
> RSVSlXW64W//LFuDZzfN+v099vj0BOh1EEWoM+lXDeT9PSpam+gEcaze/gTK5L06
> 0x/0gXgiQGNNgDFmP6ovTpcBA59IA6o8cfYTEbovnJnBr/Bx10pET9D8/QR515od
> XJyzmMIXwXltaOzdAiEe
> =p9EI
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Tomcat - OOM Perm gen

Posted by Kiran Badi <ki...@poonam.org>.
Hi Chris,

Here is link for Dump files,

https://drive.google.com/folderview?id=0BxOQjvXCnkSifjZEcEYzUTNTbUFoUWdrdmMyT18wdkZDS0hEOEgwRnl6RTBWN0V6UlA1YU0&usp=sharing

I tried using eclipse MAT Analyser and I see most of the threads on related
to tomcat web context loader. I still need to spend some time on that.

I think my apps still needs some cleanup.

After a day or 2 , it often dies a slow death with message,

Exception in thread "ajp-bio-17703-exec-11"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ajp-bio-17703-exec-11"

I am going to upload a new war today and see if it resolves the issue.

I will need some help with from this group in managing bots which are
spanning 100's of sessions in my case.I will initiate a new thread on that.

Sorry for delayed replies from my end.

- Kiran

On Mon, Jul 6, 2015 at 4:38 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Kiran,
>
> On 7/5/15 10:24 AM, Kiran Badi wrote:
> > Sorry for taking time to reply Chris.
> >
> >> Christopher Schultz wrote: Are you periodically hot-re-deploying
> >> your application in production? If so, you may want to stop doing
> >> that, as it appears that you have a ClassLoader-pinning leak in
> >> your application or some dependent library
> >
> > I do not do hot deployment in production. But somehow I feel its
> > leaking memory.
>
> What makes you think that?
>
> > There are some threads which for some reasons remains hanging.
>
> Threads related to Tomcat, or threads that are started by your
> application?
>
> > But I think I will focus on it later on. I do have copy of your
> > presentation. I will go through it and may be post couple of
> > question if I have.
>
> The presentation is markt's, not mine.
>
> >> You should be able to run a reasonable service in half a gig of
> >> heap space. Just be aware that caching information in your
> >> application is going to significantly increase the amount of heap
> >> space required by your application in a steady-state.
> >
> > Yup I am aware of this limitation. I know cache is often stored in
> > local heap and probably that's reason it consumes heap space.
>
> Well... there are many ways to cache data, but if you are caching it
> in your application (and not in an out-of-process-cache like ehcache,
> memcached, etc.) then by definition it's in your application's heap spac
> e.
>
> > Currently I am checking if ehcache or jcs meets my need. Ehcache
> > for some reasons fills my heap very fast.I tried storing some 100
> > json strings and retrieving them. My OOM's started once I
> > integrated this library in my web inf /lib folder.After I removed
> > it, frequency of OOM has substantially decreased.Probably I am not
> > using it correctly somewhere.
>
> You may also be using in-memory caching as opposed to on-disk caching.
>
> > Do you know what open source caching people frequently use for java
> > web apps ?
>
> Honestly, I would spend your time by disabling any caching and making
> sure that the application doesn't leak memory *before* you add any
> caching before you start introducing any kind of caching.
>
> >> Fetching a lot of data isn't usually a big deal. Just make sure
> >> that your "fetch size" is set to something reasonable. There are
> >> some JDBC drivers (MySQL Connector/J in particular) that will
> >> load the entire ResultSet into memory before allowing you to
> >> traverse it unless you make arrangements to limit that memory
> >> usage.
> >>
> >> But bringing-back thousands of records from a db isn't a problem
> >> -- unless you don't *have to* in which case you might want to
> >> optimize your queries to avoid pulling data you don't actually
> >> need.
> >>
> >> But too-many-records would be a performance problem, not a
> >> memory problem.
> >
> > haha.Its fetch size which my autocomplete was missing...I am not
> > sure how I missed it but it was missing.
>
> I would remove all the caching and make sure that the application
> works properly without leaking any memory before moving on to improve
> performance.
>
> To do otherwise would be considered premature optimization. Nobody
> likes an application that falls-over, even if it does so with
> high-performance. Users will tolerate a slow application if it's the
> only thing wrong with it.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVmuc6AAoJEBzwKT+lPKRYBHEP/2qtLy1IuPM5JKsMregtLaBX
> L9Hhg5YbazVzO2V7sajZJuCtYkDSOLkz6rI3Q7YgV+o3Pgl9YvPAyTmH2YGp59Ho
> CzwP7rhkN0tx6QFBU3rpxj2Xcr6FZMzZrINg6+vi8/FBFbN93kjW8tBd5OjqZSi8
> hd5P4pvkKEWM2UHkCtdqWDLACi2oldLYNHeaVcVNUCcbYJodCek9OEIHK5OeRV27
> VrhTngQvcRu7LivbjPp2yl4w4APDuctSX3YkVyHmCbNFdE2WzogKbWOfaG2ylIbx
> F/wvBIgLocVIaOL2G2TJkvRYS7oxK7Fsh4JtCdMPZs0Wpkznz+BkBQy4aCy2oWQ2
> FBB9dPxrs0o+8RXKPP7wre8MutcZ2bOgieFdSq9JRe0NrEmhvaVgYKQNT+lVjMmC
> 8+m1W8/Z1z3Rhkc0lnH0U+S2KoPN/FToEIMKNZgrkg/EoAotBYxp8kV06d5+DvP1
> cXqp/Q5HDjZTt9+elaMOvmYzFIZR9TLu8U7uH3tnaKtqtPGzqP7TOJd9mwzqescW
> Be71awY/r+WQiWga6yLHPKD8hrTKjmZjaewm7WOwxNrVcjdi/IN8YUR6j6/Gq8dh
> 4tYj4UoJLNq5yTA3jywfHRjb+L20B8Mr7aT2vMaGWFo1nCwNAotYulfEjJ5UVVit
> k+fS5XpaNzXtt/yM+a8b
> =ftJv
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Tomcat - OOM Perm gen

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Kiran,

On 7/5/15 10:24 AM, Kiran Badi wrote:
> Sorry for taking time to reply Chris.
> 
>> Christopher Schultz wrote: Are you periodically hot-re-deploying
>> your application in production? If so, you may want to stop doing
>> that, as it appears that you have a ClassLoader-pinning leak in
>> your application or some dependent library
> 
> I do not do hot deployment in production. But somehow I feel its
> leaking memory.

What makes you think that?

> There are some threads which for some reasons remains hanging.

Threads related to Tomcat, or threads that are started by your
application?

> But I think I will focus on it later on. I do have copy of your 
> presentation. I will go through it and may be post couple of
> question if I have.

The presentation is markt's, not mine.

>> You should be able to run a reasonable service in half a gig of
>> heap space. Just be aware that caching information in your
>> application is going to significantly increase the amount of heap
>> space required by your application in a steady-state.
> 
> Yup I am aware of this limitation. I know cache is often stored in
> local heap and probably that's reason it consumes heap space.

Well... there are many ways to cache data, but if you are caching it
in your application (and not in an out-of-process-cache like ehcache,
memcached, etc.) then by definition it's in your application's heap spac
e.

> Currently I am checking if ehcache or jcs meets my need. Ehcache
> for some reasons fills my heap very fast.I tried storing some 100
> json strings and retrieving them. My OOM's started once I
> integrated this library in my web inf /lib folder.After I removed
> it, frequency of OOM has substantially decreased.Probably I am not
> using it correctly somewhere.

You may also be using in-memory caching as opposed to on-disk caching.

> Do you know what open source caching people frequently use for java
> web apps ?

Honestly, I would spend your time by disabling any caching and making
sure that the application doesn't leak memory *before* you add any
caching before you start introducing any kind of caching.

>> Fetching a lot of data isn't usually a big deal. Just make sure
>> that your "fetch size" is set to something reasonable. There are
>> some JDBC drivers (MySQL Connector/J in particular) that will
>> load the entire ResultSet into memory before allowing you to
>> traverse it unless you make arrangements to limit that memory
>> usage.
>> 
>> But bringing-back thousands of records from a db isn't a problem
>> -- unless you don't *have to* in which case you might want to
>> optimize your queries to avoid pulling data you don't actually
>> need.
>> 
>> But too-many-records would be a performance problem, not a
>> memory problem.
> 
> haha.Its fetch size which my autocomplete was missing...I am not
> sure how I missed it but it was missing.

I would remove all the caching and make sure that the application
works properly without leaking any memory before moving on to improve
performance.

To do otherwise would be considered premature optimization. Nobody
likes an application that falls-over, even if it does so with
high-performance. Users will tolerate a slow application if it's the
only thing wrong with it.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVmuc6AAoJEBzwKT+lPKRYBHEP/2qtLy1IuPM5JKsMregtLaBX
L9Hhg5YbazVzO2V7sajZJuCtYkDSOLkz6rI3Q7YgV+o3Pgl9YvPAyTmH2YGp59Ho
CzwP7rhkN0tx6QFBU3rpxj2Xcr6FZMzZrINg6+vi8/FBFbN93kjW8tBd5OjqZSi8
hd5P4pvkKEWM2UHkCtdqWDLACi2oldLYNHeaVcVNUCcbYJodCek9OEIHK5OeRV27
VrhTngQvcRu7LivbjPp2yl4w4APDuctSX3YkVyHmCbNFdE2WzogKbWOfaG2ylIbx
F/wvBIgLocVIaOL2G2TJkvRYS7oxK7Fsh4JtCdMPZs0Wpkznz+BkBQy4aCy2oWQ2
FBB9dPxrs0o+8RXKPP7wre8MutcZ2bOgieFdSq9JRe0NrEmhvaVgYKQNT+lVjMmC
8+m1W8/Z1z3Rhkc0lnH0U+S2KoPN/FToEIMKNZgrkg/EoAotBYxp8kV06d5+DvP1
cXqp/Q5HDjZTt9+elaMOvmYzFIZR9TLu8U7uH3tnaKtqtPGzqP7TOJd9mwzqescW
Be71awY/r+WQiWga6yLHPKD8hrTKjmZjaewm7WOwxNrVcjdi/IN8YUR6j6/Gq8dh
4tYj4UoJLNq5yTA3jywfHRjb+L20B8Mr7aT2vMaGWFo1nCwNAotYulfEjJ5UVVit
k+fS5XpaNzXtt/yM+a8b
=ftJv
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org