PermGen space usage

[Jonathan Gallimore <jo...@gmail.com>]

Hi all,

I just thought I'd post to see whether any one had any views about our
PermGen space usage. I've noticed that when running TomEE from a recent
snapshot, when nothing but the basic openejb webapp is running, our PermGen
usage sits at around 37MB. If I deploy our moviefun example (which now
features JPA 2, JSF 2, EJBs, Webservices, JMS etc etc) that increased to
140MB - well over the 64MB that the Sun/Oracle JVMs set as a default
maximum. The class counts are 5618 and 23496 respectively. I've profiled
this with the YourKit Java Profiler (thanks for the free license if you're
reading!) and I can't see anything out of the ordinary - it looks like its
all down to the number of classes being loaded. I can't spot any obvious
duplication or class loader leakage. Heap wise, everything looks ok - usage
on this instance was around the 25MB mark.

100 MiB = 102400 KiB, which divided by 17878 extra classes works out at 5.7
KiB per class on average, which sounds reasonable to me, I guess. I've
uploaded a list of all the classes that are loaded between TomEE starting
and moviefun being fully deployed here:
http://people.apache.org/~jgallimore/loaded_classes.txt. One thing that is
interesting is the system class loader had 19357 classes loaded, and 17973
classes without instances. Prior to deploying moviefun, the numbers are 3406
classes, and 2181 without instances - anyone know if there's anything
significant in those numbers?

Does anyone have any thoughts on this? I don't necessarily have any problem
with increasing PermGen on my own installations or recommending others do
the same, but also feel it would be nice to offer as much as possible
without requiring users to allocate more memory. I'd be interested in
hearing anyone else's experiences with this.

Cheers

Jon

Re: PermGen space usage

[Jonathan Gallimore <jo...@gmail.com>]

That seems to make a big difference - PermGen is now just under the max of
64MB, and 12158 classes loaded with moviefun deployed. Startup time has
dropped from 40s down to 13s.

I've attached a diff - can anyone see any issues with applying this? Do you
think we need a switch to enable/disable this?

Cheers

Jon

On Thu, Mar 31, 2011 at 9:17 PM, Jonathan Gallimore <
jonathan.gallimore@gmail.com> wrote:

> I debugged into this a bit further. It looks like if you have a web.xml
> with a version >= 3.0 Tomcat scans the jars in the webapp for web.xml
> fragments, and by default also scans the classpath, causing it to pick up
> OpenEJB and all its associated jars. I've made a change to intercept the
> configure_start lifecycle event, and turn classpath scanning off, so it just
> scans the jars in the webapp to see if that makes any difference. It's
> building at the moment, so I'll give it a test and report back how it goes.
>
> Jon
>
>
> On Wed, Mar 30, 2011 at 11:57 PM, Jonathan Gallimore <
> jonathan.gallimore@gmail.com> wrote:
>
>> On Wed, Mar 30, 2011 at 9:34 PM, David Blevins <da...@gmail.com>wrote:
>>
>>>
>>> On Mar 30, 2011, at 6:14 AM, Jonathan Gallimore wrote:
>>>
>>> > Hi all,
>>> >
>>> > I just thought I'd post to see whether any one had any views about our
>>> > PermGen space usage. I've noticed that when running TomEE from a recent
>>> > snapshot, when nothing but the basic openejb webapp is running, our
>>> PermGen
>>> > usage sits at around 37MB. If I deploy our moviefun example (which now
>>> > features JPA 2, JSF 2, EJBs, Webservices, JMS etc etc) that increased
>>> to
>>> > 140MB - well over the 64MB that the Sun/Oracle JVMs set as a default
>>> > maximum. The class counts are 5618 and 23496 respectively. I've
>>> profiled
>>> > this with the YourKit Java Profiler (thanks for the free license if
>>> you're
>>> > reading!) and I can't see anything out of the ordinary - it looks like
>>> its
>>> > all down to the number of classes being loaded. I can't spot any
>>> obvious
>>> > duplication or class loader leakage. Heap wise, everything looks ok -
>>> usage
>>> > on this instance was around the 25MB mark.
>>> >
>>> > 100 MiB = 102400 KiB, which divided by 17878 extra classes works out at
>>> 5.7
>>> > KiB per class on average, which sounds reasonable to me, I guess. I've
>>> > uploaded a list of all the classes that are loaded between TomEE
>>> starting
>>> > and moviefun being fully deployed here:
>>> > http://people.apache.org/~jgallimore/loaded_classes.txt. One thing
>>> that is
>>> > interesting is the system class loader had 19357 classes loaded, and
>>> 17973
>>> > classes without instances. Prior to deploying moviefun, the numbers are
>>> 3406
>>> > classes, and 2181 without instances - anyone know if there's anything
>>> > significant in those numbers?
>>> >
>>> > Does anyone have any thoughts on this? I don't necessarily have any
>>> problem
>>> > with increasing PermGen on my own installations or recommending others
>>> do
>>> > the same, but also feel it would be nice to offer as much as possible
>>> > without requiring users to allocate more memory. I'd be interested in
>>> > hearing anyone else's experiences with this.
>>>
>>> Would be cool to get a before/after moviefun list so we could do some
>>> comparisons.
>>
>>
>> That file on people.apache.org is a log using -class:verbose which I
>> started after letting TomEE start up, and then I dropped the moviefun.war in
>> - so it should show the 15000 or so added classes. I've also done a log for
>> all the classes TomEE loads right from the start without deploying any extra
>> apps, and a log from the start for all the classes with moviefun deployed. I
>> can't seem to get these onto people.apache.org at the moment, but they're
>> available from http://home.jrg.me.uk/~jgallimore
>>
>>
>>>  The system class increase as you point out is very odd.  This might be
>>> releated to OWB.  I thought I cut that part out, but OWB will try to load
>>> *every* class it can find.  As I said, I thought I cut that out, but maybe
>>> it's getting invoked.  I'm not sure where that code is anymore, but grep in
>>> OWB for calls to loadClass or Class.forName.  Think it might be in their
>>> Scanner implementation.
>>>
>>
>> I haven't had a look at that yet, but I'll have a quick scan through the
>> OWB code next and report back with any findings. Thanks for the pointer!
>>
>>
>>>
>>> Does look like there might be some low hanging fruit.  Seems we are
>>> falsely including the tomcat libs in our libs and they are getting loaded:
>>>
>>> [Loaded org.apache.catalina.mbeans.ServerLifecycleListener from
>>> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>>> [Loaded org.apache.naming.NamingServiceMBean from
>>> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>>> [Loaded org.apache.naming.NamingService from
>>> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>>>
>>> We should trim the contents of our lib/ dir down and see how things look.
>>>
>>
>> Whoops! We should definitely remove that.
>>
>>
>>>
>>> Thanks for looking at this BTW!
>>>
>>
>> No problem - I've enjoyed it, and I've learned a lot too!
>>
>> Jon
>>
>
>

Re: PermGen space usage

[Jonathan Gallimore <jo...@gmail.com>]

I debugged into this a bit further. It looks like if you have a web.xml with
a version >= 3.0 Tomcat scans the jars in the webapp for web.xml fragments,
and by default also scans the classpath, causing it to pick up OpenEJB and
all its associated jars. I've made a change to intercept the configure_start
lifecycle event, and turn classpath scanning off, so it just scans the jars
in the webapp to see if that makes any difference. It's building at the
moment, so I'll give it a test and report back how it goes.

Jon

On Wed, Mar 30, 2011 at 11:57 PM, Jonathan Gallimore <
jonathan.gallimore@gmail.com> wrote:

> On Wed, Mar 30, 2011 at 9:34 PM, David Blevins <da...@gmail.com>wrote:
>
>>
>> On Mar 30, 2011, at 6:14 AM, Jonathan Gallimore wrote:
>>
>> > Hi all,
>> >
>> > I just thought I'd post to see whether any one had any views about our
>> > PermGen space usage. I've noticed that when running TomEE from a recent
>> > snapshot, when nothing but the basic openejb webapp is running, our
>> PermGen
>> > usage sits at around 37MB. If I deploy our moviefun example (which now
>> > features JPA 2, JSF 2, EJBs, Webservices, JMS etc etc) that increased to
>> > 140MB - well over the 64MB that the Sun/Oracle JVMs set as a default
>> > maximum. The class counts are 5618 and 23496 respectively. I've profiled
>> > this with the YourKit Java Profiler (thanks for the free license if
>> you're
>> > reading!) and I can't see anything out of the ordinary - it looks like
>> its
>> > all down to the number of classes being loaded. I can't spot any obvious
>> > duplication or class loader leakage. Heap wise, everything looks ok -
>> usage
>> > on this instance was around the 25MB mark.
>> >
>> > 100 MiB = 102400 KiB, which divided by 17878 extra classes works out at
>> 5.7
>> > KiB per class on average, which sounds reasonable to me, I guess. I've
>> > uploaded a list of all the classes that are loaded between TomEE
>> starting
>> > and moviefun being fully deployed here:
>> > http://people.apache.org/~jgallimore/loaded_classes.txt. One thing that
>> is
>> > interesting is the system class loader had 19357 classes loaded, and
>> 17973
>> > classes without instances. Prior to deploying moviefun, the numbers are
>> 3406
>> > classes, and 2181 without instances - anyone know if there's anything
>> > significant in those numbers?
>> >
>> > Does anyone have any thoughts on this? I don't necessarily have any
>> problem
>> > with increasing PermGen on my own installations or recommending others
>> do
>> > the same, but also feel it would be nice to offer as much as possible
>> > without requiring users to allocate more memory. I'd be interested in
>> > hearing anyone else's experiences with this.
>>
>> Would be cool to get a before/after moviefun list so we could do some
>> comparisons.
>
>
> That file on people.apache.org is a log using -class:verbose which I
> started after letting TomEE start up, and then I dropped the moviefun.war in
> - so it should show the 15000 or so added classes. I've also done a log for
> all the classes TomEE loads right from the start without deploying any extra
> apps, and a log from the start for all the classes with moviefun deployed. I
> can't seem to get these onto people.apache.org at the moment, but they're
> available from http://home.jrg.me.uk/~jgallimore
>
>
>>  The system class increase as you point out is very odd.  This might be
>> releated to OWB.  I thought I cut that part out, but OWB will try to load
>> *every* class it can find.  As I said, I thought I cut that out, but maybe
>> it's getting invoked.  I'm not sure where that code is anymore, but grep in
>> OWB for calls to loadClass or Class.forName.  Think it might be in their
>> Scanner implementation.
>>
>
> I haven't had a look at that yet, but I'll have a quick scan through the
> OWB code next and report back with any findings. Thanks for the pointer!
>
>
>>
>> Does look like there might be some low hanging fruit.  Seems we are
>> falsely including the tomcat libs in our libs and they are getting loaded:
>>
>> [Loaded org.apache.catalina.mbeans.ServerLifecycleListener from
>> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>> [Loaded org.apache.naming.NamingServiceMBean from
>> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>> [Loaded org.apache.naming.NamingService from
>> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>>
>> We should trim the contents of our lib/ dir down and see how things look.
>>
>
> Whoops! We should definitely remove that.
>
>
>>
>> Thanks for looking at this BTW!
>>
>
> No problem - I've enjoyed it, and I've learned a lot too!
>
> Jon
>

Re: PermGen space usage

[Mark Struberg <st...@yahoo.de>]

In OpenWebBeans we face a similar problem. Over there it's caused by the fact that we need to do lots of classpath scanning and basically every class.getDeclaredMethod() class.getDeclaredField() etc sucks in lots of trash to the classloader.

Grab a yourkit test license and check the classloader usage.

As already discussed with David: We'd like to extend xbean finder so that it parses fields + annotations, methods + annotations, class-annotations, etc without flooding the ClassLoader. 

That could later also been done via a central approach where projects (openejb, openwebbeans, myfaces, ...) 'register' themself on startup and then only one class scanning is been performed for the registered criterias [1]

LieGrue,
strub

[1] https://github.com/struberg/Apache-commons-classscanner


--- On Wed, 3/30/11, Jonathan Gallimore <jo...@gmail.com> wrote:

> From: Jonathan Gallimore <jo...@gmail.com>
> Subject: Re: PermGen space usage
> To: dev@openejb.apache.org
> Date: Wednesday, March 30, 2011, 10:57 PM
> On Wed, Mar 30, 2011 at 9:34 PM,
> David Blevins <da...@gmail.com>wrote:
> 
> >
> > On Mar 30, 2011, at 6:14 AM, Jonathan Gallimore
> wrote:
> >
> > > Hi all,
> > >
> > > I just thought I'd post to see whether any one
> had any views about our
> > > PermGen space usage. I've noticed that when
> running TomEE from a recent
> > > snapshot, when nothing but the basic openejb
> webapp is running, our
> > PermGen
> > > usage sits at around 37MB. If I deploy our
> moviefun example (which now
> > > features JPA 2, JSF 2, EJBs, Webservices, JMS etc
> etc) that increased to
> > > 140MB - well over the 64MB that the Sun/Oracle
> JVMs set as a default
> > > maximum. The class counts are 5618 and 23496
> respectively. I've profiled
> > > this with the YourKit Java Profiler (thanks for
> the free license if
> > you're
> > > reading!) and I can't see anything out of the
> ordinary - it looks like
> > its
> > > all down to the number of classes being loaded. I
> can't spot any obvious
> > > duplication or class loader leakage. Heap wise,
> everything looks ok -
> > usage
> > > on this instance was around the 25MB mark.
> > >
> > > 100 MiB = 102400 KiB, which divided by 17878
> extra classes works out at
> > 5.7
> > > KiB per class on average, which sounds reasonable
> to me, I guess. I've
> > > uploaded a list of all the classes that are
> loaded between TomEE starting
> > > and moviefun being fully deployed here:
> > > http://people.apache.org/~jgallimore/loaded_classes.txt.
> One thing that
> > is
> > > interesting is the system class loader had 19357
> classes loaded, and
> > 17973
> > > classes without instances. Prior to deploying
> moviefun, the numbers are
> > 3406
> > > classes, and 2181 without instances - anyone know
> if there's anything
> > > significant in those numbers?
> > >
> > > Does anyone have any thoughts on this? I don't
> necessarily have any
> > problem
> > > with increasing PermGen on my own installations
> or recommending others do
> > > the same, but also feel it would be nice to offer
> as much as possible
> > > without requiring users to allocate more memory.
> I'd be interested in
> > > hearing anyone else's experiences with this.
> >
> > Would be cool to get a before/after moviefun list so
> we could do some
> > comparisons.
> 
> 
> That file on people.apache.org is a log using
> -class:verbose which I started
> after letting TomEE start up, and then I dropped the
> moviefun.war in - so it
> should show the 15000 or so added classes. I've also done a
> log for all the
> classes TomEE loads right from the start without deploying
> any extra apps,
> and a log from the start for all the classes with moviefun
> deployed. I can't
> seem to get these onto people.apache.org at the moment, but
> they're
> available from http://home.jrg.me.uk/~jgallimore
> 
> 
> >  The system class increase as you point out is
> very odd.  This might be
> > releated to OWB.  I thought I cut that part out,
> but OWB will try to load
> > *every* class it can find.  As I said, I thought
> I cut that out, but maybe
> > it's getting invoked.  I'm not sure where that
> code is anymore, but grep in
> > OWB for calls to loadClass or Class.forName. 
> Think it might be in their
> > Scanner implementation.
> >
> 
> I haven't had a look at that yet, but I'll have a quick
> scan through the OWB
> code next and report back with any findings. Thanks for the
> pointer!
> 
> 
> >
> > Does look like there might be some low hanging
> fruit.  Seems we are falsely
> > including the tomcat libs in our libs and they are
> getting loaded:
> >
> > [Loaded
> org.apache.catalina.mbeans.ServerLifecycleListener from
> >
> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
> > [Loaded org.apache.naming.NamingServiceMBean from
> >
> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
> > [Loaded org.apache.naming.NamingService from
> >
> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
> >
> > We should trim the contents of our lib/ dir down and
> see how things look.
> >
> 
> Whoops! We should definitely remove that.
> 
> 
> >
> > Thanks for looking at this BTW!
> >
> 
> No problem - I've enjoyed it, and I've learned a lot too!
> 
> Jon
> 

Re: PermGen space usage

[Jonathan Gallimore <jo...@gmail.com>]

On Wed, Mar 30, 2011 at 9:34 PM, David Blevins <da...@gmail.com>wrote:

>
> On Mar 30, 2011, at 6:14 AM, Jonathan Gallimore wrote:
>
> > Hi all,
> >
> > I just thought I'd post to see whether any one had any views about our
> > PermGen space usage. I've noticed that when running TomEE from a recent
> > snapshot, when nothing but the basic openejb webapp is running, our
> PermGen
> > usage sits at around 37MB. If I deploy our moviefun example (which now
> > features JPA 2, JSF 2, EJBs, Webservices, JMS etc etc) that increased to
> > 140MB - well over the 64MB that the Sun/Oracle JVMs set as a default
> > maximum. The class counts are 5618 and 23496 respectively. I've profiled
> > this with the YourKit Java Profiler (thanks for the free license if
> you're
> > reading!) and I can't see anything out of the ordinary - it looks like
> its
> > all down to the number of classes being loaded. I can't spot any obvious
> > duplication or class loader leakage. Heap wise, everything looks ok -
> usage
> > on this instance was around the 25MB mark.
> >
> > 100 MiB = 102400 KiB, which divided by 17878 extra classes works out at
> 5.7
> > KiB per class on average, which sounds reasonable to me, I guess. I've
> > uploaded a list of all the classes that are loaded between TomEE starting
> > and moviefun being fully deployed here:
> > http://people.apache.org/~jgallimore/loaded_classes.txt. One thing that
> is
> > interesting is the system class loader had 19357 classes loaded, and
> 17973
> > classes without instances. Prior to deploying moviefun, the numbers are
> 3406
> > classes, and 2181 without instances - anyone know if there's anything
> > significant in those numbers?
> >
> > Does anyone have any thoughts on this? I don't necessarily have any
> problem
> > with increasing PermGen on my own installations or recommending others do
> > the same, but also feel it would be nice to offer as much as possible
> > without requiring users to allocate more memory. I'd be interested in
> > hearing anyone else's experiences with this.
>
> Would be cool to get a before/after moviefun list so we could do some
> comparisons.


That file on people.apache.org is a log using -class:verbose which I started
after letting TomEE start up, and then I dropped the moviefun.war in - so it
should show the 15000 or so added classes. I've also done a log for all the
classes TomEE loads right from the start without deploying any extra apps,
and a log from the start for all the classes with moviefun deployed. I can't
seem to get these onto people.apache.org at the moment, but they're
available from http://home.jrg.me.uk/~jgallimore


>  The system class increase as you point out is very odd.  This might be
> releated to OWB.  I thought I cut that part out, but OWB will try to load
> *every* class it can find.  As I said, I thought I cut that out, but maybe
> it's getting invoked.  I'm not sure where that code is anymore, but grep in
> OWB for calls to loadClass or Class.forName.  Think it might be in their
> Scanner implementation.
>

I haven't had a look at that yet, but I'll have a quick scan through the OWB
code next and report back with any findings. Thanks for the pointer!


>
> Does look like there might be some low hanging fruit.  Seems we are falsely
> including the tomcat libs in our libs and they are getting loaded:
>
> [Loaded org.apache.catalina.mbeans.ServerLifecycleListener from
> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
> [Loaded org.apache.naming.NamingServiceMBean from
> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
> [Loaded org.apache.naming.NamingService from
> file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
>
> We should trim the contents of our lib/ dir down and see how things look.
>

Whoops! We should definitely remove that.


>
> Thanks for looking at this BTW!
>

No problem - I've enjoyed it, and I've learned a lot too!

Jon

Re: PermGen space usage

[David Blevins <da...@gmail.com>]

On Mar 30, 2011, at 6:14 AM, Jonathan Gallimore wrote:

> Hi all,
> 
> I just thought I'd post to see whether any one had any views about our
> PermGen space usage. I've noticed that when running TomEE from a recent
> snapshot, when nothing but the basic openejb webapp is running, our PermGen
> usage sits at around 37MB. If I deploy our moviefun example (which now
> features JPA 2, JSF 2, EJBs, Webservices, JMS etc etc) that increased to
> 140MB - well over the 64MB that the Sun/Oracle JVMs set as a default
> maximum. The class counts are 5618 and 23496 respectively. I've profiled
> this with the YourKit Java Profiler (thanks for the free license if you're
> reading!) and I can't see anything out of the ordinary - it looks like its
> all down to the number of classes being loaded. I can't spot any obvious
> duplication or class loader leakage. Heap wise, everything looks ok - usage
> on this instance was around the 25MB mark.
> 
> 100 MiB = 102400 KiB, which divided by 17878 extra classes works out at 5.7
> KiB per class on average, which sounds reasonable to me, I guess. I've
> uploaded a list of all the classes that are loaded between TomEE starting
> and moviefun being fully deployed here:
> http://people.apache.org/~jgallimore/loaded_classes.txt. One thing that is
> interesting is the system class loader had 19357 classes loaded, and 17973
> classes without instances. Prior to deploying moviefun, the numbers are 3406
> classes, and 2181 without instances - anyone know if there's anything
> significant in those numbers?
> 
> Does anyone have any thoughts on this? I don't necessarily have any problem
> with increasing PermGen on my own installations or recommending others do
> the same, but also feel it would be nice to offer as much as possible
> without requiring users to allocate more memory. I'd be interested in
> hearing anyone else's experiences with this.

Would be cool to get a before/after moviefun list so we could do some comparisons.  The system class increase as you point out is very odd.  This might be releated to OWB.  I thought I cut that part out, but OWB will try to load *every* class it can find.  As I said, I thought I cut that out, but maybe it's getting invoked.  I'm not sure where that code is anymore, but grep in OWB for calls to loadClass or Class.forName.  Think it might be in their Scanner implementation.

Does look like there might be some low hanging fruit.  Seems we are falsely including the tomcat libs in our libs and they are getting loaded:

[Loaded org.apache.catalina.mbeans.ServerLifecycleListener from file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
[Loaded org.apache.naming.NamingServiceMBean from file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]
[Loaded org.apache.naming.NamingService from file:/Users/jon/demo/apache-tomee-tomcat-7.0.11/webapps/openejb/lib/tomcat-catalina-7.0.8.jar]

We should trim the contents of our lib/ dir down and see how things look.

Thanks for looking at this BTW!


-David