You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@karaf.apache.org by Tim Jones <ti...@mccarthy.co.nz> on 2013/05/03 03:52:15 UTC
Multiple restarts - effect on heap and Perm Gen memory
Hi,
I have read a few posts about the subject e.g.
http://karaf.922171.n3.nabble.com/Ability-to-self-restart-karaf-WAS-Infamous-permgen-leak-td3589167.html#a3591080.
Is it correct that stopping and starting a bundle multiple times will
ultimately lead to a PermGen out of memory error (except perhaps when using
a JRockit JVM) and that there is no avoiding this?
I have set up a test (using the org.apache.sshd packages) that repeatedly
stops and starts one of our bundles. About 20 restarts results in a PermGen
out of memory error but also what seems surprising is that the heap memory
usage, in particular the 'PS Old Gen' keeps increasing. Using JVisualM and
comparing the classes before and after 10 restarts and then a garbage
collection (invoked by clicking on 'Perform GC' JVisualM) the application
classes (classes coded by us) seem to be behaving as I would expect as
evidenced by the example below of two classes i.e. the old proxy has been
deallocated (-1) and the new allocated (+1)
Class name - allocated objects
Bytes Objects Allocated
uniworks.livestock.compiler.CompileKill.CompileAdviceLine
0 B 0
uniworks.livestock.compiler.CompileKill.CompileAdviceLine$$EnhancerByCGLIB$$286bdd54
+64 B +1
uniworks.livestock.compiler.CompileKill.CompileAdviceLine$$EnhancerByCGLIB$$4f6d4fc8
-64 B -1
uniworks.livestock.compiler.CompileKill.ProcessCarcases
0 B 0
uniworks.livestock.compiler.CompileKill.ProcessCarcases$$EnhancerByCGLIB$$c049ad79
+56 B +1
uniworks.livestock.compiler.CompileKill.ProcessCarcases$$EnhancerByCGLIB$$e74b1fed
-56 B -1
However many (there are many more than the 4 classes below) of the 3rd party
library classes seem to have increasing objects allocated as evidenced
below.
5 x restarts (+ GC)
Class name - allocated objects
Bytes Objects Allocated
org.springframework.osgi.service.importer.support.internal.aop.ImportedOsgiServiceProxyAdvice
+1,408 B +44
org.springframework.transaction.interceptor.AbstractFallbackTransactionAttributeSource$DefaultCacheKey
+3,559,872 B +148,328
org.apache.activemq.command.ActiveMQQueue
+39,840 B +996
org.apache.activemq.command.SessionId
+55,480 B +1,387
10 x restarts (+ GC)
Class name - allocated objects
Bytes Objects Allocated
org.springframework.osgi.service.importer.support.internal.aop.ImportedOsgiServiceProxyAdvice
+3,168 B +99
org.springframework.transaction.interceptor.AbstractFallbackTransactionAttributeSource$DefaultCacheKey
+8,008,968 B +333,707
org.apache.activemq.command.ActiveMQQueue
+44,280 B +1,107
org.apache.activemq.command.SessionId
+114,160 B +2,854
a) Are the results above what is expected or is there an issue with our
code/3rd party libraries holding on to references
when it shouldn't?
For us the primary driver for choosing OSGI was actually a business driver
and was the promise of 'hot deployment'
(although as a developer I recognise other technical benefits of choosing an
enabler for a modular architecture).
How far can we go with hot deployment?
b) It looks as though there will always be an upper bound of the number of
times a bundle can be stopped and started due to the class meta data being
added into the Perm Gen space?
c) Even if the Perm Gen issue was solved it looks as though there are issues
with objects unable to be GCed, if these are caused by 3rd party libraries
there is probably little we can do to fix this so again there will be an
upper bound to the number of bundle retarts before a server restart is
necessary?
--
View this message in context: http://karaf.922171.n3.nabble.com/Multiple-restarts-effect-on-heap-and-Perm-Gen-memory-tp4028550.html
Sent from the Karaf - User mailing list archive at Nabble.com.
Re: Multiple restarts - effect on heap and Perm Gen memory
Posted by Achim Nierbeck <bc...@googlemail.com>.
Hi,
you are right, the perm space issue is something that depends on the JVM.
So you might have a better work around with a newer JVM like the 7 where
there shouldn't be a perm space anymore though it'll take away parts of the
heap. The removal of the loaded classes doesn't work in that one out of the
box either, afairc. But there are some flags on the JVM to enable the
removal of unloaded classes, but I never worked with it before.
Some more comments inline:
2013/5/3 Tim Jones <ti...@mccarthy.co.nz>
> Hi,
>
> I have read a few posts about the subject e.g.
>
> http://karaf.922171.n3.nabble.com/Ability-to-self-restart-karaf-WAS-Infamous-permgen-leak-td3589167.html#a3591080
> .
> Is it correct that stopping and starting a bundle multiple times will
> ultimately lead to a PermGen out of memory error (except perhaps when using
> a JRockit JVM) and that there is no avoiding this?
>
> I have set up a test (using the org.apache.sshd packages) that repeatedly
> stops and starts one of our bundles. About 20 restarts results in a PermGen
> out of memory error but also what seems surprising is that the heap memory
> usage, in particular the 'PS Old Gen' keeps increasing. Using JVisualM and
> comparing the classes before and after 10 restarts and then a garbage
> collection (invoked by clicking on 'Perform GC' JVisualM) the application
> classes (classes coded by us) seem to be behaving as I would expect as
> evidenced by the example below of two classes i.e. the old proxy has been
> deallocated (-1) and the new allocated (+1)
>
> Class name - allocated objects
> Bytes Objects Allocated
> uniworks.livestock.compiler.CompileKill.CompileAdviceLine
> 0 B 0
>
> uniworks.livestock.compiler.CompileKill.CompileAdviceLine$$EnhancerByCGLIB$$286bdd54
> +64 B +1
>
> uniworks.livestock.compiler.CompileKill.CompileAdviceLine$$EnhancerByCGLIB$$4f6d4fc8
> -64 B -1
>
> uniworks.livestock.compiler.CompileKill.ProcessCarcases
> 0 B 0
>
> uniworks.livestock.compiler.CompileKill.ProcessCarcases$$EnhancerByCGLIB$$c049ad79
> +56 B +1
>
> uniworks.livestock.compiler.CompileKill.ProcessCarcases$$EnhancerByCGLIB$$e74b1fed
> -56 B -1
>
>
> However many (there are many more than the 4 classes below) of the 3rd
> party
> library classes seem to have increasing objects allocated as evidenced
> below.
>
> 5 x restarts (+ GC)
> Class name - allocated objects
> Bytes Objects Allocated
>
> org.springframework.osgi.service.importer.support.internal.aop.ImportedOsgiServiceProxyAdvice
> +1,408 B +44
>
> org.springframework.transaction.interceptor.AbstractFallbackTransactionAttributeSource$DefaultCacheKey
> +3,559,872 B +148,328
> org.apache.activemq.command.ActiveMQQueue
> +39,840 B +996
> org.apache.activemq.command.SessionId
> +55,480 B +1,387
>
> 10 x restarts (+ GC)
> Class name - allocated objects
> Bytes Objects Allocated
>
> org.springframework.osgi.service.importer.support.internal.aop.ImportedOsgiServiceProxyAdvice
> +3,168 B +99
>
> org.springframework.transaction.interceptor.AbstractFallbackTransactionAttributeSource$DefaultCacheKey
> +8,008,968 B +333,707
> org.apache.activemq.command.ActiveMQQueue
> +44,280 B +1,107
> org.apache.activemq.command.SessionId
> +114,160 B +2,854
>
> a) Are the results above what is expected or is there an issue with our
> code/3rd party libraries holding on to references
> when it shouldn't?
>
I think you could call it expected, cause you probably are probably
providing some services that those bundles connect to. In your case I'd try
to do a refresh on the container before and see how many of the old
"references" are gone then.
>
>
> For us the primary driver for choosing OSGI was actually a business driver
> and was the promise of 'hot deployment'
> (although as a developer I recognise other technical benefits of choosing
> an
> enabler for a modular architecture).
>
> How far can we go with hot deployment?
>
>
As far as the perm space let's you go. Might be intersting to try with JDK7
and the class unloading flag, or even with a JDK8.
JDK8 should support unloading of classes much better cause it's the first
jvm where the "old" hotspot and JRockit are actually merged.
> b) It looks as though there will always be an upper bound of the number of
> times a bundle can be stopped and started due to the class meta data being
> added into the Perm Gen space?
>
yep, see above
>
> c) Even if the Perm Gen issue was solved it looks as though there are
> issues
> with objects unable to be GCed, if these are caused by 3rd party libraries
> there is probably little we can do to fix this so again there will be an
> upper bound to the number of bundle retarts before a server restart is
> necessary?
>
>
>
I'd think doing a refresh on the container will solve most of your issues
for this, cause it will do a rewiring of all bundles, and therefore all
"old" references to your previously registered services will be gone then.
>
> --
> View this message in context:
> http://karaf.922171.n3.nabble.com/Multiple-restarts-effect-on-heap-and-Perm-Gen-memory-tp4028550.html
> Sent from the Karaf - User mailing list archive at Nabble.com.
>
Regards, Achim
--
Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
Project Lead
OPS4J Pax for Vaadin <http://team.ops4j.org/wiki/display/PAXVAADIN/Home>
Commiter & Project Lead
blog <http://notizblog.nierbeck.de/>