You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Kevan Miller <ke...@gmail.com> on 2005/10/20 23:45:45 UTC
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Hi John,
You're correct.
I have an OpenEJB patch which creates a ClassLoader per Proxy creation in
the OpenEJB client. This ClassLoader and the dynamically generated classes
can then be GCed when no longer in use. With this patch, your test runs to
completion (100,000 iterations) with no noticeable growth in heap or permgen
space.
I'd like to investigate OpenEJB's class loading structure a bit before I
call this a fix. Also, this scenario (and other Geronimo server scenarios)
seems to be screaming for a little caching of the proxy and associated
dynamically generated classes...
--kevan
On 10/17/05, John Sisson (JIRA) <de...@geronimo.apache.org> wrote:
>
> [
> http://issues.apache.org/jira/browse/GERONIMO-1062?page=comments#action_12332227]
>
> John Sisson commented on GERONIMO-1062:
> ---------------------------------------
>
> These proxy classes appear to be loaded by the AppClassLoader (system
> class loader). AFAIK, classes loaded by the system class loader never get
> unloaded.
>
> > OpenEJB client leaks memory on each JNDI lookup. Get
> OutOfMemoryException after a few thousand lookups
> >
> -------------------------------------------------------------------------------------------------------
>
> >
> > Key: GERONIMO-1062
> > URL: http://issues.apache.org/jira/browse/GERONIMO-1062
> > Project: Geronimo
> > Type: Bug
> > Components: OpenEJB
> > Versions: 1.0-M5, 1.0-M4
> > Environment: Windows XP
> > j2sdk1.4.2_08
> > Reporter: John Sisson
> > Priority: Critical
> > Fix For: 1.0
> > Attachments: AllObjects.zip, apps.zip
> >
> > I have a standalone java application that uses OpenEJB's JNDI
> implementation (org.openejb.client.RemoteInitialContextFactory ). It
> appears that each JNDI lookup of an EJB is leaking memory and it looks like
> it is related to cglib.
> > I have attached to this JIRA a simple EJB and standalone java client
> application that can be used to reproduce the problem.
> > Extract the zip file to the geronimo\sandbox directory
> > cd sandbox\echoTest
> > maven -o
> > deploy the echoTest.ear file in the sandbox\echoTest\target directory
> > cd sandbox\echoTestStandaloneClient
> > maven -o
> > cd target
> > java -jar echoTestStandaloneClient-0.1-SNAPSHOT.jar
> > The console will be updated with the number of lookups performed. E.Ghere are a few examples with different JVM settings:
> > C:\OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>set
> JAVA_HOME=C:\Program Files\Java\j2sdk1.4.2_08
> >
> C:\OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>"%JAVA_HOME%"\bin\java.exe
> -jar target\echoTestStandaloneClient-0.
> > 1-SNAPSHOT.jar
> > After 0 lookups: delta: 1,911,024, free: 1,911,024, used: 120,592,
> total: 2,031,616, max: 66,650,112
> > After 1000 lookups: delta: 417,040, free: 2,328,064, used: 2,603,520,
> total: 4,931,584, max: 66,650,112
> > After 2000 lookups: delta: 1,765,304, free: 4,093,368, used: 5,151,304,
> total: 9,244,672, max: 66,650,112
> > After 3000 lookups: delta: 1,759,288, free: 5,852,656, used: 7,303,696,
> total: 13,156,352, max: 66,650,112
> > After 4000 lookups: delta: 2,182,664, free: 8,035,320, used: 10,081,288,
> total: 18,116,608, max: 66,650,112
> > After 5000 lookups: delta: 1,579,184, free: 9,614,504, used: 11,574,104,
> total: 21,188,608, max: 66,650,112
> > Lookups performed: 5432
> > Exception in thread "main" net.sf.cglib.core.CodeGenerationException:
> java.lang.reflect.InvocationTargetExce
> > ption-->null
> > at net.sf.cglib.core.AbstractClassGenerator.create (
> AbstractClassGenerator.java:237)
> > at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
> > at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
> > at org.openejb.client.CgLibProxyFactory.newProxyInstance (
> CgLibProxyFactory.java:92)
> > at org.openejb.client.CgLibProxyFactory.newProxyInstance(
> CgLibProxyFactory.java:81)
> > at org.openejb.client.ProxyManager.newProxyInstance(ProxyManager.java
> :90)
> > at org.openejb.client.EJBHomeHandler.createEJBHomeProxy(
> EJBHomeHandler.java:106)
> > at org.openejb.client.JNDIContext.createEJBHomeProxy(JNDIContext.java
> :212)
> > at org.openejb.client.JNDIContext.lookup (JNDIContext.java:245)
> > at javax.naming.InitialContext.lookup(InitialContext.java:347)
> > at org.acme.EchoTestStandaloneClient.testLookup(
> EchoTestStandaloneClient.java:87)
> > at org.acme.EchoTestStandaloneClient.main (EchoTestStandaloneClient.java
> :76)
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke (
> DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:324)
> > at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:384)
> > at net.sf.cglib.core.AbstractClassGenerator.create (
> AbstractClassGenerator.java:219)
> > ... 11 more
> > Caused by: java.lang.OutOfMemoryError
> > If I try this on JDK 1.5.0_05, it is worse as the JVM becomes
> unresponsive and uses a lot of CPU and I have to kill the JVM:
> > C:\OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>java
> -jar target\echoTestStandaloneClient-0.1-SNAPSHOT.jar
> > After 0 lookups: delta: 1,863,336, free: 1,863,336, used: 168,280,
> total: 2,031,616, max: 66,650,112
> > After 1000 lookups: delta: 482,408, free: 2,345,744, used: 2,684,144,
> total: 5,029,888, max: 66,650,112
> > After 2000 lookups: delta: 1,865,064, free: 4,210,808, used: 5,267,336,
> total: 9,478,144, max: 66,650,112
> > After 3000 lookups: delta: 2,116,488, free: 6,327,296, used: 7,967,744,
> total: 14,295,040, max: 66,650,112
> > After 4000 lookups: delta: 2,007,008, free: 8,334,304, used: 10,499,104,
> total: 18,833,408, max: 66,650,112
> > After 5000 lookups: delta: 2,094,096, free: 10,428,400, used:
> 13,164,560, total: 23,592,960, max: 66,650,112
> > Lookups performed: 5099
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
>
>
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Posted by Kevan Miller <ke...@gmail.com>.
Hi Dain,
Thanks for the reminder. I did try the latest cglib a week or so ago. It
didn't help the problems in the server. I'm not sure if I was running with
the latest version of cglib in John's client, I'll verify (I'm not sure what
version John's build is picking up). However, this leak seems quite
fundamental. I'm not hopeful. The classes are being generated and defined to
the Default ClassLoader. Thus, these generated Class objects, their static
variables, and all transitive references will never be freed. Unless you
cglib is supposed to detect this situation and create its own ClassLoader, I
don't see how a cglib update would help...
--kevan
On 10/20/05, Dain Sundstrom <da...@iq80.com> wrote:
>
> I'd start by trying the newest version of cglib. IIRC there were
> some bugs in this area that they recently fixed.
>
> -dain
>
> On Oct 20, 2005, at 3:08 PM, David Jencks wrote:
>
> >
> > On Oct 20, 2005, at 2:45 PM, Kevan Miller wrote:
> >
> >
> >> Hi John,
> >> You're correct.
> >>
> >> I have an OpenEJB patch which creates a ClassLoader per Proxy
> >> creation in the OpenEJB client. This ClassLoader and the
> >> dynamically generated classes can then be GCed when no longer in
> >> use. With this patch, your test runs to completion (100,000
> >> iterations) with no noticeable growth in heap or permgen space.
> >>
> >> I'd like to investigate OpenEJB's class loading structure a bit
> >> before I call this a fix. Also, this scenario (and other Geronimo
> >> server scenarios) seems to be screaming for a little caching of
> >> the proxy and associated dynamically generated classes...
> >>
> >
> >
> > I spent a couple minutes looking at cglib source code and it looked
> > to me as if they were making some effort to cache enhanced
> > classes. I would start by trying to figure out why this isn't
> > working for us. AbstractClassGenerator.create(Object key) line 199
> > looks like it's key.
> >
> > thanks
> > david jencks
> >
> >>
> >> --kevan
> >>
> >> On 10/17/05, John Sisson (JIRA) <de...@geronimo.apache.org>
> >> wrote: [ http://issues.apache.org/jira/browse/GERONIMO-1062?
> >> page=comments#action_12332227 ]
> >>
> >>>
> >>> John Sisson commented on GERONIMO-1062:
> >>> ---------------------------------------
> >>>
> >>> These proxy classes appear to be loaded by the AppClassLoader
> >>> (system class loader). AFAIK, classes loaded by the system class
> >>> loader never get unloaded.
> >>>
> >>> > OpenEJB client leaks memory on each JNDI lookup. Get
> >>> OutOfMemoryException after a few thousand lookups
> >>> >
> >>> --------------------------------------------------------------------
> >>> -----------------------------------
> >>> >
> >>> > Key: GERONIMO-1062
> >>> > URL: http://issues.apache.org/jira/browse/GERONIMO-1062
> >>> > Project: Geronimo
> >>> > Type: Bug
> >>> > Components: OpenEJB
> >>> > Versions: 1.0-M5, 1.0-M4
> >>> > Environment: Windows XP
> >>> > j2sdk1.4.2_08
> >>> > Reporter: John Sisson
> >>> > Priority: Critical
> >>> > Fix For: 1.0
> >>> > Attachments: AllObjects.zip, apps.zip
> >>> >
> >>> > I have a standalone java application that uses OpenEJB's JNDI
> >>> implementation
> >>> (org.openejb.client.RemoteInitialContextFactory ). It appears
> >>> that each JNDI lookup of an EJB is leaking memory and it looks
> >>> like it is related to cglib.
> >>> > I have attached to this JIRA a simple EJB and standalone java
> >>> client application that can be used to reproduce the problem.
> >>> > Extract the zip file to the geronimo\sandbox directory
> >>> > cd sandbox\echoTest
> >>> > maven -o
> >>> > deploy the echoTest.ear file in the sandbox\echoTest\target
> >>> directory
> >>> > cd sandbox\echoTestStandaloneClient
> >>> > maven -o
> >>> > cd target
> >>> > java -jar echoTestStandaloneClient-0.1-SNAPSHOT.jar
> >>> > The console will be updated with the number of lookups
> >>> performed. E.G here are a few examples with different JVM settings:
> >>> > C:\OpenSourceJava\geronimo\trunk\sandbox
> >>> \echoTestStandaloneClient>set JAVA_HOME=C:\Program Files\Java
> >>> \j2sdk1.4.2_08
> >>> > C:\OpenSourceJava\geronimo\trunk\sandbox
> >>> \echoTestStandaloneClient>"%JAVA_HOME%"\bin\java.exe -jar target
> >>> \echoTestStandaloneClient-0.
> >>> > 1-SNAPSHOT.jar
> >>> > After 0 lookups: delta: 1,911,024, free: 1,911,024, used:
> >>> 120,592, total: 2,031,616, max: 66,650,112
> >>> > After 1000 lookups: delta: 417,040, free: 2,328,064, used:
> >>> 2,603,520, total: 4,931,584, max: 66,650,112
> >>> > After 2000 lookups: delta: 1,765,304, free: 4,093,368, used:
> >>> 5,151,304, total: 9,244,672, max: 66,650,112
> >>> > After 3000 lookups: delta: 1,759,288, free: 5,852,656, used:
> >>> 7,303,696, total: 13,156,352, max: 66,650,112
> >>> > After 4000 lookups: delta: 2,182,664, free: 8,035,320, used:
> >>> 10,081,288, total: 18,116,608, max: 66,650,112
> >>> > After 5000 lookups: delta: 1,579,184, free: 9,614,504, used:
> >>> 11,574,104, total: 21,188,608, max: 66,650,112
> >>> > Lookups performed: 5432
> >>> > Exception in thread "main"
> >>> net.sf.cglib.core.CodeGenerationException :
> >>> java.lang.reflect.InvocationTargetExce
> >>> > ption-->null
> >>> > at net.sf.cglib.core.AbstractClassGenerator.create
> >>> (AbstractClassGenerator.java:237)
> >>> > at net.sf.cglib.proxy.Enhancer.createHelper
> >>> (Enhancer.java:377)
> >>> > at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
> >>> > at
> >>> org.openejb.client.CgLibProxyFactory.newProxyInstance
> >>> (CgLibProxyFactory.java:92)
> >>> > at org.openejb.client.CgLibProxyFactory.newProxyInstance
> >>> (CgLibProxyFactory.java :81)
> >>> > at org.openejb.client.ProxyManager.newProxyInstance
> >>> (ProxyManager.java:90)
> >>> > at org.openejb.client.EJBHomeHandler.createEJBHomeProxy
> >>> ( EJBHomeHandler.java:106)
> >>> > at org.openejb.client.JNDIContext.createEJBHomeProxy
> >>> (JNDIContext.java:212)
> >>> > at org.openejb.client.JNDIContext.lookup
> >>> ( JNDIContext.java:245)
> >>> > at javax.naming.InitialContext.lookup
> >>> (InitialContext.java:347)
> >>> > at org.acme.EchoTestStandaloneClient.testLookup
> >>> ( EchoTestStandaloneClient.java:87)
> >>> > at org.acme.EchoTestStandaloneClient.main
> >>> (EchoTestStandaloneClient.java:76)
> >>> > Caused by: java.lang.reflect.InvocationTargetException
> >>> > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown
> >>> Source)
> >>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
> >>> (DelegatingMethodAccessorImpl.java :25)
> >>> > at java.lang.reflect.Method.invoke(Method.java:324)
> >>> > at net.sf.cglib.core.ReflectUtils.defineClass
> >>> (ReflectUtils.java:384)
> >>> > at net.sf.cglib.core.AbstractClassGenerator.create
> >>> (AbstractClassGenerator.java:219)
> >>> > ... 11 more
> >>> > Caused by: java.lang.OutOfMemoryError
> >>> > If I try this on JDK 1.5.0_05, it is worse as the JVM becomes
> >>> unresponsive and uses a lot of CPU and I have to kill the JVM:
> >>> > C:\OpenSourceJava\geronimo\trunk\sandbox
> >>> \echoTestStandaloneClient>java -jar target
> >>> \echoTestStandaloneClient-0.1-SNAPSHOT.jar
> >>> > After 0 lookups: delta: 1,863,336, free: 1,863,336, used:
> >>> 168,280, total: 2,031,616, max: 66,650,112
> >>> > After 1000 lookups: delta: 482,408, free: 2,345,744, used:
> >>> 2,684,144, total: 5,029,888, max: 66,650,112
> >>> > After 2000 lookups: delta: 1,865,064, free: 4,210,808, used:
> >>> 5,267,336, total: 9,478,144, max: 66,650,112
> >>> > After 3000 lookups: delta: 2,116,488, free: 6,327,296, used:
> >>> 7,967,744, total: 14,295,040, max: 66,650,112
> >>> > After 4000 lookups: delta: 2,007,008, free: 8,334,304, used:
> >>> 10,499,104, total: 18,833,408, max: 66,650,112
> >>> > After 5000 lookups: delta: 2,094,096, free: 10,428,400, used:
> >>> 13,164,560, total: 23,592,960, max: 66,650,112
> >>> > Lookups performed: 5099
> >>>
> >>> --
> >>> This message is automatically generated by JIRA.
> >>> -
> >>> If you think it was sent incorrectly contact one of the
> >>> administrators:
> >>> http://issues.apache.org/jira/secure/Administrators.jspa
> >>> -
> >>> For more information on JIRA, see:
> >>> http://www.atlassian.com/software/jira
> >>>
> >>>
> >>
> >>
> >
> >
>
>
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Posted by Dain Sundstrom <da...@iq80.com>.
I'd start by trying the newest version of cglib. IIRC there were
some bugs in this area that they recently fixed.
-dain
On Oct 20, 2005, at 3:08 PM, David Jencks wrote:
>
> On Oct 20, 2005, at 2:45 PM, Kevan Miller wrote:
>
>
>> Hi John,
>> You're correct.
>>
>> I have an OpenEJB patch which creates a ClassLoader per Proxy
>> creation in the OpenEJB client. This ClassLoader and the
>> dynamically generated classes can then be GCed when no longer in
>> use. With this patch, your test runs to completion (100,000
>> iterations) with no noticeable growth in heap or permgen space.
>>
>> I'd like to investigate OpenEJB's class loading structure a bit
>> before I call this a fix. Also, this scenario (and other Geronimo
>> server scenarios) seems to be screaming for a little caching of
>> the proxy and associated dynamically generated classes...
>>
>
>
> I spent a couple minutes looking at cglib source code and it looked
> to me as if they were making some effort to cache enhanced
> classes. I would start by trying to figure out why this isn't
> working for us. AbstractClassGenerator.create(Object key) line 199
> looks like it's key.
>
> thanks
> david jencks
>
>>
>> --kevan
>>
>> On 10/17/05, John Sisson (JIRA) <de...@geronimo.apache.org>
>> wrote: [ http://issues.apache.org/jira/browse/GERONIMO-1062?
>> page=comments#action_12332227 ]
>>
>>>
>>> John Sisson commented on GERONIMO-1062:
>>> ---------------------------------------
>>>
>>> These proxy classes appear to be loaded by the AppClassLoader
>>> (system class loader). AFAIK, classes loaded by the system class
>>> loader never get unloaded.
>>>
>>> > OpenEJB client leaks memory on each JNDI lookup. Get
>>> OutOfMemoryException after a few thousand lookups
>>> >
>>> --------------------------------------------------------------------
>>> -----------------------------------
>>> >
>>> > Key: GERONIMO-1062
>>> > URL: http://issues.apache.org/jira/browse/GERONIMO-1062
>>> > Project: Geronimo
>>> > Type: Bug
>>> > Components: OpenEJB
>>> > Versions: 1.0-M5, 1.0-M4
>>> > Environment: Windows XP
>>> > j2sdk1.4.2_08
>>> > Reporter: John Sisson
>>> > Priority: Critical
>>> > Fix For: 1.0
>>> > Attachments: AllObjects.zip, apps.zip
>>> >
>>> > I have a standalone java application that uses OpenEJB's JNDI
>>> implementation
>>> (org.openejb.client.RemoteInitialContextFactory ). It appears
>>> that each JNDI lookup of an EJB is leaking memory and it looks
>>> like it is related to cglib.
>>> > I have attached to this JIRA a simple EJB and standalone java
>>> client application that can be used to reproduce the problem.
>>> > Extract the zip file to the geronimo\sandbox directory
>>> > cd sandbox\echoTest
>>> > maven -o
>>> > deploy the echoTest.ear file in the sandbox\echoTest\target
>>> directory
>>> > cd sandbox\echoTestStandaloneClient
>>> > maven -o
>>> > cd target
>>> > java -jar echoTestStandaloneClient-0.1-SNAPSHOT.jar
>>> > The console will be updated with the number of lookups
>>> performed. E.G here are a few examples with different JVM settings:
>>> > C:\OpenSourceJava\geronimo\trunk\sandbox
>>> \echoTestStandaloneClient>set JAVA_HOME=C:\Program Files\Java
>>> \j2sdk1.4.2_08
>>> > C:\OpenSourceJava\geronimo\trunk\sandbox
>>> \echoTestStandaloneClient>"%JAVA_HOME%"\bin\java.exe -jar target
>>> \echoTestStandaloneClient-0.
>>> > 1-SNAPSHOT.jar
>>> > After 0 lookups: delta: 1,911,024, free: 1,911,024, used:
>>> 120,592, total: 2,031,616, max: 66,650,112
>>> > After 1000 lookups: delta: 417,040, free: 2,328,064, used:
>>> 2,603,520, total: 4,931,584, max: 66,650,112
>>> > After 2000 lookups: delta: 1,765,304, free: 4,093,368, used:
>>> 5,151,304, total: 9,244,672, max: 66,650,112
>>> > After 3000 lookups: delta: 1,759,288, free: 5,852,656, used:
>>> 7,303,696, total: 13,156,352, max: 66,650,112
>>> > After 4000 lookups: delta: 2,182,664, free: 8,035,320, used:
>>> 10,081,288, total: 18,116,608, max: 66,650,112
>>> > After 5000 lookups: delta: 1,579,184, free: 9,614,504, used:
>>> 11,574,104, total: 21,188,608, max: 66,650,112
>>> > Lookups performed: 5432
>>> > Exception in thread "main"
>>> net.sf.cglib.core.CodeGenerationException:
>>> java.lang.reflect.InvocationTargetExce
>>> > ption-->null
>>> > at net.sf.cglib.core.AbstractClassGenerator.create
>>> (AbstractClassGenerator.java:237)
>>> > at net.sf.cglib.proxy.Enhancer.createHelper
>>> (Enhancer.java:377)
>>> > at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
>>> > at
>>> org.openejb.client.CgLibProxyFactory.newProxyInstance
>>> (CgLibProxyFactory.java:92)
>>> > at org.openejb.client.CgLibProxyFactory.newProxyInstance
>>> (CgLibProxyFactory.java:81)
>>> > at org.openejb.client.ProxyManager.newProxyInstance
>>> (ProxyManager.java:90)
>>> > at org.openejb.client.EJBHomeHandler.createEJBHomeProxy
>>> (EJBHomeHandler.java:106)
>>> > at org.openejb.client.JNDIContext.createEJBHomeProxy
>>> (JNDIContext.java:212)
>>> > at org.openejb.client.JNDIContext.lookup
>>> (JNDIContext.java:245)
>>> > at javax.naming.InitialContext.lookup
>>> (InitialContext.java:347)
>>> > at org.acme.EchoTestStandaloneClient.testLookup
>>> (EchoTestStandaloneClient.java:87)
>>> > at org.acme.EchoTestStandaloneClient.main
>>> (EchoTestStandaloneClient.java:76)
>>> > Caused by: java.lang.reflect.InvocationTargetException
>>> > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown
>>> Source)
>>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
>>> (DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:324)
>>> > at net.sf.cglib.core.ReflectUtils.defineClass
>>> (ReflectUtils.java:384)
>>> > at net.sf.cglib.core.AbstractClassGenerator.create
>>> (AbstractClassGenerator.java:219)
>>> > ... 11 more
>>> > Caused by: java.lang.OutOfMemoryError
>>> > If I try this on JDK 1.5.0_05, it is worse as the JVM becomes
>>> unresponsive and uses a lot of CPU and I have to kill the JVM:
>>> > C:\OpenSourceJava\geronimo\trunk\sandbox
>>> \echoTestStandaloneClient>java -jar target
>>> \echoTestStandaloneClient-0.1-SNAPSHOT.jar
>>> > After 0 lookups: delta: 1,863,336, free: 1,863,336, used:
>>> 168,280, total: 2,031,616, max: 66,650,112
>>> > After 1000 lookups: delta: 482,408, free: 2,345,744, used:
>>> 2,684,144, total: 5,029,888, max: 66,650,112
>>> > After 2000 lookups: delta: 1,865,064, free: 4,210,808, used:
>>> 5,267,336, total: 9,478,144, max: 66,650,112
>>> > After 3000 lookups: delta: 2,116,488, free: 6,327,296, used:
>>> 7,967,744, total: 14,295,040, max: 66,650,112
>>> > After 4000 lookups: delta: 2,007,008, free: 8,334,304, used:
>>> 10,499,104, total: 18,833,408, max: 66,650,112
>>> > After 5000 lookups: delta: 2,094,096, free: 10,428,400, used:
>>> 13,164,560, total: 23,592,960, max: 66,650,112
>>> > Lookups performed: 5099
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> If you think it was sent incorrectly contact one of the
>>> administrators:
>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>> -
>>> For more information on JIRA, see:
>>> http://www.atlassian.com/software/jira
>>>
>>>
>>
>>
>
>
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Posted by Kevan Miller <ke...@gmail.com>.
Hi David,
Thanks for the pointer. I'll take a look to understand why there
wasn't/isn't any caching going on.
--kevan
On 10/20/05, David Jencks <da...@yahoo.com> wrote:
>
>
> On Oct 20, 2005, at 2:45 PM, Kevan Miller wrote:
>
> > Hi John,
> > You're correct.
> >
> > I have an OpenEJB patch which creates a ClassLoader per Proxy
> > creation in the OpenEJB client. This ClassLoader and the dynamically
> > generated classes can then be GCed when no longer in use. With this
> > patch, your test runs to completion (100,000 iterations) with no
> > noticeable growth in heap or permgen space.
> >
> > I'd like to investigate OpenEJB's class loading structure a bit
> > before I call this a fix. Also, this scenario (and other Geronimo
> > server scenarios) seems to be screaming for a little caching of the
> > proxy and associated dynamically generated classes...
>
>
> I spent a couple minutes looking at cglib source code and it looked to
> me as if they were making some effort to cache enhanced classes. I
> would start by trying to figure out why this isn't working for us.
> AbstractClassGenerator.create(Object key) line 199 looks like it's key.
>
> thanks
> david jencks
> >
> > --kevan
> >
> > On 10/17/05, John Sisson (JIRA) <de...@geronimo.apache.org> wrote: [
> > http://issues.apache.org/jira/browse/GERONIMO-1062?
> > page=comments#action_12332227 ]
> >>
> >> John Sisson commented on GERONIMO-1062:
> >> ---------------------------------------
> >>
> >> These proxy classes appear to be loaded by the AppClassLoader (system
> >> class loader).AFAIK, classes loaded by the system class loader
> >> never get unloaded.
> >>
> >> > OpenEJB client leaks memory on each JNDI lookup.Get
> >> OutOfMemoryException after a few thousand lookups
> >> >
> >> ----------------------------------------------------------------------
> >> ---------------------------------
> >> >
> >> >Key: GERONIMO-1062
> >> >URL: http://issues.apache.org/jira/browse/GERONIMO-1062
> >> >Project: Geronimo
> >> > Type: Bug
> >> > Components: OpenEJB
> >> > Versions: 1.0-M5, 1.0-M4
> >> >Environment: Windows XP
> >> > j2sdk1.4.2_08
> >> > Reporter: John Sisson
> >> > Priority: Critical
> >> >Fix For: 1.0
> >> >Attachments: AllObjects.zip, apps.zip
> >> >
> >> > I have a standalone java application that uses OpenEJB's JNDI
> >> implementation (org.openejb.client.RemoteInitialContextFactory ).It
> >> appears that each JNDI lookup of an EJB is leaking memory and it
> >> looks like it is related to cglib.
> >> > I have attached to this JIRA a simple EJB and standalone java
> >> client application that can be used to reproduce the problem.
> >> > Extract the zip file to the geronimo\sandbox directory
> >> > cd sandbox\echoTest
> >> > maven -o
> >> > deploy the echoTest.ear file in the sandbox\echoTest\target
> >> directory
> >> > cd sandbox\echoTestStandaloneClient
> >> > maven -o
> >> > cd target
> >> > java -jar echoTestStandaloneClient-0.1-SNAPSHOT.jar
> >> > The console will be updated with the number of lookups
> >> performed.E.G here are a few examples with different JVM settings:
> >> >
> >> C:\OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>set
> >> JAVA_HOME=C:\Program Files\Java\j2sdk1.4.2_08
> >> >
> >> C:
> >> \OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>"%JAVA
> >> _HOME%"\bin\java.exe -jar target\echoTestStandaloneClient-0.
> >> > 1-SNAPSHOT.jar
> >> > After 0 lookups: delta: 1,911,024, free: 1,911,024, used: 120,592,
> >> total: 2,031,616, max: 66,650,112
> >> > After 1000 lookups: delta: 417,040, free: 2,328,064, used:
> >> 2,603,520, total: 4,931,584, max: 66,650,112
> >> > After 2000 lookups: delta: 1,765,304, free: 4,093,368, used:
> >> 5,151,304, total: 9,244,672, max: 66,650,112
> >> > After 3000 lookups: delta: 1,759,288, free: 5,852,656, used:
> >> 7,303,696, total: 13,156,352, max: 66,650,112
> >> > After 4000 lookups: delta: 2,182,664, free: 8,035,320, used:
> >> 10,081,288, total: 18,116,608, max: 66,650,112
> >> > After 5000 lookups: delta: 1,579,184, free: 9,614,504, used:
> >> 11,574,104, total: 21,188,608, max: 66,650,112
> >> >Lookups performed: 5432
> >> > Exception in thread "main"
> >> net.sf.cglib.core.CodeGenerationException:
> >> java.lang.reflect.InvocationTargetExce
> >> > ption-->null
> >> > at net.sf.cglib.core.AbstractClassGenerator.create
> >> (AbstractClassGenerator.java:237)
> >> > at
> >> net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
> >> > at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
> >> > at org.openejb.client.CgLibProxyFactory.newProxyInstance
> >> (CgLibProxyFactory.java:92)
> >> > at
> >> org.openejb.client.CgLibProxyFactory.newProxyInstance(CgLibProxyFactor
> >> y.java:81)
> >> > at
> >> org.openejb.client.ProxyManager.newProxyInstance(ProxyManager.java:
> >> 90)
> >> > at
> >> org.openejb.client.EJBHomeHandler.createEJBHomeProxy(EJBHomeHandler.ja
> >> va:106)
> >> > at
> >> org.openejb.client.JNDIContext.createEJBHomeProxy(JNDIContext.java:
> >> 212)
> >> > at org.openejb.client.JNDIContext.lookup
> >> (JNDIContext.java:245)
> >> > at
> >> javax.naming.InitialContext.lookup(InitialContext.java:347)
> >> > at
> >> org.acme.EchoTestStandaloneClient.testLookup(EchoTestStandaloneClient.
> >> java:87)
> >> > at org.acme.EchoTestStandaloneClient.main
> >> (EchoTestStandaloneClient.java:76)
> >> > Caused by: java.lang.reflect.InvocationTargetException
> >> > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown
> >> Source)
> >> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
> >> (DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:324)
> >> > at
> >> net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:384)
> >> > at net.sf.cglib.core.AbstractClassGenerator.create
> >> (AbstractClassGenerator.java:219)
> >> > ... 11 more
> >> > Caused by: java.lang.OutOfMemoryError
> >> > If I try this on JDK 1.5.0_05, it is worse as the JVM becomes
> >> unresponsive and uses a lot of CPU and I have to kill the JVM:
> >> >
> >> C:
> >> \OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>java
> >> -jar target\echoTestStandaloneClient-0.1-SNAPSHOT.jar
> >> > After 0 lookups: delta: 1,863,336, free: 1,863,336, used: 168,280,
> >> total: 2,031,616, max: 66,650,112
> >> > After 1000 lookups: delta: 482,408, free: 2,345,744, used:
> >> 2,684,144, total: 5,029,888, max: 66,650,112
> >> > After 2000 lookups: delta: 1,865,064, free: 4,210,808, used:
> >> 5,267,336, total: 9,478,144, max: 66,650,112
> >> > After 3000 lookups: delta: 2,116,488, free: 6,327,296, used:
> >> 7,967,744, total: 14,295,040, max: 66,650,112
> >> > After 4000 lookups: delta: 2,007,008, free: 8,334,304, used:
> >> 10,499,104, total: 18,833,408, max: 66,650,112
> >> > After 5000 lookups: delta: 2,094,096, free: 10,428,400, used:
> >> 13,164,560, total: 23,592,960, max: 66,650,112
> >> >Lookups performed: 5099
> >>
> >> --
> >> This message is automatically generated by JIRA.
> >> -
> >> If you think it was sent incorrectly contact one of the
> >> administrators:
> >> http://issues.apache.org/jira/secure/Administrators.jspa
> >> -
> >> For more information on JIRA, see:
> >> http://www.atlassian.com/software/jira
> >>
> >
>
>
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Posted by Kevan Miller <ke...@gmail.com>.
Thanks, Chris. Was having little fun wading through
KEY_FACTORY.newInstance... Cutting to the chase is much appreciated. I'll
give this a try...
--kevan
On 10/21/05, Chris Nokleberg <ch...@sixlegs.com> wrote:
>
> On Thu, 20 Oct 2005 15:08:05 -0700, David Jencks wrote:
> > I spent a couple minutes looking at cglib source code and it looked to
> > me as if they were making some effort to cache enhanced classes. I
> > would start by trying to figure out why this isn't working for us.
> > AbstractClassGenerator.create(Object key) line 199 looks like it's key.
>
> CGLIB uses a key constructed of all the arguments passed to Enhancer that
> could affect the bytecode of the generated class.
>
> This is the code in org.openejb.client.CgLibProxyFactory:
>
> private Enhancer getEnhancer(Class superClass, Class[] interfaces) {
> Enhancer enhancer;
> enhancer = new Enhancer();
> enhancer.setSuperclass (superClass);
> enhancer.setInterfaces(interfaces);
> enhancer.setCallbackFilter(new NoOverrideCallbackFilter(superClass));
> enhancer.setCallbackTypes(new Class[]{NoOp.class, MethodInterceptor.class});
> enhancer.setUseFactory(false);
> return enhancer;
> }
>
> The NoOverrideCallbackFilter class does not implement equals/hashCode, so
> using a new instance of it for each Enhancer will force generation of a
> new class. Because superClass is already part of the key (via
> setSuperclass) even something this simple should work:
>
> public int hashCode() { return 0; }
> public boolean equals(Object o) { return true; }
>
> The memory fixes that Dain alluded to were eliminating some unintentional
> references to old classloaders that would prevent them from getting
> GC'ed. I don't think they would affect Geronimo but I recommend upgrading
> anyway.
>
> Chris
>
>
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Posted by Chris Nokleberg <ch...@sixlegs.com>.
On Thu, 20 Oct 2005 15:08:05 -0700, David Jencks wrote:
> I spent a couple minutes looking at cglib source code and it looked to
> me as if they were making some effort to cache enhanced classes. I
> would start by trying to figure out why this isn't working for us.
> AbstractClassGenerator.create(Object key) line 199 looks like it's key.
CGLIB uses a key constructed of all the arguments passed to Enhancer that
could affect the bytecode of the generated class.
This is the code in org.openejb.client.CgLibProxyFactory:
private Enhancer getEnhancer(Class superClass, Class[] interfaces) {
Enhancer enhancer;
enhancer = new Enhancer();
enhancer.setSuperclass(superClass);
enhancer.setInterfaces(interfaces);
enhancer.setCallbackFilter(new NoOverrideCallbackFilter(superClass));
enhancer.setCallbackTypes(new Class[]{NoOp.class, MethodInterceptor.class});
enhancer.setUseFactory(false);
return enhancer;
}
The NoOverrideCallbackFilter class does not implement equals/hashCode, so
using a new instance of it for each Enhancer will force generation of a
new class. Because superClass is already part of the key (via
setSuperclass) even something this simple should work:
public int hashCode() { return 0; }
public boolean equals(Object o) { return true; }
The memory fixes that Dain alluded to were eliminating some unintentional
references to old classloaders that would prevent them from getting
GC'ed. I don't think they would affect Geronimo but I recommend upgrading
anyway.
Chris
Re: [jira] Commented: (GERONIMO-1062) OpenEJB client leaks memory on each JNDI lookup. Get OutOfMemoryException after a few thousand lookups
Posted by David Jencks <da...@yahoo.com>.
On Oct 20, 2005, at 2:45 PM, Kevan Miller wrote:
> Hi John,
> You're correct.
>
> I have an OpenEJB patch which creates a ClassLoader per Proxy
> creation in the OpenEJB client. This ClassLoader and the dynamically
> generated classes can then be GCed when no longer in use. With this
> patch, your test runs to completion (100,000 iterations) with no
> noticeable growth in heap or permgen space.
>
> I'd like to investigate OpenEJB's class loading structure a bit
> before I call this a fix. Also, this scenario (and other Geronimo
> server scenarios) seems to be screaming for a little caching of the
> proxy and associated dynamically generated classes...
I spent a couple minutes looking at cglib source code and it looked to
me as if they were making some effort to cache enhanced classes. I
would start by trying to figure out why this isn't working for us.
AbstractClassGenerator.create(Object key) line 199 looks like it's key.
thanks
david jencks
>
> --kevan
>
> On 10/17/05, John Sisson (JIRA) <de...@geronimo.apache.org> wrote: [
> http://issues.apache.org/jira/browse/GERONIMO-1062?
> page=comments#action_12332227 ]
>>
>> John Sisson commented on GERONIMO-1062:
>> ---------------------------------------
>>
>> These proxy classes appear to be loaded by the AppClassLoader (system
>> class loader). AFAIK, classes loaded by the system class loader
>> never get unloaded.
>>
>> > OpenEJB client leaks memory on each JNDI lookup. Get
>> OutOfMemoryException after a few thousand lookups
>> >
>> ----------------------------------------------------------------------
>> ---------------------------------
>> >
>> > Key: GERONIMO-1062
>> > URL: http://issues.apache.org/jira/browse/GERONIMO-1062
>> > Project: Geronimo
>> > Type: Bug
>> > Components: OpenEJB
>> > Versions: 1.0-M5, 1.0-M4
>> > Environment: Windows XP
>> > j2sdk1.4.2_08
>> > Reporter: John Sisson
>> > Priority: Critical
>> > Fix For: 1.0
>> > Attachments: AllObjects.zip, apps.zip
>> >
>> > I have a standalone java application that uses OpenEJB's JNDI
>> implementation (org.openejb.client.RemoteInitialContextFactory ). It
>> appears that each JNDI lookup of an EJB is leaking memory and it
>> looks like it is related to cglib.
>> > I have attached to this JIRA a simple EJB and standalone java
>> client application that can be used to reproduce the problem.
>> > Extract the zip file to the geronimo\sandbox directory
>> > cd sandbox\echoTest
>> > maven -o
>> > deploy the echoTest.ear file in the sandbox\echoTest\target
>> directory
>> > cd sandbox\echoTestStandaloneClient
>> > maven -o
>> > cd target
>> > java -jar echoTestStandaloneClient-0.1-SNAPSHOT.jar
>> > The console will be updated with the number of lookups
>> performed. E.G here are a few examples with different JVM settings:
>> >
>> C:\OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>set
>> JAVA_HOME=C:\Program Files\Java\j2sdk1.4.2_08
>> >
>> C:
>> \OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>"%JAVA
>> _HOME%"\bin\java.exe -jar target\echoTestStandaloneClient-0.
>> > 1-SNAPSHOT.jar
>> > After 0 lookups: delta: 1,911,024, free: 1,911,024, used: 120,592,
>> total: 2,031,616, max: 66,650,112
>> > After 1000 lookups: delta: 417,040, free: 2,328,064, used:
>> 2,603,520, total: 4,931,584, max: 66,650,112
>> > After 2000 lookups: delta: 1,765,304, free: 4,093,368, used:
>> 5,151,304, total: 9,244,672, max: 66,650,112
>> > After 3000 lookups: delta: 1,759,288, free: 5,852,656, used:
>> 7,303,696, total: 13,156,352, max: 66,650,112
>> > After 4000 lookups: delta: 2,182,664, free: 8,035,320, used:
>> 10,081,288, total: 18,116,608, max: 66,650,112
>> > After 5000 lookups: delta: 1,579,184, free: 9,614,504, used:
>> 11,574,104, total: 21,188,608, max: 66,650,112
>> > Lookups performed: 5432
>> > Exception in thread "main"
>> net.sf.cglib.core.CodeGenerationException:
>> java.lang.reflect.InvocationTargetExce
>> > ption-->null
>> > at net.sf.cglib.core.AbstractClassGenerator.create
>> (AbstractClassGenerator.java:237)
>> > at
>> net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>> > at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
>> > at org.openejb.client.CgLibProxyFactory.newProxyInstance
>> (CgLibProxyFactory.java:92)
>> > at
>> org.openejb.client.CgLibProxyFactory.newProxyInstance(CgLibProxyFactor
>> y.java:81)
>> > at
>> org.openejb.client.ProxyManager.newProxyInstance(ProxyManager.java:
>> 90)
>> > at
>> org.openejb.client.EJBHomeHandler.createEJBHomeProxy(EJBHomeHandler.ja
>> va:106)
>> > at
>> org.openejb.client.JNDIContext.createEJBHomeProxy(JNDIContext.java:
>> 212)
>> > at org.openejb.client.JNDIContext.lookup
>> (JNDIContext.java:245)
>> > at
>> javax.naming.InitialContext.lookup(InitialContext.java:347)
>> > at
>> org.acme.EchoTestStandaloneClient.testLookup(EchoTestStandaloneClient.
>> java:87)
>> > at org.acme.EchoTestStandaloneClient.main
>> (EchoTestStandaloneClient.java:76)
>> > Caused by: java.lang.reflect.InvocationTargetException
>> > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown
>> Source)
>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:324)
>> > at
>> net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:384)
>> > at net.sf.cglib.core.AbstractClassGenerator.create
>> (AbstractClassGenerator.java:219)
>> > ... 11 more
>> > Caused by: java.lang.OutOfMemoryError
>> > If I try this on JDK 1.5.0_05, it is worse as the JVM becomes
>> unresponsive and uses a lot of CPU and I have to kill the JVM:
>> >
>> C:
>> \OpenSourceJava\geronimo\trunk\sandbox\echoTestStandaloneClient>java
>> -jar target\echoTestStandaloneClient-0.1-SNAPSHOT.jar
>> > After 0 lookups: delta: 1,863,336, free: 1,863,336, used: 168,280,
>> total: 2,031,616, max: 66,650,112
>> > After 1000 lookups: delta: 482,408, free: 2,345,744, used:
>> 2,684,144, total: 5,029,888, max: 66,650,112
>> > After 2000 lookups: delta: 1,865,064, free: 4,210,808, used:
>> 5,267,336, total: 9,478,144, max: 66,650,112
>> > After 3000 lookups: delta: 2,116,488, free: 6,327,296, used:
>> 7,967,744, total: 14,295,040, max: 66,650,112
>> > After 4000 lookups: delta: 2,007,008, free: 8,334,304, used:
>> 10,499,104, total: 18,833,408, max: 66,650,112
>> > After 5000 lookups: delta: 2,094,096, free: 10,428,400, used:
>> 13,164,560, total: 23,592,960, max: 66,650,112
>> > Lookups performed: 5099
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the
>> administrators:
>> http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>