You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geronimo.apache.org by Seth Ladd <se...@gmail.com> on 2004/11/11 21:17:16 UTC
Strategies for Eliminating OOM Exceptions on Redeploys?
Hello,
Congrats on releasing another version of Geronimo! I'm hoping someone
can shed some light on this subject.
On all the application servers/web servers I've tried, they all seem
susceptible to (potential) OutOfMemoryExceptions when certain webapps
are deployed over and over. This is often the result of some
libraries not releasing classes it has loaded. This causes the WebApp
ClassLoader to never be released, and thus the memory leak. The
Hibernate library comes to mind here with the way it uses cglib.
I'm wondering if Geronimo has any strategies for dealing with this
problem? I'd love to be able to run a Java Application Server as long
as the Apache HTTPD server, but I've never been able to. Too many
application redeploys will always crash the server.
If Geronimo doesn't have any direct code that explicitly handles this
problem, maybe there are other strategies involved? Being able to
explicitly destroy the classloader that loaded the application would
be incredibly invaluable. We could then have Java App Servers running
24/7!
Thanks very much!
Seth
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by Seth Ladd <se...@gmail.com>.
On Mon, 15 Nov 2004 16:02:38 -0800, Dain Sundstrom
<ds...@gluecode.com> wrote:
> Can you add a jira entry?
>
> http://nagoya.apache.org/jira/secure/BrowseProject.jspa?id=10220
Done and done.
http://nagoya.apache.org/jira/browse/GERONIMO-484
Thanks for your help. I'll do anything I can to help out with this
issue. I've been dreaming of a day where I can have a java
application server up for as long as Apache HTTPD, or my linux box for
that matter. :)
Seth
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by Dain Sundstrom <ds...@gluecode.com>.
Can you add a jira entry?
http://nagoya.apache.org/jira/secure/BrowseProject.jspa?id=10220
-dain
--
Dain Sundstrom
Chief Architect
Gluecode Software
310.536.8355, ext. 26
On Nov 15, 2004, at 10:26 AM, Seth Ladd wrote:
> Hello,
>
> I have just run a test that tests Geronimo's ability to redeploy .war
> files. The results don't look so good, but I'm hoping the info I've
> collected will help expose the problem (whether the problem is with me
> or the server :)
>
> I've included the script, .war file, web.xml, and two exception traces
> (one from the deploy client and one from the server) I apologize for
> the length of this post. If there's a better way to include all this
> information, please let me know.
>
> It appears as if the server is running out of memory. Note that I did
> not alter the native configuration of Geronimo in any way. This is a
> stock M3 download, using JDK 1.4.2_05 on Windows XP SP2.
>
> It's my expectation that the app server would stay up and running
> indefinitely after redeploys of .war files. Especially this .war file
> since it merely contains a single welcome.jsp. I hope this info is
> helpful, and let me know what else you might need. I'll gladly send
> the script and .war file to others that might want to test this.
>
> Thanks,
> Seth
>
> My test script:
> $ cat over-and-over.sh
> while /bin/true; do
> java -jar bin/deployer.jar --user system --password manager deploy
> ../eclipse/wo
> rkspace/MinimalWebapp/build/minimaltest.war
> sleep 1
> wget -q -O - http://localhost:8080/minimaltest/welcome.jsp > /dev/null
> java -jar bin/deployer.jar --user system --password manager undeploy
> minimaltest
> sleep 1
> done
>
> Structure of minimaltest.war:
> $ jar tf ../eclipse/workspace/MinimalWebapp/build/minimaltest.war
> META-INF/
> META-INF/MANIFEST.MF
> WEB-INF/
> welcome.jsp
> WEB-INF/web.xml
>
> Contents of web.xml:
> $ cat ../eclipse/workspace/MinimalWebapp/web/WEB-INF/web.xml
> <?xml version="1.0" encoding="UTF-8"?>
> <web-app xmlns="http://java.sun.com/xml/ns/j2ee"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
> http://java.sun.com/
> xml/ns/j2ee/web-app_2_4.xsd"
> version="2.4">
> <welcome-file-list>
> <welcome-file>welcome.jsp</welcome-file>
> </welcome-file-list>
> </web-app>
>
> So, as you can see, it's a very minimal webapp. It doesn't initialize
> anything, nor does it include any 3rd party jars or libs. It doesn't
> even load up any classes, and the welcome.jsp only says "Hello,
> world!"
>
> After 1434 deploy cycles, we receive this exception from the deploy
> client:
> ---------------------------------
> Deployment failed
> Server reports: null
> java.lang.IllegalStateException
> at
> org.apache.geronimo.kernel.Kernel.stopConfiguration(Kernel.java:437)
> at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:324)
> at
> mx4j.server.ReflectionMBeanInvoker.invokeImpl(ReflectionMBeanInvoker.ja
> va:152)
> at
> mx4j.server.ReflectionMBeanInvoker.doInvoke(ReflectionMBeanInvoker.java
> :119)
> at
> mx4j.server.ReflectionMBeanInvoker.invoke(ReflectionMBeanInvoker.java:
> 54)
> at
> mx4j.server.interceptor.InvokerMBeanServerInterceptor.invoke(InvokerMBe
> anServerInterceptor.java:235)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBe
> anServerInterceptor.java:121)
> at
> mx4j.server.interceptor.SecurityMBeanServerInterceptor.invoke(SecurityM
> BeanServerInterceptor.java:86)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBe
> anServerInterceptor.java:121)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBe
> anServerInterceptor.java:121)
> at
> mx4j.server.interceptor.ContextClassLoaderMBeanServerInterceptor.invoke
> (ContextClassLoaderMBeanServerInterceptor.java:205)
> at mx4j.server.MX4JMBeanServer.invoke(MX4JMBeanServer.java:1079)
> at
> mx4j.remote.rmi.RMIConnectionInvoker.invoke(RMIConnectionInvoker.java:
> 222)
> at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:324)
> at
> mx4j.remote.rmi.RMIConnectionProxy.invoke(RMIConnectionProxy.java:36)
> at
> mx4j.remote.rmi.RMIConnectionSubjectInvoker.chain(RMIConnectionSubjectI
> nvoker.java:98)
> at
> mx4j.remote.rmi.RMIConnectionSubjectInvoker.access$000(RMIConnectionSub
> jectInvoker.java:32)
> at
> mx4j.remote.rmi.RMIConnectionSubjectInvoker$1.run(RMIConnectionSubjectI
> nvoker.java:89)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAsPrivileged(Subject.java:500)
> at mx4j.remote.MX4JRemoteUtils.subjectInvoke(MX4JRemoteUtils.java:151)
> at
> mx4j.remote.rmi.RMIConnectionSubjectInvoker.subjectInvoke(RMIConnection
> SubjectInvoker.java:85)
> at
> mx4j.remote.rmi.RMIConnectionSubjectInvoker.invoke(RMIConnectionSubject
> Invoker.java:79)
> at $Proxy0.invoke(Unknown Source)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.
> java:224)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> va:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:324)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
> at sun.rmi.transport.Transport$1.run(Transport.java:148)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:
> 460)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.j
> ava:701)
> at java.lang.Thread.run(Thread.java:534)
> Caused by: org.apache.geronimo.kernel.config.InvalidConfigException:
> Invalid GBean configuration for geronimo.config:name="minimaltest"
> at org.apache.geronimo.kernel.Kernel.stopGBean(Kernel.java:389)
> at
> org.apache.geronimo.kernel.Kernel.stopConfiguration(Kernel.java:431)
> ... 39 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> org.apache.geronimo.gbean.jmx.GBeanMBeanOperation.invoke(GBeanMBeanOper
> ation.java:146)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean.invoke(GBeanMBean.java:823)
> at org.apache.geronimo.gbean.jmx.RawInvoker.invoke(RawInvoker.java:89)
> at
> org.apache.geronimo.gbean.jmx.RawOperationInvoker.invoke(RawOperationIn
> voker.java:34)
> at
> org.apache.geronimo.gbean.jmx.CGLibMethodInterceptor.intercept(CGLibMet
> hodInterceptor.java:112)
> at
> org.apache.geronimo.kernel.config.ConfigurationStore$$EnhancerByCGLIB$$
> 45c69f36.updateConfiguration(<generated>)
> at
> org.apache.geronimo.kernel.config.Configuration.doStop(Configuration.ja
> va:269)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean.doStop(GBeanMBean.java:622)
> at
> org.apache.geronimo.gbean.jmx.AbstractManagedObject.attemptFullStop(Abs
> tractManagedObject.java:583)
> at
> org.apache.geronimo.gbean.jmx.AbstractManagedObject.stop(AbstractManage
> dObject.java:396)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean$12.invoke(GBeanMBean.java:
> 1048)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBeanOperation.invoke(GBeanMBeanOper
> ation.java:142)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean.invoke(GBeanMBean.java:844)
> at
> mx4j.server.interceptor.InvokerMBeanServerInterceptor.invoke(InvokerMBe
> anServerInterceptor.java:218)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBe
> anServerInterceptor.java:121)
> at
> mx4j.server.interceptor.SecurityMBeanServerInterceptor.invoke(SecurityM
> BeanServerInterceptor.java:86)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBe
> anServerInterceptor.java:121)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBe
> anServerInterceptor.java:121)
> at
> mx4j.server.interceptor.ContextClassLoaderMBeanServerInterceptor.invoke
> (ContextClassLoaderMBeanServerInterceptor.java:205)
> at mx4j.server.MX4JMBeanServer.invoke(MX4JMBeanServer.java:1079)
> at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:288)
> at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:283)
> at org.apache.geronimo.kernel.Kernel.stopGBean(Kernel.java:385)
> ... 40 more
> Caused by: java.lang.OutOfMemoryError
> -----------------------------------
>
> And this exception block occurs in the server log:
> -----------------------------------
> 07:53:57,796 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID
> org/apache/geronimo/DeployerSystem which ca
> n not be loaded.
> 07:53:57,796 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/DebugConsole
> which can
> not be loaded.
> 07:53:57,796 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/ClientSystem
> which can
> not be loaded.
> 07:53:57,812 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/J2EEDeployer
> which can
> not be loaded.
> 07:53:57,812 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/Client which
> can not be
> loaded.
> 07:53:57,812 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/Secure which
> can not be
> loaded.
> 07:53:58,265 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/System which
> can not be
> loaded.
> 07:53:58,265 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/SystemJMS
> which can not
> be loaded.
> 07:53:58,281 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/Demo which
> can not be l
> oaded.
> 07:53:58,281 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID minimaltest which can not be
> loaded.
> 07:53:58,281 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID org/apache/geronimo/Server which
> can not be
> loaded.
> 07:53:58,281 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID
> org/apache/geronimo/DefaultDatabase which c
> an not be loaded.
> 07:53:58,281 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID
> org/apache/geronimo/SystemDatabase which ca
> n not be loaded.
> 07:53:58,281 ERROR [ConfigurationManagerImpl]
> [org.apache.geronimo.kernel.config
> .ConfigurationStore$$EnhancerByCGLIB$$45c69f36:
> geronimo.system:role=Configurati
> onStore,type=Local] defines configID
> org/apache/geronimo/ActiveMQServer which ca
> n not be loaded.
> 07:53:58,312 INFO [Configuration] Stopping configuration minimaltest
> 07:53:58,312 WARN [Configuration] Could not unregister child
> geronimo.server:J2
> EEApplication=null,J2EEServer=geronimo,j2eeType=WebModule,name=minimalt
> est
> javax.management.InstanceNotFoundException: MBeanServer cannot find
> MBean with O
> bjectName
> geronimo.server:J2EEApplication=null,J2EEServer=geronimo,j2eeType=WebM
> odule,name=minimaltest
> at
> mx4j.server.MX4JMBeanServer.findMBeanMetaData(MX4JMBeanServer.java:49
> 1)
> at
> mx4j.server.MX4JMBeanServer.unregisterMBean(MX4JMBeanServer.java:940)
>
> at
> org.apache.geronimo.kernel.Kernel.unloadGBean(Kernel.java:395)
> at
> org.apache.geronimo.kernel.config.Configuration.doStop(Configuration.
> java:254)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean.doStop(GBeanMBean.java:622)
> at
> org.apache.geronimo.gbean.jmx.AbstractManagedObject.attemptFullStop(A
> bstractManagedObject.java:583)
> at
> org.apache.geronimo.gbean.jmx.AbstractManagedObject.stop(AbstractMana
> gedObject.java:396)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean$12.invoke(GBeanMBean.java:10
> 48)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBeanOperation.invoke(GBeanMBeanOp
> eration.java:142)
> at
> org.apache.geronimo.gbean.jmx.GBeanMBean.invoke(GBeanMBean.java:844)
> at
> mx4j.server.interceptor.InvokerMBeanServerInterceptor.invoke(InvokerM
> BeanServerInterceptor.java:218)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM
> BeanServerInterceptor.java:121)
> at
> mx4j.server.interceptor.SecurityMBeanServerInterceptor.invoke(Securit
> yMBeanServerInterceptor.java:86)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM
> BeanServerInterceptor.java:121)
> at
> mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM
> BeanServerInterceptor.java:121)
> at
> mx4j.server.interceptor.ContextClassLoaderMBeanServerInterceptor.invo
> ke(ContextClassLoaderMBeanServerInterceptor.java:205)
> at
> mx4j.server.MX4JMBeanServer.invoke(MX4JMBeanServer.java:1079)
> at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:288)
> at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:283)
> at org.apache.geronimo.kernel.Kernel.stopGBean(Kernel.java:385)
> at
> org.apache.geronimo.kernel.Kernel.stopConfiguration(Kernel.java:431)
> at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown
> Source)
> ---------------------------------
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by Seth Ladd <se...@gmail.com>.
Hello,
I have just run a test that tests Geronimo's ability to redeploy .war
files. The results don't look so good, but I'm hoping the info I've
collected will help expose the problem (whether the problem is with me
or the server :)
I've included the script, .war file, web.xml, and two exception traces
(one from the deploy client and one from the server) I apologize for
the length of this post. If there's a better way to include all this
information, please let me know.
It appears as if the server is running out of memory. Note that I did
not alter the native configuration of Geronimo in any way. This is a
stock M3 download, using JDK 1.4.2_05 on Windows XP SP2.
It's my expectation that the app server would stay up and running
indefinitely after redeploys of .war files. Especially this .war file
since it merely contains a single welcome.jsp. I hope this info is
helpful, and let me know what else you might need. I'll gladly send
the script and .war file to others that might want to test this.
Thanks,
Seth
My test script:
$ cat over-and-over.sh
while /bin/true; do
java -jar bin/deployer.jar --user system --password manager deploy ../eclipse/wo
rkspace/MinimalWebapp/build/minimaltest.war
sleep 1
wget -q -O - http://localhost:8080/minimaltest/welcome.jsp > /dev/null
java -jar bin/deployer.jar --user system --password manager undeploy minimaltest
sleep 1
done
Structure of minimaltest.war:
$ jar tf ../eclipse/workspace/MinimalWebapp/build/minimaltest.war
META-INF/
META-INF/MANIFEST.MF
WEB-INF/
welcome.jsp
WEB-INF/web.xml
Contents of web.xml:
$ cat ../eclipse/workspace/MinimalWebapp/web/WEB-INF/web.xml
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/
xml/ns/j2ee/web-app_2_4.xsd"
version="2.4">
<welcome-file-list>
<welcome-file>welcome.jsp</welcome-file>
</welcome-file-list>
</web-app>
So, as you can see, it's a very minimal webapp. It doesn't initialize
anything, nor does it include any 3rd party jars or libs. It doesn't
even load up any classes, and the welcome.jsp only says "Hello,
world!"
After 1434 deploy cycles, we receive this exception from the deploy client:
---------------------------------
Deployment failed
Server reports: null
java.lang.IllegalStateException
at org.apache.geronimo.kernel.Kernel.stopConfiguration(Kernel.java:437)
at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at mx4j.server.ReflectionMBeanInvoker.invokeImpl(ReflectionMBeanInvoker.java:152)
at mx4j.server.ReflectionMBeanInvoker.doInvoke(ReflectionMBeanInvoker.java:119)
at mx4j.server.ReflectionMBeanInvoker.invoke(ReflectionMBeanInvoker.java:54)
at mx4j.server.interceptor.InvokerMBeanServerInterceptor.invoke(InvokerMBeanServerInterceptor.java:235)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:121)
at mx4j.server.interceptor.SecurityMBeanServerInterceptor.invoke(SecurityMBeanServerInterceptor.java:86)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:121)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:121)
at mx4j.server.interceptor.ContextClassLoaderMBeanServerInterceptor.invoke(ContextClassLoaderMBeanServerInterceptor.java:205)
at mx4j.server.MX4JMBeanServer.invoke(MX4JMBeanServer.java:1079)
at mx4j.remote.rmi.RMIConnectionInvoker.invoke(RMIConnectionInvoker.java:222)
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at mx4j.remote.rmi.RMIConnectionProxy.invoke(RMIConnectionProxy.java:36)
at mx4j.remote.rmi.RMIConnectionSubjectInvoker.chain(RMIConnectionSubjectInvoker.java:98)
at mx4j.remote.rmi.RMIConnectionSubjectInvoker.access$000(RMIConnectionSubjectInvoker.java:32)
at mx4j.remote.rmi.RMIConnectionSubjectInvoker$1.run(RMIConnectionSubjectInvoker.java:89)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:500)
at mx4j.remote.MX4JRemoteUtils.subjectInvoke(MX4JRemoteUtils.java:151)
at mx4j.remote.rmi.RMIConnectionSubjectInvoker.subjectInvoke(RMIConnectionSubjectInvoker.java:85)
at mx4j.remote.rmi.RMIConnectionSubjectInvoker.invoke(RMIConnectionSubjectInvoker.java:79)
at $Proxy0.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:224)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
at sun.rmi.transport.Transport$1.run(Transport.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:534)
Caused by: org.apache.geronimo.kernel.config.InvalidConfigException:
Invalid GBean configuration for geronimo.config:name="minimaltest"
at org.apache.geronimo.kernel.Kernel.stopGBean(Kernel.java:389)
at org.apache.geronimo.kernel.Kernel.stopConfiguration(Kernel.java:431)
... 39 more
Caused by: java.lang.reflect.InvocationTargetException
at org.apache.geronimo.gbean.jmx.GBeanMBeanOperation.invoke(GBeanMBeanOperation.java:146)
at org.apache.geronimo.gbean.jmx.GBeanMBean.invoke(GBeanMBean.java:823)
at org.apache.geronimo.gbean.jmx.RawInvoker.invoke(RawInvoker.java:89)
at org.apache.geronimo.gbean.jmx.RawOperationInvoker.invoke(RawOperationInvoker.java:34)
at org.apache.geronimo.gbean.jmx.CGLibMethodInterceptor.intercept(CGLibMethodInterceptor.java:112)
at org.apache.geronimo.kernel.config.ConfigurationStore$$EnhancerByCGLIB$$45c69f36.updateConfiguration(<generated>)
at org.apache.geronimo.kernel.config.Configuration.doStop(Configuration.java:269)
at org.apache.geronimo.gbean.jmx.GBeanMBean.doStop(GBeanMBean.java:622)
at org.apache.geronimo.gbean.jmx.AbstractManagedObject.attemptFullStop(AbstractManagedObject.java:583)
at org.apache.geronimo.gbean.jmx.AbstractManagedObject.stop(AbstractManagedObject.java:396)
at org.apache.geronimo.gbean.jmx.GBeanMBean$12.invoke(GBeanMBean.java:1048)
at org.apache.geronimo.gbean.jmx.GBeanMBeanOperation.invoke(GBeanMBeanOperation.java:142)
at org.apache.geronimo.gbean.jmx.GBeanMBean.invoke(GBeanMBean.java:844)
at mx4j.server.interceptor.InvokerMBeanServerInterceptor.invoke(InvokerMBeanServerInterceptor.java:218)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:121)
at mx4j.server.interceptor.SecurityMBeanServerInterceptor.invoke(SecurityMBeanServerInterceptor.java:86)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:121)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:121)
at mx4j.server.interceptor.ContextClassLoaderMBeanServerInterceptor.invoke(ContextClassLoaderMBeanServerInterceptor.java:205)
at mx4j.server.MX4JMBeanServer.invoke(MX4JMBeanServer.java:1079)
at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:288)
at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:283)
at org.apache.geronimo.kernel.Kernel.stopGBean(Kernel.java:385)
... 40 more
Caused by: java.lang.OutOfMemoryError
-----------------------------------
And this exception block occurs in the server log:
-----------------------------------
07:53:57,796 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/DeployerSystem which ca
n not be loaded.
07:53:57,796 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/DebugConsole which can
not be loaded.
07:53:57,796 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/ClientSystem which can
not be loaded.
07:53:57,812 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/J2EEDeployer which can
not be loaded.
07:53:57,812 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/Client which can not be
loaded.
07:53:57,812 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/Secure which can not be
loaded.
07:53:58,265 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/System which can not be
loaded.
07:53:58,265 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/SystemJMS which can not
be loaded.
07:53:58,281 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/Demo which can not be l
oaded.
07:53:58,281 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID minimaltest which can not be loaded.
07:53:58,281 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/Server which can not be
loaded.
07:53:58,281 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/DefaultDatabase which c
an not be loaded.
07:53:58,281 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/SystemDatabase which ca
n not be loaded.
07:53:58,281 ERROR [ConfigurationManagerImpl] [org.apache.geronimo.kernel.config
.ConfigurationStore$$EnhancerByCGLIB$$45c69f36: geronimo.system:role=Configurati
onStore,type=Local] defines configID org/apache/geronimo/ActiveMQServer which ca
n not be loaded.
07:53:58,312 INFO [Configuration] Stopping configuration minimaltest
07:53:58,312 WARN [Configuration] Could not unregister child geronimo.server:J2
EEApplication=null,J2EEServer=geronimo,j2eeType=WebModule,name=minimaltest
javax.management.InstanceNotFoundException: MBeanServer cannot find MBean with O
bjectName geronimo.server:J2EEApplication=null,J2EEServer=geronimo,j2eeType=WebM
odule,name=minimaltest
at mx4j.server.MX4JMBeanServer.findMBeanMetaData(MX4JMBeanServer.java:49
1)
at mx4j.server.MX4JMBeanServer.unregisterMBean(MX4JMBeanServer.java:940)
at org.apache.geronimo.kernel.Kernel.unloadGBean(Kernel.java:395)
at org.apache.geronimo.kernel.config.Configuration.doStop(Configuration.
java:254)
at org.apache.geronimo.gbean.jmx.GBeanMBean.doStop(GBeanMBean.java:622)
at org.apache.geronimo.gbean.jmx.AbstractManagedObject.attemptFullStop(A
bstractManagedObject.java:583)
at org.apache.geronimo.gbean.jmx.AbstractManagedObject.stop(AbstractMana
gedObject.java:396)
at org.apache.geronimo.gbean.jmx.GBeanMBean$12.invoke(GBeanMBean.java:10
48)
at org.apache.geronimo.gbean.jmx.GBeanMBeanOperation.invoke(GBeanMBeanOp
eration.java:142)
at org.apache.geronimo.gbean.jmx.GBeanMBean.invoke(GBeanMBean.java:844)
at mx4j.server.interceptor.InvokerMBeanServerInterceptor.invoke(InvokerM
BeanServerInterceptor.java:218)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM
BeanServerInterceptor.java:121)
at mx4j.server.interceptor.SecurityMBeanServerInterceptor.invoke(Securit
yMBeanServerInterceptor.java:86)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM
BeanServerInterceptor.java:121)
at mx4j.server.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM
BeanServerInterceptor.java:121)
at mx4j.server.interceptor.ContextClassLoaderMBeanServerInterceptor.invo
ke(ContextClassLoaderMBeanServerInterceptor.java:205)
at mx4j.server.MX4JMBeanServer.invoke(MX4JMBeanServer.java:1079)
at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:288)
at org.apache.geronimo.kernel.Kernel.invoke(Kernel.java:283)
at org.apache.geronimo.kernel.Kernel.stopGBean(Kernel.java:385)
at org.apache.geronimo.kernel.Kernel.stopConfiguration(Kernel.java:431)
at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
---------------------------------
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by Seth Ladd <se...@gmail.com>.
> Can you be more specific on the cglib problem you have seen? We use
> cglib for proxies.
It's only a theory, really. I've put together different .war files
that have different levels of size. One for just nothing but a
web.xml all the way up to a full blown spring + hibernate application.
Some of these .war files are known to leak memory on constant
redeploys for other servlet containers, while others seem to do just
fine.
> We do our best to release any handles to classes and class loaders, but
> it is the vm that must decide to garbage collect classes. From what I
> have seen, this is a bigger problem on Java 5, because the current
> release does puts classes in a permeant store which is not (or rarely)
> collected.
Hmm... well, luckily I'm using 1.4.2. :)
I was running my tests on friday and left them running before I left
work. Should be interesting to return Monday.
Thanks,
Seth
ps The tests merely deploy a .war file using the command line tool,
wait 1 second, request a single .jsp using wget, then undeploy.
Repeat!
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by Dain Sundstrom <ds...@gluecode.com>.
On Nov 12, 2004, at 5:52 PM, Seth Ladd wrote:
> On Thu, 11 Nov 2004 12:50:12 -0800, David Jencks
> <dj...@gluecode.com> wrote:
>> I certainly agree this is an important issue. I think so far our main
>> defense has been "good design". Do you have a specific example that
>> causes problems on geronimo, or are you just warning us to check this
>> thoroughly? I take it you are also hinting that we should test with a
>> web app that uses hibernate:-)
>
> Hi! Thanks for the quick response.
>
> I do not have a example webapp at this moment, but I have just started
> one to test. I'm skeptical due to previous experiences with Tomcat,
> Resin, and Jetty. Certain 3rd party libraries (totally out of my
> control) are not built with ClassLoader release in mind. For
> instance, cglib (used by hibernate) does not release its dynamic
> proxies when the webapp is destroyed on a redeploy. In a Single
> Application, this is not a problem. In the dynamic world of
> application containers, this is certainly a problem.
Can you be more specific on the cglib problem you have seen? We use
cglib for proxies.
> Thanks for being interested in this. I've just downloaded M3 and will
> run my test (that I've used for other application servers) on it.
> I'll respond back with results.
>
> AFAICT, though, there isn't much we can do about the problem because
> there is no way to completely (and forcibly) garbage collect
> ClassLoaders. But maybe Geronimo has solved the problem? :)
We do our best to release any handles to classes and class loaders, but
it is the vm that must decide to garbage collect classes. From what I
have seen, this is a bigger problem on Java 5, because the current
release does puts classes in a permeant store which is not (or rarely)
collected.
-dain
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by Seth Ladd <se...@gmail.com>.
On Thu, 11 Nov 2004 12:50:12 -0800, David Jencks <dj...@gluecode.com> wrote:
> I certainly agree this is an important issue. I think so far our main
> defense has been "good design". Do you have a specific example that
> causes problems on geronimo, or are you just warning us to check this
> thoroughly? I take it you are also hinting that we should test with a
> web app that uses hibernate:-)
Hi! Thanks for the quick response.
I do not have a example webapp at this moment, but I have just started
one to test. I'm skeptical due to previous experiences with Tomcat,
Resin, and Jetty. Certain 3rd party libraries (totally out of my
control) are not built with ClassLoader release in mind. For
instance, cglib (used by hibernate) does not release its dynamic
proxies when the webapp is destroyed on a redeploy. In a Single
Application, this is not a problem. In the dynamic world of
application containers, this is certainly a problem.
Thanks for being interested in this. I've just downloaded M3 and will
run my test (that I've used for other application servers) on it.
I'll respond back with results.
AFAICT, though, there isn't much we can do about the problem because
there is no way to completely (and forcibly) garbage collect
ClassLoaders. But maybe Geronimo has solved the problem? :)
More info soon,
Seth
Re: Strategies for Eliminating OOM Exceptions on Redeploys?
Posted by David Jencks <dj...@gluecode.com>.
I certainly agree this is an important issue. I think so far our main
defense has been "good design". Do you have a specific example that
causes problems on geronimo, or are you just warning us to check this
thoroughly? I take it you are also hinting that we should test with a
web app that uses hibernate:-)
many thanks,
david jencks
On Nov 11, 2004, at 12:17 PM, Seth Ladd wrote:
> Hello,
>
> Congrats on releasing another version of Geronimo! I'm hoping someone
> can shed some light on this subject.
>
> On all the application servers/web servers I've tried, they all seem
> susceptible to (potential) OutOfMemoryExceptions when certain webapps
> are deployed over and over. This is often the result of some
> libraries not releasing classes it has loaded. This causes the WebApp
> ClassLoader to never be released, and thus the memory leak. The
> Hibernate library comes to mind here with the way it uses cglib.
>
> I'm wondering if Geronimo has any strategies for dealing with this
> problem? I'd love to be able to run a Java Application Server as long
> as the Apache HTTPD server, but I've never been able to. Too many
> application redeploys will always crash the server.
>
> If Geronimo doesn't have any direct code that explicitly handles this
> problem, maybe there are other strategies involved? Being able to
> explicitly destroy the classloader that loaded the application would
> be incredibly invaluable. We could then have Java App Servers running
> 24/7!
>
> Thanks very much!
> Seth
>