You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Glen Vermeylen <gl...@gmail.com> on 2007/02/21 12:17:48 UTC
Classloading deadlock in tomcat 5.5.20 / tomcat 6.0.9?
Hello all,
We've seen strange behaviour on production (tomcat 5.5.20/java 6):
immediately after restart no-one could login. Using jconsole we saw that
there was a deadlock in the classloader.
Today I've also had the problem on development (tomcat 6.0.9/ java 6). Its a
shame I was using tomcat 6.0.9 instead of 5.5.20, but it seems the two
problems are the same.
The problem occurs sometimes when multiple browsers are trying to connect to
the server while it is still starting up.
I've tried to reproduce the problem on both 5.5.20 and 6.0.9 by trying to
start multiple browsers during application startup, but I cannot trigger the
deadlock again.
Using jconsole I saw there were 2 httpthreads locking eachother.
Below are the stacktraces of both threads (I also have screenshots of
jconsole showing the threads the deadlocks-screen).
I couldn't find the problem myself, so I turn to this list for help.
Thanks in advance,
Glen.
Thread 1:
=======
Name: http-8080-1
State: BLOCKED on org.apache.catalina.loader.WebappClassLoader@682406 owned
by: http-8080-2
Total blocked: 25 Total waited: 2
Stack trace:
org.apache.catalina.loader.WebappClassLoader.findClassInternal (
WebappClassLoader.java:1780)
- locked org.apache.catalina.loader.ResourceEntry@136836c
org.apache.catalina.loader.WebappClassLoader.findClass(
WebappClassLoader.java:872)
org.apache.catalina.loader.WebappClassLoader.loadClass (
WebappClassLoader.java:1325)
org.apache.catalina.loader.WebappClassLoader.loadClass(
WebappClassLoader.java:1204)
<...remove company...>
net.sf.jzeno.echo.AbstractHardCodedStyle.<init>(AbstractHardCodedStyle.java:51)
<...remove company...>
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance (
DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
java.lang.Class.newInstance0(Class.java:355)
java.lang.Class.newInstance(Class.java:308)
net.sf.jzeno.echo.EchoSupport.getStyleManager (EchoSupport.java:2137)
net.sf.jzeno.echo.EchoSupport.getStyle(EchoSupport.java:2115)
net.sf.jzeno.echo.components.MenuBar.rebuild(MenuBar.java:162)
net.sf.jzeno.echo.components.MenuBar.<init>(MenuBar.java:124)
net.sf.jzeno.echo.components.MenuBar.<init>(MenuBar.java:246)
<...remove company...>
<...remove company...>
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance (
NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
java.lang.Class.newInstance0 (Class.java:355)
java.lang.Class.newInstance(Class.java:308)
net.sf.jzeno.util.FastFactory.newInstance(FastFactory.java:204)
net.sf.jzeno.util.FastFactory.create(FastFactory.java:105)
net.sf.jzeno.echo.EchoSupport.createDefaultLayout (EchoSupport.java:1165)
net.sf.jzeno.echo.EchoSupport.getLayout(EchoSupport.java:1137)
net.sf.jzeno.aop.ServletSupport.clearErrorsAndMessages(ServletSupport.java
:319)
nextapp.echoservlet.ContextFilter.preProcess( ContextFilter.java:73)
nextapp.echoservlet.EchoServer.process(EchoServer.java:384)
nextapp.echoservlet.EchoServer.doPost(EchoServer.java:281)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
javax.servlet.http.HttpServlet.service (HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke (
StandardWrapperValve.java:228)
org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:175)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java
:104)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:212)
org.apache.coyote.http11.Http11AprProcessor.process (Http11AprProcessor.java
:866)
org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(
Http11AprProtocol.java:716)
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1498)
java.lang.Thread.run (Thread.java:619)
Thread 2:
=======
Name: http-8080-2
State: BLOCKED on org.apache.catalina.loader.ResourceEntry@136836c owned by:
http-8080-1
Total blocked: 23 Total waited: 1
Stack trace:
org.apache.catalina.loader.WebappClassLoader.findClassInternal(
WebappClassLoader.java:1767)
org.apache.catalina.loader.WebappClassLoader.findClass(
WebappClassLoader.java:872)
org.apache.catalina.loader.WebappClassLoader.loadClass (
WebappClassLoader.java:1325)
org.apache.catalina.loader.WebappClassLoader.loadClass(
WebappClassLoader.java:1204)
java.lang.ClassLoader.defineClass1(Native Method)
java.lang.ClassLoader.defineClass(ClassLoader.java :620)
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
org.apache.catalina.loader.WebappClassLoader.findClassInternal(
WebappClassLoader.java:1815)
- locked org.apache.catalina.loader.ResourceEntry@154598e
org.apache.catalina.loader.WebappClassLoader.findClass(
WebappClassLoader.java:872)
org.apache.catalina.loader.WebappClassLoader.loadClass(
WebappClassLoader.java:1325)
org.apache.catalina.loader.WebappClassLoader.loadClass (
WebappClassLoader.java:1204)
<... remove company....>
net.sf.jzeno.echo.AbstractHardCodedStyle.<init>(AbstractHardCodedStyle.java
:51)
<... remove company....>
sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance (Constructor.java:513)
java.lang.Class.newInstance0(Class.java:355)
java.lang.Class.newInstance(Class.java:308)
net.sf.jzeno.echo.EchoSupport.getStyleManager(EchoSupport.java:2137)
net.sf.jzeno.echo.EchoSupport.getStyle (EchoSupport.java:2115)
net.sf.jzeno.echo.components.MenuBar.rebuild(MenuBar.java:162)
net.sf.jzeno.echo.components.MenuBar.<init>(MenuBar.java:124)
net.sf.jzeno.echo.components.MenuBar.<init>(MenuBar.java :246)
<... remove company....>
<... remove company....>
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java :39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
java.lang.Class.newInstance0(Class.java:355)
java.lang.Class.newInstance(Class.java:308)
net.sf.jzeno.util.FastFactory.newInstance(FastFactory.java:204)
net.sf.jzeno.util.FastFactory.create(FastFactory.java:105)
net.sf.jzeno.echo.EchoSupport.createDefaultLayout (EchoSupport.java:1165)
net.sf.jzeno.echo.EchoSupport.getLayout(EchoSupport.java:1137)
net.sf.jzeno.aop.ServletSupport.clearErrorsAndMessages(ServletSupport.java
:319)
nextapp.echoservlet.ContextFilter.preProcess( ContextFilter.java:73)
nextapp.echoservlet.EchoServer.process(EchoServer.java:384)
nextapp.echoservlet.EchoServer.doPost(EchoServer.java:281)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
javax.servlet.http.HttpServlet.service (HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke (
StandardWrapperValve.java:228)
org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:175)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java
:104)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:212)
org.apache.coyote.http11.Http11AprProcessor.process (Http11AprProcessor.java
:866)
org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(
Http11AprProtocol.java:716)
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1498)
java.lang.Thread.run (Thread.java:619)
Re: Classloading deadlock in tomcat 5.5.20 / tomcat 6.0.9?
Posted by Glen Vermeylen <gl...@gmail.com>.
Thanks for the info.
We haven't had the problem again so far, so we will leave it as is (the
deadlock can should occur after a restart, so we can monitor it)
I may look into the fix on my development pc and let you know.
Glen.
2007/2/21, Remy Maucherat <re...@apache.org>:
>
> Glen Vermeylen wrote:
> > Hello all,
> >
> > We've seen strange behaviour on production (tomcat 5.5.20/java 6):
> > immediately after restart no-one could login. Using jconsole we saw that
> > there was a deadlock in the classloader.
> > Today I've also had the problem on development (tomcat 6.0.9/ java 6).
> > Its a
> > shame I was using tomcat 6.0.9 instead of 5.5.20, but it seems the two
> > problems are the same.
> > The problem occurs sometimes when multiple browsers are trying to
> > connect to
> > the server while it is still starting up.
> > I've tried to reproduce the problem on both 5.5.20 and 6.0.9 by trying
> to
> > start multiple browsers during application startup, but I cannot trigger
> > the
> > deadlock again.
> > Using jconsole I saw there were 2 httpthreads locking eachother.
> >
> > Below are the stacktraces of both threads (I also have screenshots of
> > jconsole showing the threads the deadlocks-screen).
> >
> > I couldn't find the problem myself, so I turn to this list for help.
>
> Please test the patch if you can.
>
> Rémy
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Classloading deadlock in tomcat 5.5.20 / tomcat 6.0.9?
Posted by Remy Maucherat <re...@apache.org>.
Glen Vermeylen wrote:
> Hello all,
>
> We've seen strange behaviour on production (tomcat 5.5.20/java 6):
> immediately after restart no-one could login. Using jconsole we saw that
> there was a deadlock in the classloader.
> Today I've also had the problem on development (tomcat 6.0.9/ java 6).
> Its a
> shame I was using tomcat 6.0.9 instead of 5.5.20, but it seems the two
> problems are the same.
> The problem occurs sometimes when multiple browsers are trying to
> connect to
> the server while it is still starting up.
> I've tried to reproduce the problem on both 5.5.20 and 6.0.9 by trying to
> start multiple browsers during application startup, but I cannot trigger
> the
> deadlock again.
> Using jconsole I saw there were 2 httpthreads locking eachother.
>
> Below are the stacktraces of both threads (I also have screenshots of
> jconsole showing the threads the deadlocks-screen).
>
> I couldn't find the problem myself, so I turn to this list for help.
Please test the patch if you can.
Rémy
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Classloading deadlock in tomcat 5.5.20 / tomcat 6.0.9?
Posted by Remy Maucherat <re...@apache.org>.
Glen Vermeylen wrote:
> Hello all,
>
> We've seen strange behaviour on production (tomcat 5.5.20/java 6):
> immediately after restart no-one could login. Using jconsole we saw that
> there was a deadlock in the classloader.
> Today I've also had the problem on development (tomcat 6.0.9/ java 6).
> Its a
> shame I was using tomcat 6.0.9 instead of 5.5.20, but it seems the two
> problems are the same.
> The problem occurs sometimes when multiple browsers are trying to
> connect to
> the server while it is still starting up.
> I've tried to reproduce the problem on both 5.5.20 and 6.0.9 by trying to
> start multiple browsers during application startup, but I cannot trigger
> the
> deadlock again.
> Using jconsole I saw there were 2 httpthreads locking eachother.
>
> Below are the stacktraces of both threads (I also have screenshots of
> jconsole showing the threads the deadlocks-screen).
>
> I couldn't find the problem myself, so I turn to this list for help.
The fix for bug 37458 did not turn out well (findClassInternal may loop,
so syncing on an entry during the defineClass call is bad).
Rémy
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org