You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Zampani, Michael" <za...@amazon.com> on 2011/11/29 00:37:40 UTC
Tomcat-7.0.23 startup deadlocks
Hi,
I'm trying to upgrade an existing webapp installation from 7.0.22 -> 7.0.23.
On startup, it appears to deadlock, with the threads stuck in
ContainerBase.setRealm
HostConfig.deployDirectories
ContainerBase.startInternal
server.xml and thread dump attached.
Note that I do not define any <Realm>s in server.xml
The webapp is simple and contains only static html, js, css, and images.
Copying the web.xml from the example webapp shipped with tomcat seems to fix it, but this worked in previous versions of Tomcat.
Not sure if it's relevant, but it is the ROOT webapp.
I've attached a sample WAR file to reproduce it. It should work fine once deployed. Remove WEB-INF/web.xml and restart Tomcat to see it fail.
Any ideas?
-Michael
RE: Tomcat-7.0.23 startup deadlocks
Posted by "Zampani, Michael" <za...@amazon.com>.
Looks like the attachments didn't all make it. Trying again
> Hi,
>
> I'm trying to upgrade an existing webapp installation from 7.0.22 -> 7.0.23.
>
> On startup, it appears to deadlock, with the threads stuck in
> ContainerBase.setRealm HostConfig.deployDirectories
> ContainerBase.startInternal
>
> server.xml and thread dump attached.
> Note that I do not define any <Realm>s in server.xml
>
> The webapp is simple and contains only static html, js, css, and images.
> Copying the web.xml from the example webapp shipped with tomcat seems
> to fix it, but this worked in previous versions of Tomcat.
> Not sure if it's relevant, but it is the ROOT webapp.
>
> I've attached a sample WAR file to reproduce it. It should work fine once
> deployed. Remove WEB-INF/web.xml and restart Tomcat to see it fail.
>
> Any ideas?
>
> -Michael
Re: Tomcat-7.0.23 startup deadlocks
Posted by Mark Thomas <ma...@apache.org>.
On 29/11/2011 19:27, Zampani, Michael wrote:
>> I think you are right. When no realm is configured in server.xml and in the
>> context of a webapp, StandardEngine tries to set a default realm.
>> That is guarded by a lock, which is held by the main thread.
>>
>> To work around this problem, you can define a simple realm inside the
>> context of your webapp or globally in server.xml.
>>
>> To fix the problem we will have to look at the locking for setRealm in
>> ContainerBase.
>>
>> Regards
>> Felix
>
> I can confirm that reverting CLN 1190190
> http://svn.apache.org/viewvc?view=revision&revision=1190190
> fixes this.
>
> Alternatively, removing the synchronized block from setRealm() also 'fixes' this.
> It looks like we can add some locks around Realm,
That fix has just been applied to trunk and will be back-ported shortly.
> but it doesn't look like ContainerBase is even close to being actually threadsafe.
Agreed.
> I'm not sure what the existing standards for Tomcat code are in this area.
That is a grey area. Generally, we avoid making code thread safe unless
we have to. The ContainerBase and related code generally doesn't need to
be thread safe as there is only ever one thread working with that code.
The threaded deployment doesn't really change that. Where thread safety
does come in is when we consider on-the-fly updating e.g. via JMX while
the component is in the STARTED state. Generally, we look at those
issues on a case-by-case basis and where there is a good case for
on-the-fly updates, we make it thread safe, where there isn't we make it
read-only. One of the parts of the GSOC project that looked at JMX a few
years ago was meant to review all attributes for thread-safety (and
expected to make most of them read-only). Unfortunately, that project
didn't make much progress.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat-7.0.23 startup deadlocks
Posted by "Zampani, Michael" <za...@amazon.com>.
> I think you are right. When no realm is configured in server.xml and in the
> context of a webapp, StandardEngine tries to set a default realm.
> That is guarded by a lock, which is held by the main thread.
>
> To work around this problem, you can define a simple realm inside the
> context of your webapp or globally in server.xml.
>
> To fix the problem we will have to look at the locking for setRealm in
> ContainerBase.
>
> Regards
> Felix
I can confirm that reverting CLN 1190190
http://svn.apache.org/viewvc?view=revision&revision=1190190
fixes this.
Alternatively, removing the synchronized block from setRealm() also 'fixes' this.
It looks like we can add some locks around Realm, but it doesn't look like ContainerBase is even close to being actually threadsafe.
I'm not sure what the existing standards for Tomcat code are in this area.
-Michael
RE: Tomcat-7.0.23 startup deadlocks
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 29.11.2011 01:28, schrieb Zampani, Michael:
>> -----Original Message-----
>> From: markt@apache.org [mailto:markt@apache.org]
>> Sent: Monday, November 28, 2011 4:02 PM
>> To: Tomcat Users List
>> Subject: Re: Tomcat-7.0.23 startup deadlocks
>>
>> "Zampani, Michael" <za...@amazon.com> wrote:
>>
>> >Hi,
>> >
>> >I'm trying to upgrade an existing webapp installation from 7.0.22
>> ->
>> >7.0.23.
>> >
>> >On startup, it appears to deadlock, with the threads stuck in
>> >ContainerBase.setRealm HostConfig.deployDirectories
>> >ContainerBase.startInternal
>> >
>> >server.xml and thread dump attached.
>> >Note that I do not define any <Realm>s in server.xml
>> >
>> >The webapp is simple and contains only static html, js, css, and
>> >images.
>> >Copying the web.xml from the example webapp shipped with tomcat
>> seems
>> >to fix it, but this worked in previous versions of Tomcat.
>> >Not sure if it's relevant, but it is the ROOT webapp.
>> >
>> >I've attached a sample WAR file to reproduce it. It should work
>> fine
>> >once deployed. Remove WEB-INF/web.xml and restart Tomcat to see it
>> >fail.
>> >
>> >Any ideas?
>> >
>> >-Michael
>> >
>>
>> >---------------------------------------------------------------------
>> >To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> Attachments are usually blocked. Better to paste in-line.
>>
>> Is there an actual deadlock or do things just seem to stop moving
>> forwards?
>>
>> Possible causes:
>> - threaded container starts (although they are still single threaded
>> by default)
>> - caching of global web.xml to improve start times
>> - changes to redeployment code to correctly redeploy on context.xml
>> changes
>>
>> If adding an empty web.xml works around this then at least there is
>> a simple
>> fix for anyone that hits this issue (whatever it is).
>>
>> Mark
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> Things certainly seem to stop moving forward.
> If I attach JProfiler, I see two threads waiting on the same lock,
> but I'm not familiar enough with the application to know if it's a
> 'real' deadlock.
I think you are right. When no realm is configured in server.xml and in
the context of a webapp, StandardEngine tries to set a default realm.
That is guarded by a lock, which is held by the main thread.
To work around this problem, you can define a simple realm inside the
context of your webapp or globally in server.xml.
To fix the problem we will have to look at the locking for setRealm in
ContainerBase.
Regards
Felix
>
> The webapp can be built simply by:
> rm -r webapps/*
> mkdir -p webapps/ROOT/WEB-INF
> echo "<html><head><title>Hello world</title></head><body>Hello
> World</body></html>" > webapps/ROOT/index.html
>
> Web.xml[1] included before(copied from the example shipped with
> tomcat)
> Thread dump attached at [2]. Hopefully my email client doesn’t'
> clobber it beyond recognition.
> Server.xml at [3]
>
> [1]
> <?xml version="1.0" encoding="ISO-8859-1"?>
> <web-app xmlns="http://java.sun.com/xml/ns/javaee"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
>
> http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
> version="3.0"
> metadata-complete="true">
>
> <display-name>Welcome to Tomcat</display-name>
> <description>
> Welcome to Tomcat
> </description>
>
> </web-app>
>
> [2]
> Full thread dump Java HotSpot(TM) Server VM (20.2-b06 mixed mode):
>
> "pool-2-thread-1" prio=10 tid=0x0a0a0800 nid=0x326a waiting for
> monitor entry [0x7bb94000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
>
> org.apache.catalina.core.ContainerBase.setRealm(ContainerBase.java:738)
> - waiting to lock <0x80aa2d70> (a
> org.apache.catalina.core.StandardEngine)
> at
>
> org.apache.catalina.core.StandardEngine.getRealm(StandardEngine.java:135)
> at
>
> org.apache.catalina.core.ContainerBase.getRealm(ContainerBase.java:723)
> at
>
> org.apache.catalina.core.ContainerBase.getRealm(ContainerBase.java:723)
> at
>
> org.apache.catalina.startup.ContextConfig.authenticatorConfig(ContextConfig.java:380)
> - locked <0xa32e0780> (a
> org.apache.catalina.startup.ContextConfig)
> at
>
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:839)
> - locked <0xa32e0780> (a
> org.apache.catalina.startup.ContextConfig)
> at
>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:302)
> at
>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
> at
>
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5148)
> - locked <0xa3265ec8> (a
> org.apache.catalina.core.StandardContext)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> - locked <0xa3265ec8> (a
> org.apache.catalina.core.StandardContext)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:866)
> at
>
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:842)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
> at
>
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
> at
>
> org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
> at
>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-1-thread-1" prio=10 tid=0x7bcde000 nid=0x3269 waiting on
> condition [0x7bbe5000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xa31e0918> (a
> java.util.concurrent.FutureTask$Sync)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
>
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1018)
> at
>
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473)
> at
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1412)
> at
>
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:312)
> at
>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
> at
>
> org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401)
> - locked <0x80aa2ef8> (a
> org.apache.catalina.core.StandardHost)
> at
>
> org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346)
> - locked <0x80aa2ef8> (a
> org.apache.catalina.core.StandardHost)
> at
>
> org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1117)
> - locked <0x80aa2ef8> (a
> org.apache.catalina.core.StandardHost)
> at
>
> org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:782)
> - locked <0x80aa2ef8> (a
> org.apache.catalina.core.StandardHost)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> - locked <0x80aa2ef8> (a
> org.apache.catalina.core.StandardHost)
> at
>
> org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1525)
> at
>
> org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1515)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "GC Daemon" daemon prio=10 tid=0x7bca1c00 nid=0x3267 in Object.wait()
> [0x7bd9c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x80a3bd78> (a sun.misc.GC$LatencyLock)
> at sun.misc.GC$Daemon.run(GC.java:100)
> - locked <0x80a3bd78> (a sun.misc.GC$LatencyLock)
>
> "Low Memory Detector" daemon prio=10 tid=0x09f83800 nid=0x3264
> runnable [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x09f82000 nid=0x3263 waiting
> on condition [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x09f76800 nid=0x3262 waiting
> on condition [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x09f75000 nid=0x3261 waiting
> on condition [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x09f63800 nid=0x3260 in Object.wait()
> [0x7c37e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x80a3c488> (a
> java.lang.ref.ReferenceQueue$Lock)
> at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x80a3c488> (a java.lang.ref.ReferenceQueue$Lock)
> at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x09f62000 nid=0x325f in
> Object.wait() [0x7c3cf000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x80a3bd70> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x80a3bd70> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x09ee2400 nid=0x325b waiting on condition
> [0xb7367000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xa3182228> (a
> java.util.concurrent.FutureTask$Sync)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
>
> org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1100)
> - locked <0x80aa2d70> (a
> org.apache.catalina.core.StandardEngine)
> at
>
> org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:291)
> - locked <0x80aa2d70> (a
> org.apache.catalina.core.StandardEngine)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> - locked <0x80aa2d70> (a
> org.apache.catalina.core.StandardEngine)
> at
>
> org.apache.catalina.core.StandardService.startInternal(StandardService.java:443)
> - locked <0x80aa2d70> (a
> org.apache.catalina.core.StandardEngine)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> - locked <0x80aa2150> (a
> org.apache.catalina.core.StandardService)
> at
>
> org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:727)
> - locked <0x80aa2140> (a [Lorg.apache.catalina.Service;)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> - locked <0x80a3d1f0> (a
> org.apache.catalina.core.StandardServer)
> at
> org.apache.catalina.startup.Catalina.start(Catalina.java:624)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
> at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)
>
> "VM Thread" prio=10 tid=0x09f5e000 nid=0x325e runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x09ee9800 nid=0x325c
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x09eeb000 nid=0x325d
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x09f85800 nid=0x3265 waiting
> on condition
>
> JNI global references: 1019
>
> Heap
> PSYoungGen total 15360K, used 11150K [0xa2f70000, 0xa4090000,
> 0xb4210000)
> eden space 13184K, 84% used [0xa2f70000,0xa3a538b0,0xa3c50000)
> from space 2176K, 0% used [0xa3c50000,0xa3c50000,0xa3e70000)
> to space 2176K, 0% used [0xa3e70000,0xa3e70000,0xa4090000)
> PSOldGen total 35136K, used 1757K [0x80a10000, 0x82c60000,
> 0xa2f70000)
> object space 35136K, 5% used [0x80a10000,0x80bc77a8,0x82c60000)
> PSPermGen total 16384K, used 9687K [0x7ca10000, 0x7da10000,
> 0x80a10000)
> object space 16384K, 59% used [0x7ca10000,0x7d385d90,0x7da10000)
>
>
> [3]
> <?xml version='1.0' encoding='utf-8'?>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed
> with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version
> 2.0
> (the "License"); you may not use this file except in compliance
> with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
> <!-- Note: A "Server" is not itself a "Container", so you may not
> define subcomponents such as "Valves" at this level.
> Documentation at /docs/config/server.html
> -->
> <Server port="8005" shutdown="SHUTDOWN">
> <!--Initialize Jasper prior to webapps are loaded. Documentation at
> /docs/jasper-howto.html -->
> <Listener className="org.apache.catalina.core.JasperListener" />
> <!-- Prevent memory leaks due to use of particular java/javax
> APIs-->
> <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> />
> <Listener
>
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
> <Listener
>
> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> />
>
> <!-- A "Service" is a collection of one or more "Connectors" that
> share
> a single "Container" Note: A "Service" is not itself a
> "Container",
> so you may not define subcomponents such as "Valves" at this
> level.
> Documentation at /docs/config/service.html
> -->
> <Service name="Catalina">
>
> <!-- A "Connector" represents an endpoint by which requests are
> received
> and responses are returned. Documentation at :
> Java HTTP Connector: /docs/config/http.html (blocking &
> non-blocking)
> Java AJP Connector: /docs/config/ajp.html
> APR (HTTP/AJP) Connector: /docs/apr.html
> Define a non-SSL HTTP/1.1 Connector on port 8080
> -->
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" server="Server"/>
> <!-- Define a SSL HTTP/1.1 Connector on port 8443
> This connector uses the JSSE configuration, when using APR,
> the
> connector should be using the OpenSSL style configuration
> described in the APR documentation -->
> <!--
> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
> maxThreads="150" scheme="https" secure="true"
> clientAuth="false" sslProtocol="TLS" server="Server"/>
> -->
>
>
> <!-- An Engine represents the entry point (within Catalina) that
> processes
> every request. The Engine implementation for Tomcat stand
> alone
> analyzes the HTTP headers included with the request, and
> passes them
> on to the appropriate Host (virtual host).
> Documentation at /docs/config/engine.html -->
>
> <!-- You should set jvmRoute to support load-balancing via AJP ie
> :
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
> -->
> <Engine name="Catalina" defaultHost="localhost">
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true">
>
> <!-- Access log processes all example.
> Documentation at: /docs/config/valve.html
> Note: The pattern used is equivalent to using
> pattern="common" -->
> <Valve className="org.apache.catalina.valves.AccessLogValve"
> directory="logs"
> prefix="localhost_access_log." suffix=".txt"
> pattern="%h %l %u %t "%r" %s %b" />
>
> </Host>
> </Engine>
> </Service>
> </Server>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat-7.0.23 startup deadlocks
Posted by "Zampani, Michael" <za...@amazon.com>.
> Which two threads are waiting on the same lock? I don't see that in your
> thread dump.
It's not obvious from the thread dump, but if I attach with jconsole, I see:
Name: main
State: WAITING on java.util.concurrent.FutureTask$Sync@6cc2a4
Name: pool-2-thread-1
State: BLOCKED on org.apache.catalina.core.StandardEngine@b41166 owned by: main
Total blocked: 1 Total waited: 0
The future main is waiting on is the start task for the child container.
-Michael
Re: Tomcat-7.0.23 startup deadlocks
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michael,
On 11/28/11 7:28 PM, Zampani, Michael wrote:
> Things certainly seem to stop moving forward. If I attach
> JProfiler, I see two threads waiting on the same lock, but I'm not
> familiar enough with the application to know if it's a 'real'
> deadlock.
Which two threads are waiting on the same lock? I don't see that in
your thread dump.
> "pool-2-thread-1" prio=10 tid=0x0a0a0800 nid=0x326a waiting for
> monitor entry [0x7bb94000] "pool-1-thread-1" prio=10 tid=0x7bcde000
> nid=0x3269 waiting on condition [0x7bbe5000] "C2 CompilerThread1"
> daemon prio=10 tid=0x09f82000 nid=0x3263 waiting on condition
> [0x00000000] "C2 CompilerThread0" daemon prio=10 tid=0x09f76800
> nid=0x3262 waiting on condition [0x00000000] "Signal Dispatcher"
> daemon prio=10 tid=0x09f75000 nid=0x3261 waiting on condition
> [0x00000000] "Finalizer" daemon prio=10 tid=0x09f63800 nid=0x3260
> in Object.wait() [0x7c37e000] "Reference Handler" daemon prio=10
> tid=0x09f62000 nid=0x325f in Object.wait() [0x7c3cf000] "main"
> prio=10 tid=0x09ee2400 nid=0x325b waiting on condition
> [0xb7367000]
??
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7UNcUACgkQ9CaO5/Lv0PCtvQCgpOAQa6bpL/+umvLAc6aEyIdt
KDQAoJP5sM9hXK/GHjz3oxn1/+9RvGQo
=0NF2
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat-7.0.23 startup deadlocks
Posted by "Zampani, Michael" <za...@amazon.com>.
> -----Original Message-----
> From: markt@apache.org [mailto:markt@apache.org]
> Sent: Monday, November 28, 2011 4:02 PM
> To: Tomcat Users List
> Subject: Re: Tomcat-7.0.23 startup deadlocks
>
> "Zampani, Michael" <za...@amazon.com> wrote:
>
> >Hi,
> >
> >I'm trying to upgrade an existing webapp installation from 7.0.22 ->
> >7.0.23.
> >
> >On startup, it appears to deadlock, with the threads stuck in
> >ContainerBase.setRealm HostConfig.deployDirectories
> >ContainerBase.startInternal
> >
> >server.xml and thread dump attached.
> >Note that I do not define any <Realm>s in server.xml
> >
> >The webapp is simple and contains only static html, js, css, and
> >images.
> >Copying the web.xml from the example webapp shipped with tomcat
> seems
> >to fix it, but this worked in previous versions of Tomcat.
> >Not sure if it's relevant, but it is the ROOT webapp.
> >
> >I've attached a sample WAR file to reproduce it. It should work fine
> >once deployed. Remove WEB-INF/web.xml and restart Tomcat to see it
> >fail.
> >
> >Any ideas?
> >
> >-Michael
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >For additional commands, e-mail: users-help@tomcat.apache.org
>
> Attachments are usually blocked. Better to paste in-line.
>
> Is there an actual deadlock or do things just seem to stop moving forwards?
>
> Possible causes:
> - threaded container starts (although they are still single threaded by default)
> - caching of global web.xml to improve start times
> - changes to redeployment code to correctly redeploy on context.xml
> changes
>
> If adding an empty web.xml works around this then at least there is a simple
> fix for anyone that hits this issue (whatever it is).
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
Things certainly seem to stop moving forward.
If I attach JProfiler, I see two threads waiting on the same lock, but I'm not familiar enough with the application to know if it's a 'real' deadlock.
The webapp can be built simply by:
rm -r webapps/*
mkdir -p webapps/ROOT/WEB-INF
echo "<html><head><title>Hello world</title></head><body>Hello World</body></html>" > webapps/ROOT/index.html
Web.xml[1] included before(copied from the example shipped with tomcat)
Thread dump attached at [2]. Hopefully my email client doesn’t' clobber it beyond recognition.
Server.xml at [3]
[1]
<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
version="3.0"
metadata-complete="true">
<display-name>Welcome to Tomcat</display-name>
<description>
Welcome to Tomcat
</description>
</web-app>
[2]
Full thread dump Java HotSpot(TM) Server VM (20.2-b06 mixed mode):
"pool-2-thread-1" prio=10 tid=0x0a0a0800 nid=0x326a waiting for monitor entry [0x7bb94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.catalina.core.ContainerBase.setRealm(ContainerBase.java:738)
- waiting to lock <0x80aa2d70> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardEngine.getRealm(StandardEngine.java:135)
at org.apache.catalina.core.ContainerBase.getRealm(ContainerBase.java:723)
at org.apache.catalina.core.ContainerBase.getRealm(ContainerBase.java:723)
at org.apache.catalina.startup.ContextConfig.authenticatorConfig(ContextConfig.java:380)
- locked <0xa32e0780> (a org.apache.catalina.startup.ContextConfig)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:839)
- locked <0xa32e0780> (a org.apache.catalina.startup.ContextConfig)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:302)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5148)
- locked <0xa3265ec8> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
- locked <0xa3265ec8> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:866)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:842)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"pool-1-thread-1" prio=10 tid=0x7bcde000 nid=0x3269 waiting on condition [0x7bbe5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xa31e0918> (a java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1018)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:473)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1412)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:312)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401)
- locked <0x80aa2ef8> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346)
- locked <0x80aa2ef8> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1117)
- locked <0x80aa2ef8> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:782)
- locked <0x80aa2ef8> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
- locked <0x80aa2ef8> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1525)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1515)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"GC Daemon" daemon prio=10 tid=0x7bca1c00 nid=0x3267 in Object.wait() [0x7bd9c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x80a3bd78> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x80a3bd78> (a sun.misc.GC$LatencyLock)
"Low Memory Detector" daemon prio=10 tid=0x09f83800 nid=0x3264 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x09f82000 nid=0x3263 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x09f76800 nid=0x3262 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x09f75000 nid=0x3261 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x09f63800 nid=0x3260 in Object.wait() [0x7c37e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x80a3c488> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x80a3c488> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x09f62000 nid=0x325f in Object.wait() [0x7c3cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x80a3bd70> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x80a3bd70> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x09ee2400 nid=0x325b waiting on condition [0xb7367000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xa3182228> (a java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1100)
- locked <0x80aa2d70> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:291)
- locked <0x80aa2d70> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
- locked <0x80aa2d70> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:443)
- locked <0x80aa2d70> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
- locked <0x80aa2150> (a org.apache.catalina.core.StandardService)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:727)
- locked <0x80aa2140> (a [Lorg.apache.catalina.Service;)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
- locked <0x80a3d1f0> (a org.apache.catalina.core.StandardServer)
at org.apache.catalina.startup.Catalina.start(Catalina.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)
"VM Thread" prio=10 tid=0x09f5e000 nid=0x325e runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x09ee9800 nid=0x325c runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x09eeb000 nid=0x325d runnable
"VM Periodic Task Thread" prio=10 tid=0x09f85800 nid=0x3265 waiting on condition
JNI global references: 1019
Heap
PSYoungGen total 15360K, used 11150K [0xa2f70000, 0xa4090000, 0xb4210000)
eden space 13184K, 84% used [0xa2f70000,0xa3a538b0,0xa3c50000)
from space 2176K, 0% used [0xa3c50000,0xa3c50000,0xa3e70000)
to space 2176K, 0% used [0xa3e70000,0xa3e70000,0xa4090000)
PSOldGen total 35136K, used 1757K [0x80a10000, 0x82c60000, 0xa2f70000)
object space 35136K, 5% used [0x80a10000,0x80bc77a8,0x82c60000)
PSPermGen total 16384K, used 9687K [0x7ca10000, 0x7da10000, 0x80a10000)
object space 16384K, 59% used [0x7ca10000,0x7d385d90,0x7da10000)
[3]
<?xml version='1.0' encoding='utf-8'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Note: A "Server" is not itself a "Container", so you may not
define subcomponents such as "Valves" at this level.
Documentation at /docs/config/server.html
-->
<Server port="8005" shutdown="SHUTDOWN">
<!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
<Listener className="org.apache.catalina.core.JasperListener" />
<!-- Prevent memory leaks due to use of particular java/javax APIs-->
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<!-- A "Service" is a collection of one or more "Connectors" that share
a single "Container" Note: A "Service" is not itself a "Container",
so you may not define subcomponents such as "Valves" at this level.
Documentation at /docs/config/service.html
-->
<Service name="Catalina">
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" server="Server"/>
<!-- Define a SSL HTTP/1.1 Connector on port 8443
This connector uses the JSSE configuration, when using APR, the
connector should be using the OpenSSL style configuration
described in the APR documentation -->
<!--
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS" server="Server"/>
-->
<!-- An Engine represents the entry point (within Catalina) that processes
every request. The Engine implementation for Tomcat stand alone
analyzes the HTTP headers included with the request, and passes them
on to the appropriate Host (virtual host).
Documentation at /docs/config/engine.html -->
<!-- You should set jvmRoute to support load-balancing via AJP ie :
<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
-->
<Engine name="Catalina" defaultHost="localhost">
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
</Engine>
</Service>
</Server>
Re: Tomcat-7.0.23 startup deadlocks
Posted by ma...@apache.org.
"Zampani, Michael" <za...@amazon.com> wrote:
>Hi,
>
>I'm trying to upgrade an existing webapp installation from 7.0.22 ->
>7.0.23.
>
>On startup, it appears to deadlock, with the threads stuck in
>ContainerBase.setRealm
>HostConfig.deployDirectories
>ContainerBase.startInternal
>
>server.xml and thread dump attached.
>Note that I do not define any <Realm>s in server.xml
>
>The webapp is simple and contains only static html, js, css, and
>images.
>Copying the web.xml from the example webapp shipped with tomcat seems
>to fix it, but this worked in previous versions of Tomcat.
>Not sure if it's relevant, but it is the ROOT webapp.
>
>I've attached a sample WAR file to reproduce it. It should work fine
>once deployed. Remove WEB-INF/web.xml and restart Tomcat to see it
>fail.
>
>Any ideas?
>
>-Michael
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
Attachments are usually blocked. Better to paste in-line.
Is there an actual deadlock or do things just seem to stop moving forwards?
Possible causes:
- threaded container starts (although they are still single threaded by default)
- caching of global web.xml to improve start times
- changes to redeployment code to correctly redeploy on context.xml changes
If adding an empty web.xml works around this then at least there is a simple fix for anyone that hits this issue (whatever it is).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org