You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by James Murphy <kl...@klazen.com> on 2009/11/02 19:32:10 UTC

Re: Tomcat hangs when reloading context if user is logged in

Chris,

Thanks for the response. I went ahead and got the info you requested.

Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> James,
>
> On 10/30/2009 9:05 AM, James Murphy wrote:
>   
>> Some background info:
>> OS: Windows Server 2003
>> Tomcat Version: 5.5
>> JDK Version: 1.6.0
>>
>> I've run into a very annoying problem while trying to enable
>> automatic context reload on our development server. It works well
>> when there is not a user logged on to the system (I guess that means
>> while there is session info on the server), but if someone is logged
>> in, then Tomcat fails to reload the context.
>>     
>
> I can confirm that Tomcat 5.5.26 both reloads and stops properly when
> multiple sessions are active. Persist-to-disk works as expected, and
> sessions are retained across the restart/reload.
>
> I suspect there is a problem with your specific webapp and/or setup.
>   
My thoughts too, I'm just not sure where to look.
>   
>> Another issue that may be related to this is that Tomcat will not
>> shutdown cleanly when someone is logged in (Tomcat hangs at "INFO:
>> Stopping service Catalina). Rather, we must close the console window
>> forcefully, then start the server back up to an error (IOException
>> while loading persisted sessions: java.io.EOFException)
>>     
>
> That's interesting: your session (de)serialization is failing because
> the file is incomplete. I suspect this is because the sessions are not
> correctly written to the file before application stop. Perhaps session
> serialization is the actual problem.
>
> Can you attempt to shut down Tomcat and then check:
> 1. A few thread dumps
> (http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F)
>   
I put them at the bottom of the message.
> 2. The CPU usage of your webapp and/or threads
>   
The CPU usage jumped to about 05% right before it hung, and then fell to 
00% and stayed there. Memory usage held steady at 37K.
> I'm wondering if serialization is either running-away (infinite loop,
> etc.) or stalling (waiting for access to a locked object) and it's never
> completing.
>
>   
Looks to me like it's waiting for a locked object, like you said, rather 
than hitting an infinite loop.
>> I used jstack to get a stack trace on the Tomcat process when it hung
>> on context reload. I won't post the details unless someone thinks
>> they might be helpful (Don't want to fill the email with too much
>> stuff).
>>     
>
> I think they might be. Go ahead and post the thread dump.
>   
See the bottom of the message.
>   
>> However, I know I read somewhere that Tomcat will hang on shutdown
>> if there is a non-daemon thread still running.
>>     
>
> While this is true, it shouldn't block a webapp from reloading (it will
> leave the old webapp in memory, so the old webapp won't really go away
> completely, but the new webapp should load in its place and handle
> requests).
>
>   
>> jstack reports three non-daemon threads: main, VM Thread, and VM
>> Periodic Task Thread. I don't think that's causing the problem, but
>> figured I'd better let you know anyway.
>>     
>
> Nope, those are normal.
>   

Here are a couple of thread dumps of tomcat, the first taken about 5 
seconds after the hang, and the second a couple of minutes later, while 
still hung.
Note that in the main methods, there is one stack frame with some XX's 
in it. I put those there to cover the name of a method which could 
identify our company. Nothing personal, I'm just not allowed to let that 
information leak out.
After seeing that stack frame, however, I looked into the method which 
it referenced. It appears to be sending a HTTP POST message to a servlet 
which we have running, which is supposed to log the user out when it 
recieves said message. Could the problem be that the servlet is shut 
down before the message reaches it?

--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------

2009-11-02 13:10:11
Full thread dump Java HotSpot(TM) Client VM (1.6.0-b105 mixed mode, 
sharing):

"http-80-6" daemon prio=6 tid=0x0322a400 nid=0xb10 in Object.wait() 
[0x03d5f000..0x03d5fb18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x181530c0> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x181530c0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-5" daemon prio=6 tid=0x02f89400 nid=0xe2c in Object.wait() 
[0x03cbf000..0x03cbfc18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x18153148> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x18153148> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-4" daemon prio=6 tid=0x02ed6400 nid=0xde8 in Object.wait() 
[0x03c6f000..0x03c6fc98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1808dcb0> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x1808dcb0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-3" daemon prio=6 tid=0x02ee3c00 nid=0xbb8 in Object.wait() 
[0x03c1f000..0x03c1fd18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x180603d0> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x180603d0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-2" daemon prio=6 tid=0x0324f400 nid=0x434 in Object.wait() 
[0x037bf000..0x037bfd98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1800e578> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x1800e578> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-1" daemon prio=6 tid=0x02f9bc00 nid=0xc7c in Object.wait() 
[0x0376f000..0x0376fa18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17febf38> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x17febf38> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"TP-Monitor" daemon prio=6 tid=0x031ebc00 nid=0x478 in Object.wait() 
[0x036af000..0x036afa98]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdc9b8> (a 
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at 
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
    - locked <0x17fdc9b8> (a 
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor4" daemon prio=6 tid=0x03219c00 nid=0x9b8 in Object.wait() 
[0x0365f000..0x0365fb18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fce7d0> (a org.apache.jk.common.ChannelSocket)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:301)
    - locked <0x17fce7d0> (a org.apache.jk.common.ChannelSocket)
    at 
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:661)
    at 
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:872)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor3" daemon prio=6 tid=0x03219400 nid=0x1e8 in Object.wait() 
[0x0360f000..0x0360fb98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdccc0> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
    - locked <0x17fdccc0> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor2" daemon prio=6 tid=0x031f8800 nid=0x5ec in Object.wait() 
[0x035bf000..0x035bfc18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdcd68> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
    - locked <0x17fdcd68> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor1" daemon prio=6 tid=0x0322b400 nid=0x9c0 in Object.wait() 
[0x0356f000..0x0356fc98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdce10> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
    - locked <0x17fdce10> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:619)

"http-80-Acceptor-0" daemon prio=6 tid=0x03227400 nid=0xd70 waiting on 
condition [0x0351f000..0x0351fd18]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)
    at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=6 tid=0x02a9d800 nid=0xd8 runnable 
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x02a98c00 nid=0xe34 waiting on 
condition [0x00000000..0x02d4f79c]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x02a97800 nid=0x760 waiting on 
condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x02a96c00 nid=0xf30 runnable 
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x02a8ec00 nid=0x140 in Object.wait() 
[0x02c5f000..0x02c5fc98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17d44ef8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x17d44ef8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x02a8dc00 nid=0x148 in 
Object.wait() [0x02c0f000..0x02c0fd18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17d44f88> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x17d44f88> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00297000 nid=0xb18 runnable [0x0088f000..0x0088fe58]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x1698bc98> (a java.io.BufferedInputStream)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:632)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
    at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
    - locked <0x16981bd0> (a sun.net.www.protocol.http.HttpURLConnection)
    at com.oreilly.servlet.HttpMessage.sendPostMessage(HttpMessage.java:141)
    at 
com.XXXXXXX.bean.XXXXSessionListener.valueUnbound(XXXXSessionListener.java:44)
    at 
org.apache.catalina.session.StandardSession.removeAttributeInternal(StandardSession.java:1654)
    at 
org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1507)
    at 
org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:959)
    at 
org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:517)
    - locked <0x17f5b960> (a java.util.concurrent.ConcurrentHashMap)
    at 
org.apache.catalina.session.StandardManager.unload(StandardManager.java:463)
    at 
org.apache.catalina.session.StandardManager.stop(StandardManager.java:667)
    at 
org.apache.catalina.core.StandardContext.stop(StandardContext.java:4573)
    - locked <0x17e55690> (a org.apache.catalina.core.StandardContext)
    at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
    - locked <0x17e445f8> (a org.apache.catalina.core.StandardHost)
    at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
    - locked <0x17e444f8> (a org.apache.catalina.core.StandardEngine)
    at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:448)
    at 
org.apache.catalina.core.StandardService.stop(StandardService.java:584)
    - locked <0x17e444f8> (a org.apache.catalina.core.StandardEngine)
    at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
    at org.apache.catalina.startup.Catalina.stop(Catalina.java:633)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:608)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

"VM Thread" prio=10 tid=0x02a84c00 nid=0xc28 runnable

"VM Periodic Task Thread" prio=10 tid=0x02a9f000 nid=0xbc0 waiting on 
condition

JNI global references: 1191

--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------

2009-11-02 13:16:44
Full thread dump Java HotSpot(TM) Client VM (1.6.0-b105 mixed mode, 
sharing):

"http-80-6" daemon prio=6 tid=0x0322a400 nid=0xb10 in Object.wait() 
[0x03d5f000..0x03d5fb18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x181530c0> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x181530c0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-5" daemon prio=6 tid=0x02f89400 nid=0xe2c in Object.wait() 
[0x03cbf000..0x03cbfc18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x18153148> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x18153148> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-4" daemon prio=6 tid=0x02ed6400 nid=0xde8 in Object.wait() 
[0x03c6f000..0x03c6fc98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1808dcb0> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x1808dcb0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-3" daemon prio=6 tid=0x02ee3c00 nid=0xbb8 in Object.wait() 
[0x03c1f000..0x03c1fd18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x180603d0> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x180603d0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-2" daemon prio=6 tid=0x0324f400 nid=0x434 in Object.wait() 
[0x037bf000..0x037bfd98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1800e578> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x1800e578> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"http-80-1" daemon prio=6 tid=0x02f9bc00 nid=0xc7c in Object.wait() 
[0x0376f000..0x0376fa18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17febf38> (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
    - locked <0x17febf38> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
    at java.lang.Thread.run(Thread.java:619)

"TP-Monitor" daemon prio=6 tid=0x031ebc00 nid=0x478 in Object.wait() 
[0x036af000..0x036afa98]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdc9b8> (a 
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at 
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
    - locked <0x17fdc9b8> (a 
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor4" daemon prio=6 tid=0x03219c00 nid=0x9b8 in Object.wait() 
[0x0365f000..0x0365fb18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fce7d0> (a org.apache.jk.common.ChannelSocket)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:301)
    - locked <0x17fce7d0> (a org.apache.jk.common.ChannelSocket)
    at 
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:661)
    at 
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:872)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor3" daemon prio=6 tid=0x03219400 nid=0x1e8 in Object.wait() 
[0x0360f000..0x0360fb98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdccc0> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
    - locked <0x17fdccc0> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor2" daemon prio=6 tid=0x031f8800 nid=0x5ec in Object.wait() 
[0x035bf000..0x035bfc18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdcd68> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
    - locked <0x17fdcd68> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:619)

"TP-Processor1" daemon prio=6 tid=0x0322b400 nid=0x9c0 in Object.wait() 
[0x0356f000..0x0356fc98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17fdce10> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:485)
    at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
    - locked <0x17fdce10> (a 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:619)

"http-80-Acceptor-0" daemon prio=6 tid=0x03227400 nid=0xd70 waiting on 
condition [0x0351f000..0x0351fd18]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)
    at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=6 tid=0x02a9d800 nid=0xd8 runnable 
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x02a98c00 nid=0xe34 waiting on 
condition [0x00000000..0x02d4f79c]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x02a97800 nid=0x760 waiting on 
condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x02a96c00 nid=0xf30 runnable 
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x02a8ec00 nid=0x140 in Object.wait() 
[0x02c5f000..0x02c5fc98]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17d44ef8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x17d44ef8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x02a8dc00 nid=0x148 in 
Object.wait() [0x02c0f000..0x02c0fd18]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x17d44f88> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x17d44f88> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00297000 nid=0xb18 runnable [0x0088f000..0x0088fe58]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x1698bc98> (a java.io.BufferedInputStream)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:632)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
    at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
    - locked <0x16981bd0> (a sun.net.www.protocol.http.HttpURLConnection)
    at com.oreilly.servlet.HttpMessage.sendPostMessage(HttpMessage.java:141)
    at 
com.XXXXXXX.bean.XXXXSessionListener.valueUnbound(XXXXSessionListener.java:44)
    at 
org.apache.catalina.session.StandardSession.removeAttributeInternal(StandardSession.java:1654)
    at 
org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1507)
    at 
org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:959)
    at 
org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:517)
    - locked <0x17f5b960> (a java.util.concurrent.ConcurrentHashMap)
    at 
org.apache.catalina.session.StandardManager.unload(StandardManager.java:463)
    at 
org.apache.catalina.session.StandardManager.stop(StandardManager.java:667)
    at 
org.apache.catalina.core.StandardContext.stop(StandardContext.java:4573)
    - locked <0x17e55690> (a org.apache.catalina.core.StandardContext)
    at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
    - locked <0x17e445f8> (a org.apache.catalina.core.StandardHost)
    at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
    - locked <0x17e444f8> (a org.apache.catalina.core.StandardEngine)
    at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:448)
    at 
org.apache.catalina.core.StandardService.stop(StandardService.java:584)
    - locked <0x17e444f8> (a org.apache.catalina.core.StandardEngine)
    at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
    at org.apache.catalina.startup.Catalina.stop(Catalina.java:633)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:608)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

"VM Thread" prio=10 tid=0x02a84c00 nid=0xc28 runnable

"VM Periodic Task Thread" prio=10 tid=0x02a9f000 nid=0xbc0 waiting on 
condition

JNI global references: 1191


--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------

> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkrrE6IACgkQ9CaO5/Lv0PCseACgvojWegC03QeaM0Twjd4Ai/Wo
> dxQAn0rqHwMGpu7rYj0Ihqsgndr854D4
> =aj9o
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat hangs when reloading context if user is logged in

Posted by James Murphy <kl...@klazen.com>.
Chris,

Thanks for all you help :) . We tried commenting out that method (just 
to see if it was the root of the problem) and Tomcat behaved like it is 
supposed to! Now we can work on implementing a work-around for the 
functionality.

Thanks!

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat hangs when reloading context if user is logged in

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

James,

On 11/2/2009 1:32 PM, James Murphy wrote:
>> Here are a couple of thread dumps of tomcat, the first taken about 5
>> seconds after the hang, and the second a couple of minutes later, while
>> still hung.

Okay.

>> Note that in the main methods, there is one stack frame with some XX's
>> in it. I put those there to cover the name of a method which could
>> identify our company. Nothing personal, I'm just not allowed to let that
>> information leak out.

No problem: this is always good practice. Thanks for mentioning it,
though :)

>> After seeing that stack frame, however, I looked into the method which
>> it referenced. It appears to be sending a HTTP POST message to a servlet
>> which we have running, which is supposed to log the user out when it
>> recieves said message. Could the problem be that the servlet is shut
>> down before the message reaches it?


Most of these all look the same. That is, they are just waiting for
requests to come in:

>> "http-80-6" daemon prio=6 tid=0x0322a400 nid=0xb10 in Object.wait()
>> [0x03d5f000..0x03d5fb18]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>    at java.lang.Object.wait(Native Method)
>>    - waiting on <0x181530c0> (a

[snip]

>> "main" prio=6 tid=0x00297000 nid=0xb18 runnable [0x0088f000..0x0088fe58]
>>   java.lang.Thread.State: RUNNABLE
>>    at java.net.SocketInputStream.socketRead0(Native Method)
>>    at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>    - locked <0x1698bc98> (a java.io.BufferedInputStream)
>>    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:632)
>>    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
>>    at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
> 
>>    - locked <0x16981bd0> (a sun.net.www.protocol.http.HttpURLConnection)
>>    at com.oreilly.servlet.HttpMessage.sendPostMessage(HttpMessage.java:141)
>>    at
>> com.XXXXXXX.bean.XXXXSessionListener.valueUnbound(XXXXSessionListener.java:44)
> 
>>    at
>> org.apache.catalina.session.StandardSession.removeAttributeInternal(StandardSession.java:1654)
> 
>>    at
>> org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1507)
> 
>>    at
>> org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:959)
> 
>>    at
>> org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:517)

It looks like the StandardManager is trying to persist your sessions,
which is proper behavior (unless you have disabled it). You have a
listener that, as you mentioned, is making an outgoing HTTP connection.

Is it connecting to another webapp, or to the same webapp? If you're
trying to connect to the same webapp, then it's possible that Tomcat has
paused all your <Connector>s, and they aren't accepting any connections.
In that case, you'll wait for a long time (and possibly repeatedly
attempt connections) before your POST fails.

If the above describes your situation, may I recommend that, instead of
making an HTTP POST back to your own webapp, you instead make a regular
Java method call to process this logout stuff? It shouldn't be that
tough to refactor your code to allow your logout logic to be called
either from an HTTP request or from an explicit method call.

Your second thread dump shows the "main" thread stuck in the same place:

>> "main" prio=6 tid=0x00297000 nid=0xb18 runnable [0x0088f000..0x0088fe58]
>>   java.lang.Thread.State: RUNNABLE
>>    at java.net.SocketInputStream.socketRead0(Native Method)
>>    at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>    - locked <0x1698bc98> (a java.io.BufferedInputStream)
>>    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:632)
>>    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
>>    at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
> 
>>    - locked <0x16981bd0> (a sun.net.www.protocol.http.HttpURLConnection)
>>    at com.oreilly.servlet.HttpMessage.sendPostMessage(HttpMessage.java:141)
>>    at
>> com.XXXXXXX.bean.XXXXSessionListener.valueUnbound(XXXXSessionListener.java:44)

Yup: still waiting.

Try calling your logout logic without using an HTTP POST and I think
you'll find that things start working more reasonably for you.

Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrvLKAACgkQ9CaO5/Lv0PCuZwCgrpXZFpDQKvWcHXqy3eqw/Ra2
sAIAnjnhp5a8eeK/vf3sBsTtCeJg3hCZ
=dILD
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org