You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Na...@wellsfargo.com on 2006/04/06 08:35:50 UTC
High CPU with tomcat 5.5.7
Hello all,
I'm facing an issue of high CPU with tomcat process. Because of this,
I've to bounce server once in a week.
I am using tomcat 5.5.7 + AXIS 1.1 + jdk 1.4.2_07 on an Solaris box.
I've a sessionless webservice app deployed in this tomcat.
Even when the server is idle, tomcat shows high CPU value !
I took a snapshot of thread list for the tomcat process using prstat -L
-p command.
I can see 120 lwp threads, but only one of them takes high cpu usage.
Is this a GC thread or something else ? {I believe - Axis creates lot of
temp variables..this might result in GC at regular intervals}
There are so many threads at 0 ???
I guess, this might be the threads I get in core dump when I do kill -3.
I did kill -3 to see what each lwp thread is doing..
Out of 120, 70 were like -
"http-8082-Processor168" daemon prio=5 tid=0x00407348 nid=0x1065d in
Object.wait() [62d7f000..62d7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x801e2470> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool
.java:656)
- locked <0x801e2470> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
and 5 were -
"TP-Processor3" daemon prio=5 tid=0x00ffee70 nid=0x32 in Object.wait()
[6b77f000..6b77fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x78d29888> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool
.java:656)
- locked <0x78d29888> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
I've got no idea what this trace is about to know whether its causing
high CPU..
Even with high CPU, application shows normal response times and
behaviour.
I've attached core dump and snapshot of prstat command results.
I really appreciate your time and inputs for this issue.
Thanks!!
Naveen. <<coredump.txt>> <<prstat_snap.JPG>>
Re: High CPU with tomcat 5.5.7
Posted by Rainer Jung <to...@kippdata.de>.
The Solaris thread number shown in prstat -L kan be easily mapped to the
threads oin the Java thread dump. It's the nid in the Stack head. So
when prstat -L shows that for instance thread 62 takes all the CPU
(decimal number) you will find that thread in the java dump with
nid=0x3e (hex). I would also definitely update to a more stable 5.5 version.
Naveenkumar.Darapureddi@wellsfargo.com wrote:
> Hello all,
>
> I'm facing an issue of high CPU with tomcat process. Because of this,
> I've to bounce server once in a week.
> I am using tomcat 5.5.7 + AXIS 1.1 + jdk 1.4.2_07 on an Solaris box.
> I've a sessionless webservice app deployed in this tomcat.
>
> Even when the server is idle, tomcat shows high CPU value !
>
> I took a snapshot of thread list for the tomcat process using prstat -L
> -p command.
> I can see 120 lwp threads,* but only one of them takes high cpu usage*.
> Is this a GC thread or something else ? {I believe - Axis creates lot of
> temp variables..this might result in GC at regular intervals}
>
> There are so many threads at 0 ???
> I guess, this might be the threads I get in core dump when I do kill -3.
>
> I did kill -3 to see what each lwp thread is doing..
> *Out of 120, 70 were like -*
> "http-8082-Processor168" daemon prio=5 tid=0x00407348 nid=0x1065d in
> Object.wait() [62d7f000..62d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x801e2470> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
>
> - locked <0x801e2470> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
> *and 5 were -*
> "TP-Processor3" daemon prio=5 tid=0x00ffee70 nid=0x32 in Object.wait()
> [6b77f000..6b77fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d29888> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
>
> - locked <0x78d29888> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> I've got no idea what this trace is about to know whether its causing
> high CPU..
> Even with high CPU, application shows normal response times and behaviour.
>
> I've attached core dump and snapshot of prstat command results.
> I really appreciate your time and inputs for this issue.
>
> Thanks!!
> Naveen. <<coredump.txt>> <<prstat_snap.JPG>>
>
>
>
> ------------------------------------------------------------------------
>
> at java.lang.Thread.run(Thread.java:534)
>
> "Signal Dispatcher" daemon prio=10 tid=0x0034f8c0 nid=0x10 waiting on condition [0..0]
>
> "Surrogate Locker Thread (CMS)" daemon prio=5 tid=0x0034d1a0 nid=0xe waiting on condition [0..70b7feb0]
>
> "Finalizer" daemon prio=8 tid=0x0034bdb8 nid=0xd in Object.wait() [74e7f000..74e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x77c00010> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
> - locked <0x77c00010> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x0034ac48 nid=0xc in Object.wait() [74f7f000..74f7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x77c02680> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:429)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
> - locked <0x77c02680> (a java.lang.ref.Reference$Lock)
>
> "main" prio=5 tid=0x000374a0 nid=0x1 runnable [ffbee000..ffbeecfc]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> - locked <0x78622810> (a java.net.PlainSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> at java.net.ServerSocket.accept(ServerSocket.java:419)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:362)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:600)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:560)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:324)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)
>
> "VM Thread" prio=5 tid=0x00349ff8 nid=0xb runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x003540f8 nid=0x14 waiting on condition
> "Suspend Checker Thread" prio=10 tid=0x0034d450 nid=0xf runnable
> Full thread dump Java HotSpot(TM) Server VM (1.4.2_07-b05 mixed mode):
>
> "http-8082-Processor175" daemon prio=5 tid=0x00815f68 nid=0x10665 in Object.wait() [6247f000..6247fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f508260> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f508260> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor174" daemon prio=5 tid=0x0085a8c8 nid=0x10664 in Object.wait() [6257f000..6257fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x821b8388> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x821b8388> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor172" daemon prio=5 tid=0x00f2b7a8 nid=0x10661 in Object.wait() [6287f000..6287fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc558> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc558> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor171" daemon prio=5 tid=0x00f2b608 nid=0x10660 in Object.wait() [6297f000..6297fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7a444ee0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7a444ee0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor170" daemon prio=5 tid=0x00e0f848 nid=0x1065f in Object.wait() [62a7f000..62a7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7fa01ea0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7fa01ea0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor169" daemon prio=5 tid=0x00f8c838 nid=0x1065e in Object.wait() [62b7f000..62b7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7fdd7240> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7fdd7240> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor168" daemon prio=5 tid=0x00407348 nid=0x1065d in Object.wait() [62d7f000..62d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x801e2470> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x801e2470> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor167" daemon prio=5 tid=0x00cb2c30 nid=0x1065c in Object.wait() [6477f000..6477fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1de0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1de0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor166" daemon prio=5 tid=0x00970748 nid=0x1065b in Object.wait() [6497f000..6497fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc578> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc578> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor165" daemon prio=5 tid=0x01475b90 nid=0x1065a in Object.wait() [64d7f000..64d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1e00> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1e00> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor164" daemon prio=5 tid=0x014770d8 nid=0x10659 in Object.wait() [64e7f000..64e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc598> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc598> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor163" daemon prio=5 tid=0x00b528b8 nid=0x10658 in Object.wait() [64f7f000..64f7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1e20> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1e20> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor162" daemon prio=5 tid=0x00c369a8 nid=0x10657 in Object.wait() [6507f000..6507fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc5b8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc5b8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor161" daemon prio=5 tid=0x0159d5e0 nid=0x10656 in Object.wait() [6517f000..6517fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1e40> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1e40> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor160" daemon prio=5 tid=0x00b17b68 nid=0x10655 in Object.wait() [6537f000..6537fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc5d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc5d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor159" daemon prio=5 tid=0x00e7df70 nid=0x10654 in Object.wait() [6587f000..6587fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1e60> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1e60> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor158" daemon prio=5 tid=0x00bdf558 nid=0x10653 in Object.wait() [65e7f000..65e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc5f8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc5f8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor157" daemon prio=5 tid=0x00be1d98 nid=0x10652 in Object.wait() [6627f000..6627fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1e80> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1e80> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor156" daemon prio=5 tid=0x008cce38 nid=0x10651 in Object.wait() [6637f000..6637fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc618> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc618> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor155" daemon prio=5 tid=0x00a7afe8 nid=0x10650 in Object.wait() [66b7f000..66b7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1ea0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1ea0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor154" daemon prio=5 tid=0x00d841b0 nid=0x1064f in Object.wait() [66c7f000..66c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc638> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc638> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor153" daemon prio=5 tid=0x00ac31a0 nid=0x1064e in Object.wait() [66f7f000..66f7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b3d1ec0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b3d1ec0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor152" daemon prio=5 tid=0x0126e698 nid=0x1064d in Object.wait() [6707f000..6707fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7b8dc658> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7b8dc658> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor151" daemon prio=5 tid=0x00abfa98 nid=0x1064c in Object.wait() [6797f000..6797fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f9cb800> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f9cb800> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor150" daemon prio=5 tid=0x00a72ba0 nid=0x10647 in Object.wait() [67a7f000..67a7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdbea6228> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdbea6228> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor149" daemon prio=5 tid=0x00d84908 nid=0x10646 in Object.wait() [67d7f000..67d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xcda21cb8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xcda21cb8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor146" daemon prio=5 tid=0x00d7d048 nid=0x10643 in Object.wait() [6857f000..6857fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdc1bd1c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdc1bd1c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor144" daemon prio=5 tid=0x00ab7410 nid=0x10641 in Object.wait() [68a7f000..68a7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdd1c1690> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdd1c1690> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor142" daemon prio=5 tid=0x00332f38 nid=0x1063f in Object.wait() [68e7f000..68e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xda5230c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xda5230c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor139" daemon prio=5 tid=0x008b82c0 nid=0x1063c in Object.wait() [6c27f000..6c27fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xda629a38> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xda629a38> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor135" daemon prio=5 tid=0x00ab3508 nid=0x10638 in Object.wait() [6d37f000..6d37fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdbeb3338> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdbeb3338> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor134" daemon prio=5 tid=0x00483618 nid=0x10637 in Object.wait() [6d27f000..6d27fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdc0393e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdc0393e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor133" daemon prio=5 tid=0x0054e918 nid=0x10636 in Object.wait() [6c57f000..6c57fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xda6348f8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xda6348f8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor131" daemon prio=5 tid=0x00d7eb80 nid=0x10634 in Object.wait() [66e7f000..66e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdc047f80> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdc047f80> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor128" daemon prio=5 tid=0x0126ef70 nid=0x10631 in Object.wait() [62c7f000..62c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdc064710> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdc064710> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor127" daemon prio=5 tid=0x0074e710 nid=0x10630 in Object.wait() [6787f000..6787fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xdbee8330> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xdbee8330> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor122" daemon prio=5 tid=0x00cb0268 nid=0x104a6 in Object.wait() [62f7f000..62f7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xca15dcc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xca15dcc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor118" daemon prio=5 tid=0x0126da10 nid=0x104a2 in Object.wait() [6337f000..6337fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xbea22590> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xbea22590> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor115" daemon prio=5 tid=0x0126e9f0 nid=0x1049f in Object.wait() [6367f000..6367fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xbea225b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xbea225b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor114" daemon prio=5 tid=0x003fcff0 nid=0x1049e in Object.wait() [6377f000..6377fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xf4bb3088> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xf4bb3088> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor113" daemon prio=5 tid=0x0126e850 nid=0x1049d in Object.wait() [6387f000..6387fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc6d302a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xc6d302a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor111" daemon prio=5 tid=0x007c4200 nid=0x1049b in Object.wait() [63a7f000..63a7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xf4bb30a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xf4bb30a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor109" daemon prio=5 tid=0x008238b0 nid=0x10499 in Object.wait() [63c7f000..63c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc6d302c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xc6d302c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor108" daemon prio=5 tid=0x00823128 nid=0x10498 in Object.wait() [63d7f000..63d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xbea22610> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xbea22610> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor104" daemon prio=5 tid=0x0072da40 nid=0x10494 in Object.wait() [6417f000..6417fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xf4bb30c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xf4bb30c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor101" daemon prio=5 tid=0x00b17df0 nid=0x10491 in Object.wait() [6437f000..6437fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xf4bb30e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xf4bb30e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor100" daemon prio=5 tid=0x008b2ba0 nid=0x10488 in Object.wait() [6457f000..6457fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe616c3d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe616c3d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor95" daemon prio=5 tid=0x0085b368 nid=0x10483 in Object.wait() [64a7f000..64a7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe8565798> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe8565798> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor94" daemon prio=5 tid=0x0085b1c8 nid=0x10482 in Object.wait() [64b7f000..64b7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe95ad278> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe95ad278> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor93" daemon prio=5 tid=0x00ac5bd8 nid=0x10481 in Object.wait() [64c7f000..64c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe9a78c98> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe9a78c98> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor87" daemon prio=5 tid=0x00be11e8 nid=0x1047b in Object.wait() [6527f000..6527fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe85b2e30> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe85b2e30> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor83" daemon prio=5 tid=0x0086e210 nid=0x10477 in Object.wait() [6567f000..6567fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe89325c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe89325c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor78" daemon prio=5 tid=0x01423a38 nid=0x10472 in Object.wait() [65b7f000..65b7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe8d901c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe8d901c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor77" daemon prio=5 tid=0x007c6710 nid=0x10471 in Object.wait() [6757f000..6757fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe81886a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe81886a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor76" daemon prio=5 tid=0x003bcd88 nid=0x10470 in Object.wait() [65c7f000..65c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe8191808> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0xe8191808> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor73" daemon prio=5 tid=0x0044aa60 nid=0xe851 in Object.wait() [6607f000..6607fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79e76418> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x79e76418> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor72" daemon prio=5 tid=0x00d85138 nid=0xe850 in Object.wait() [6617f000..6617fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79e05fb8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x79e05fb8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor69" daemon prio=5 tid=0x00c2f828 nid=0xe84d in Object.wait() [6647f000..6647fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79f160f0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x79f160f0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor64" daemon prio=5 tid=0x00f56d60 nid=0xe848 in Object.wait() [6697f000..6697fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79b58eb8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x79b58eb8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor63" daemon prio=5 tid=0x0082d280 nid=0xe847 runnable [66a7f000..66a7fc28]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> - locked <0x77d86358> (a java.net.PlainSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> at java.net.ServerSocket.accept(ServerSocket.java:419)
> at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
> at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:406)
> at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor55" daemon prio=5 tid=0x0046d290 nid=0xe83f in Object.wait() [6767f000..6767fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79e0d9e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x79e0d9e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor50" daemon prio=5 tid=0x00ab5e28 nid=0x136c in Object.wait() [6777f000..6777fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f6626d0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f6626d0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor46" daemon prio=5 tid=0x00489718 nid=0x1368 in Object.wait() [67b7f000..67b7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f634768> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f634768> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor45" daemon prio=5 tid=0x00488c38 nid=0x1367 in Object.wait() [67c7f000..67c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f63f508> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f63f508> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor43" daemon prio=5 tid=0x0096d130 nid=0x1365 in Object.wait() [67e7f000..67e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f63fe48> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f63fe48> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor37" daemon prio=5 tid=0x00ab06a0 nid=0x135f in Object.wait() [6847f000..6847fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f64c230> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f64c230> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor34" daemon prio=5 tid=0x01577bb8 nid=0x135c in Object.wait() [6877f000..6877fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f64d680> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f64d680> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor33" daemon prio=5 tid=0x0096dce0 nid=0x135b in Object.wait() [6887f000..6887fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x7f63f588> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x7f63f588> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "PoolScavenger0" daemon prio=5 tid=0x0040b498 nid=0x64 in Object.wait() [6907f000..6907fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79689bf8> (a java.lang.Object)
> at com.ibm.mq.PoolScavenger.run(PoolScavenger.java:131)
> - locked <0x79689bf8> (a java.lang.Object)
> - locked <0x79683fb8> (a com.ibm.mq.PoolScavenger)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadPool" daemon prio=5 tid=0x00d6d580 nid=0x63 in Object.wait() [6917f000..6917fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79843da0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79843da0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79843da0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPool.run(ThreadPool.java:243)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(9)" daemon prio=5 tid=0x00e59a98 nid=0x62 in Object.wait() [6927f000..6927fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79845fc0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79845fc0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79845fc0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(8)" daemon prio=5 tid=0x00e59080 nid=0x61 in Object.wait() [6937f000..6937fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79842df0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79842df0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79842df0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(7)" daemon prio=5 tid=0x00d25fc0 nid=0x60 in Object.wait() [6947f000..6947fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79844b90> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79844b90> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79844b90> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(6)" daemon prio=5 tid=0x0126ac78 nid=0x5f in Object.wait() [6957f000..6957fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x798473b0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x798473b0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x798473b0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(5)" daemon prio=5 tid=0x00684e60 nid=0x5e in Object.wait() [6967f000..6967fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x798479c0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x798479c0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x798479c0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(4)" daemon prio=5 tid=0x00d2b088 nid=0x5d in Object.wait() [6977f000..6977fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79847fe0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79847fe0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79847fe0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(3)" daemon prio=5 tid=0x015f6dc0 nid=0x5c in Object.wait() [6987f000..6987fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79848300> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79848300> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79848300> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(2)" daemon prio=5 tid=0x008436d0 nid=0x5b in Object.wait() [6997f000..6997fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79843dc0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79843dc0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79843dc0> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(1)" daemon prio=5 tid=0x00e611d8 nid=0x5a in Object.wait() [69a7f000..69a7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79848340> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79848340> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79848340> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(0)" daemon prio=5 tid=0x00d2bcf0 nid=0x59 in Object.wait() [69b7f000..69b7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x79843e00> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:511)
> - locked <0x79843e00> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ObjectFIFO.remove(ObjectFIFO.java:321)
> - locked <0x79843e00> (a com.wfb.ivs.delayedImage.thread.ObjectFIFO)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.runWork(ThreadPoolWorker.java:303)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker.access$000(ThreadPoolWorker.java:95)
> at com.wfb.ivs.delayedImage.thread.ThreadPoolWorker$1.run(ThreadPoolWorker.java:209)
> at java.lang.Thread.run(Thread.java:534)
>
> "StmtThreadPool" daemon prio=5 tid=0x00d27e68 nid=0x4d in Object.wait() [69c7f000..69c7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797f7360> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797f7360> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797f7360> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.statement.persistence.StmtThreadPool.run(StmtThreadPool.java:217)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(9)" daemon prio=5 tid=0x00d274c8 nid=0x4c in Object.wait() [69d7f000..69d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797f74f8> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797f74f8> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797f74f8> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(8)" daemon prio=5 tid=0x00d26b28 nid=0x4b in Object.wait() [69e7f000..69e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797fa108> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797fa108> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797fa108> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(7)" daemon prio=5 tid=0x00897408 nid=0x4a in Object.wait() [69f7f000..69f7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797f7598> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797f7598> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797f7598> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(6)" daemon prio=5 tid=0x0114bd48 nid=0x49 in Object.wait() [6a07f000..6a07fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797f8038> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797f8038> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797f8038> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(5)" daemon prio=5 tid=0x00a22878 nid=0x48 in Object.wait() [6a17f000..6a17fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797fa510> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797fa510> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797fa510> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(4)" daemon prio=5 tid=0x014dce90 nid=0x47 in Object.wait() [6a27f000..6a27fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797fa8c0> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797fa8c0> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797fa8c0> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(3)" daemon prio=5 tid=0x00685e88 nid=0x46 in Object.wait() [6a37f000..6a37fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797faae0> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797faae0> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797faae0> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(2)" daemon prio=5 tid=0x00a23748 nid=0x45 in Object.wait() [6a47f000..6a47fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797f7380> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797f7380> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797f7380> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(1)" daemon prio=5 tid=0x006c0260 nid=0x44 in Object.wait() [6a57f000..6a57fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797fab20> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797fab20> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797fab20> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "ThreadWorker(0)" daemon prio=5 tid=0x015a6fa8 nid=0x43 in Object.wait() [6a67f000..6a67fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x797f75d8> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at java.lang.Object.wait(Object.java:429)
> at com.wfb.ivs.image.persistence.ObjectFIFO.waitWhileEmpty(ObjectFIFO.java:581)
> - locked <0x797f75d8> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ObjectFIFO.remove(ObjectFIFO.java:359)
> - locked <0x797f75d8> (a com.wfb.ivs.image.persistence.ObjectFIFO)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.runWork(ThreadPoolWorker.java:307)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker.access$000(ThreadPoolWorker.java:87)
> at com.wfb.ivs.image.persistence.ThreadPoolWorker$1.run(ThreadPoolWorker.java:203)
> at java.lang.Thread.run(Thread.java:534)
>
> "Thread-33" daemon prio=5 tid=0x00698d00 nid=0x38 waiting on condition [6a77f000..6a77fc28]
> at java.lang.Thread.sleep(Native Method)
> at com.wfb.ivs.metadata.persistence.dao.PSORResponseAdaptor.getMessage(PSORResponseAdaptor.java:474)
> at com.wfb.ivs.metadata.persistence.dao.PSORResponseAdaptor.startListener(PSORResponseAdaptor.java:387)
> at com.wfb.ivs.metadata.persistence.dao.PSORResponseAdaptor.run(PSORResponseAdaptor.java:300)
>
> "TP-Monitor" daemon prio=5 tid=0x00f10978 nid=0x34 in Object.wait() [6b57f000..6b57fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d2dd10> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
> - locked <0x78d2dd10> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "TP-Processor4" daemon prio=5 tid=0x00f101b8 nid=0x33 runnable [6b67f000..6b67fc28]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> - locked <0x78622770> (a java.net.PlainSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> at java.net.ServerSocket.accept(ServerSocket.java:419)
> at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:295)
> at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:638)
> at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:849)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> at java.lang.Thread.run(Thread.java:534)
>
> "TP-Processor3" daemon prio=5 tid=0x00ffee70 nid=0x32 in Object.wait() [6b77f000..6b77fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d29888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d29888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "TP-Processor2" daemon prio=5 tid=0x00ffecd0 nid=0x31 in Object.wait() [6b87f000..6b87fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d1b888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d1b888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "TP-Processor1" daemon prio=5 tid=0x00b457a0 nid=0x30 in Object.wait() [6b97f000..6b97fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d20d58> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d20d58> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Monitor" prio=5 tid=0x00d36ca0 nid=0x2f in Object.wait() [6ba7f000..6ba7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d2e640> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
> - locked <0x78d2e640> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor24" daemon prio=5 tid=0x00c359b8 nid=0x2d in Object.wait() [6bc7f000..6bc7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d31708> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d31708> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor22" daemon prio=5 tid=0x00c341d8 nid=0x2b in Object.wait() [6be7f000..6be7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d310c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d310c8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor19" daemon prio=5 tid=0x008cd1f8 nid=0x28 in Object.wait() [6c17f000..6c17fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d00868> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d00868> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor17" daemon prio=5 tid=0x0074fec0 nid=0x26 in Object.wait() [6c37f000..6c37fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d31728> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d31728> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor13" daemon prio=5 tid=0x0159c5f0 nid=0x22 in Object.wait() [6c77f000..6c77fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d31788> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d31788> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor8" daemon prio=5 tid=0x00916498 nid=0x1d in Object.wait() [6cc7f000..6cc7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d20db8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d20db8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor6" daemon prio=5 tid=0x00ba82f8 nid=0x1b in Object.wait() [6ce7f000..6ce7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d31108> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d31108> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "http-8082-Processor3" daemon prio=5 tid=0x00fe1bf0 nid=0x18 in Object.wait() [6d17f000..6d17fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d20dd8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d20dd8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=5 tid=0x016bbb60 nid=0x15 waiting on condition [6d47f000..6d47fc28]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1539)
> at java.lang.Thread.run(Thread.java:534)
>
> "Signal Dispatcher" daemon prio=10 tid=0x0034f8c0 nid=0x10 waiting on condition [0..0]
>
> "Surrogate Locker Thread (CMS)" daemon prio=5 tid=0x0034d1a0 nid=0xe waiting on condition [0..70b7feb0]
>
> "Finalizer" daemon prio=8 tid=0x0034bdb8 nid=0xd in Object.wait() [74e7f000..74e7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x77c00010> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
> - locked <0x77c00010> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x0034ac48 nid=0xc in Object.wait() [74f7f000..74f7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x77c02680> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:429)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
> - locked <0x77c02680> (a java.lang.ref.Reference$Lock)
>
> "main" prio=5 tid=0x000374a0 nid=0x1 runnable [ffbee000..ffbeecfc]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> - locked <0x78622810> (a java.net.PlainSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:448)
> at java.net.ServerSocket.accept(ServerSocket.java:419)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:362)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:600)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:560)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:324)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)
>
> "VM Thread" prio=5 tid=0x00349ff8 nid=0xb runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x003540f8 nid=0x14 waiting on condition
> "Suspend Checker Thread" prio=10 tid=0x0034d450 nid=0xf runnable
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU with tomcat 5.5.7
Posted by Leon Rosenberg <ro...@googlemail.com>.
Hi,
I don't see anything strange in your thread dump. The threads you
mentioned are the http-worker threads which are waiting for new
requests to server, no reason to be worried. I don't now what your
image-processing threads do (
com.wfb.ivs.delayedImage.thread.ObjectFIFO), is there an underlying
native lib, which could be eating cpu?
regards
Leon
On 4/6/06, Naveenkumar.Darapureddi@wellsfargo.com
<Na...@wellsfargo.com> wrote:
>
>
> Hello all,
>
> I'm facing an issue of high CPU with tomcat process. Because of this, I've
> to bounce server once in a week.
> I am using tomcat 5.5.7 + AXIS 1.1 + jdk 1.4.2_07 on an Solaris box.
> I've a sessionless webservice app deployed in this tomcat.
>
> Even when the server is idle, tomcat shows high CPU value !
>
> I took a snapshot of thread list for the tomcat process using prstat -L -p
> command.
> I can see 120 lwp threads, but only one of them takes high cpu usage.
> Is this a GC thread or something else ? {I believe - Axis creates lot of
> temp variables..this might result in GC at regular intervals}
>
> There are so many threads at 0 ???
> I guess, this might be the threads I get in core dump when I do kill -3.
>
> I did kill -3 to see what each lwp thread is doing..
> Out of 120, 70 were like -
> "http-8082-Processor168" daemon prio=5 tid=0x00407348 nid=0x1065d in
> Object.wait() [62d7f000..62d7fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x801e2470> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x801e2470> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
> and 5 were -
> "TP-Processor3" daemon prio=5 tid=0x00ffee70 nid=0x32 in Object.wait()
> [6b77f000..6b77fc28]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x78d29888> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Object.wait(Object.java:429)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
> - locked <0x78d29888> (a
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
> at java.lang.Thread.run(Thread.java:534)
>
> I've got no idea what this trace is about to know whether its causing high
> CPU..
> Even with high CPU, application shows normal response times and behaviour.
>
> I've attached core dump and snapshot of prstat command results.
> I really appreciate your time and inputs for this issue.
>
> Thanks!!
> Naveen. <<coredump.txt>> <<prstat_snap.JPG>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org