You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alexander Ramos Jardim <al...@gmail.com> on 2008/11/25 22:15:26 UTC
Stuck threads on Weblogic
Hello guys,
I am getting some stuck threads on my application when it connects to Solr.
The stuck threads occur in an even time, in such a way that each 3 days the
app is online it hangs up the entire cluster.
I don't know if there's any direct relation to Solr, but I get the following
exception on some sparse connections the application does to Solr.
Is there any know bug about Solr writing wrong responses?
####<Nov 25, 2008 6:14:35 PM BRST> <Error> <HTTP> <localhost> <cluster0>
<[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (s
elf-tuning)'> <<WLS Kernel>> <> <> <1227644075142> <BEA-101083> <Connection
failure.
java.net.ProtocolException: Didn't meet stated Content-Length, wrote: '259'
bytes instead of stated: '258' bytes.
at weblogic.servlet.internal.
ServletOutputStreamImpl.ensureContentLength(ServletOutputStreamImpl.java:410)
at
weblogic.servlet.internal.ServletResponseImpl.ensureContentLength(ServletResponseImpl.java:1358)
at
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1400)
at
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1375)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
--
Alexander Ramos Jardim
Re: Stuck threads on Weblogic
Posted by Geoff Hopson <ge...@gmail.com>.
Dunno if it helps, but I have had some issues with Solr running
standalone using the BEA JRockit JVM with Solr hanging - when I
switched to Sun's JVM, hanging disappeared completely. Not
investigated the cause etc, so who knows the reason.
YMMV with swapping out JRockit with Sun when running WebLogic...
2008/11/25 Alexander Ramos Jardim <al...@gmail.com>:
> Hello guys,
>
> I am getting some stuck threads on my application when it connects to Solr.
> The stuck threads occur in an even time, in such a way that each 3 days the
> app is online it hangs up the entire cluster.
>
> I don't know if there's any direct relation to Solr, but I get the following
> exception on some sparse connections the application does to Solr.
>
> Is there any know bug about Solr writing wrong responses?
>
> ####<Nov 25, 2008 6:14:35 PM BRST> <Error> <HTTP> <localhost> <cluster0>
> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (s
> elf-tuning)'> <<WLS Kernel>> <> <> <1227644075142> <BEA-101083> <Connection
> failure.
> java.net.ProtocolException: Didn't meet stated Content-Length, wrote: '259'
> bytes instead of stated: '258' bytes.
> at weblogic.servlet.internal.
> ServletOutputStreamImpl.ensureContentLength(ServletOutputStreamImpl.java:410)
> at
> weblogic.servlet.internal.ServletResponseImpl.ensureContentLength(ServletResponseImpl.java:1358)
> at
> weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1400)
> at
> weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1375)
> at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
> at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
>
>
> --
> Alexander Ramos Jardim
>
--
Light travels faster than sound. This is why some people appear bright
until you hear them speak………
Mario Kart Wii: 2320 6406 5974
LinkedIn:http://www.linkedin.com/in/geoffhopson
Re: Stuck threads on Weblogic
Posted by Alexander Ramos Jardim <al...@gmail.com>.
Guys,
The stuck thread problem persists after I corrected my application. Now it
just takes more time to hang my application.
I identified some strange things at the thread dump, like a lot of pools
with just one connection.
Follow the thread dump:
===== FULL THREAD DUMP ===============
Wed Dec 17 12:45:39 2008
BEA JRockit(R)
R27.5.0-110-94909-1.5.0_14-20080204-1558-linux-ia32
"Main Thread" id=1 idx=0x4 tid=8163 prio=5 alive, in native,
waiting
-- Waiting for notification on:
weblogic/t3/srvr/T3Srvr@0x205ff080[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:747)
^-- Lock released while waiting:
weblogic/t3/srvr/T3Srvr@0x205ff080[fat lock]
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:396)
at weblogic/Server.main(Server.java:67)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(GC Main Thread)" id=2 idx=0x8 tid=8175 prio=5 alive, in
native, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0xc tid=8184 prio=5 alive,
in native, daemon
"(GC Worker Thread 2)" id=? idx=0x10 tid=8187 prio=5
alive, in native, daemon
"(GC Worker Thread 3)" id=? idx=0x14 tid=8193 prio=5
alive, in native, daemon
"(GC Worker Thread 4)" id=? idx=0x18 tid=8199 prio=5
alive, in native, daemon
"(GC Worker Thread 5)" id=? idx=0x1c tid=8204 prio=5
alive, in native, daemon
"(GC Worker Thread 6)" id=? idx=0x20 tid=8209 prio=5
alive, in native, daemon
"(GC Worker Thread 7)" id=? idx=0x24 tid=8213 prio=5
alive, in native, daemon
"(GC Worker Thread 8)" id=? idx=0x28 tid=8217 prio=5
alive, in native, daemon
"(Code Generation Thread 1)" id=3 idx=0x2c tid=8231 prio=5
alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=4 idx=0x30 tid=8242
prio=5 alive, in native, native_waiting, daemon
"(VM Periodic Task)" id=5 idx=0x34 tid=8244 prio=10 alive,
in native, daemon
"(Attach Listener)" id=6 idx=0x38 tid=8249 prio=5 alive,
in native, daemon
"Finalizer" id=7 idx=0x3c tid=8262 prio=8 alive, in
native, native_waiting, daemon
at
jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native
Method)
at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Reference Handler" id=8 idx=0x40 tid=8263 prio=10 alive,
in native, native_waiting, daemon
at
java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native
Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at
java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=9 idx=0x44 tid=8264 prio=5
alive, in native, daemon
"Timer-0" id=12 idx=0x48 tid=8267 prio=5 alive, in native,
waiting, daemon
-- Waiting for notification on:
java/util/TaskQueue@0x20217050[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting:
java/util/TaskQueue@0x20217050[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Timer-1" id=13 idx=0x4c tid=8359 prio=5 alive, in native,
waiting, daemon
-- Waiting for notification on:
java/util/TaskQueue@0x202170b8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting:
java/util/TaskQueue@0x202170b8[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '0' for queue:
'weblogic.kernel.Default (self-tuning)'" id=14 idx=0x50 tid=8360
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x1958b868[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x1958b868[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"weblogic.time.TimeEventGenerator" id=15 idx=0x54 tid=8361
prio=9 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/time/common/internal/TimeTable@0x192fdfd8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at
weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
^-- Lock released while waiting:
weblogic/time/common/internal/TimeTable@0x192fdfd8[fat lock]
at
weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"JMAPI event thread" id=16 idx=0x58 tid=8362 prio=5 alive,
in native, native_waiting, daemon
"weblogic.timers.TimerThread" id=17 idx=0x5c tid=8363
prio=9 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/timers/internal/TimerThread@0x20217228[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at
weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:265)
^-- Lock released while waiting:
weblogic/timers/internal/TimerThread@0x20217228[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '1' for queue:
'weblogic.kernel.Default (self-tuning)'" id=18 idx=0x60 tid=8364
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x1958b930[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x1958b930[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-7" id=21 idx=0x64 tid=8365 prio=5 alive, in
native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at
weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)
at
weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)
at
weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'"
id=22 idx=0x68 tid=8366 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.fatLockReacquire(Locks.java:1913)
at jrockit/vm/Locks.lockFat(Locks.java:1781)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'"
id=23 idx=0x6c tid=8367 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'"
id=24 idx=0x70 tid=8368 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'"
id=25 idx=0x74 tid=8369 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'"
id=26 idx=0x78 tid=8370 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 prio=5 alive, in native, daemon
at jrockit/ext/epoll/EPoll.epollWait0(IIII)I(Native Method)
at jrockit/ext/epoll/EPoll.epollWait(EPoll.java:103)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:156)
^-- Holding lock: java/lang/String@0x196625c0[fat lock]
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '6' for queue: 'weblogic.socket.Muxer'"
id=28 idx=0x80 tid=8372 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'"
id=29 idx=0x84 tid=8373 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"ExecuteThread: '8' for queue: 'weblogic.socket.Muxer'"
id=30 idx=0x88 tid=8374 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/String@0x196625c0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)
at jrockit/vm/Locks.lockFat(Locks.java:1776)
at
jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
at jrockit/vm/Locks.monitorEnter(Locks.java:2439)
at
weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153)
at
weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at
weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"VDE Transaction Processor Thread" id=33 idx=0x8c tid=8376
prio=2 alive, in native, waiting, daemon
-- Waiting for notification on:
com/octetstring/vde/backend/standard/TransactionProcessor@0x206c66d8[fat
lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
^-- Lock released while waiting:
com/octetstring/vde/backend/standard/TransactionProcessor@0x206c66d8[fat
lock]
at
com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"DoSManager" id=36 idx=0x94 tid=8378 prio=6 alive, in
native, sleeping, native_waiting, daemon
at java/lang/Thread.sleep(J)V(Native Method)
at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'" id=37 idx=0x98 tid=8411
prio=5 alive, in native, daemon
at
jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native
Method)
at
jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)
at
java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
at
java/net/SocketInputStream.read(SocketInputStream.java:129)
at
java/io/BufferedInputStream.fill(BufferedInputStream.java:218)
at
java/io/BufferedInputStream.read(BufferedInputStream.java:235)
^-- Holding lock:
java/io/BufferedInputStream@0xc812ed0[thin lock]
at
weblogic/net/http/MessageHeader.isHTTP(MessageHeader.java:220)
at
weblogic/net/http/MessageHeader.parseHeader(MessageHeader.java:143)
at
weblogic/net/http/HttpClient.parseHTTP(HttpClient.java:479)
^-- Holding lock:
weblogic/net/http/HttpClient@0xc812f48[thin lock]
at
weblogic/net/http/HttpURLConnection.getInputStream(HttpURLConnection.java:365)
^-- Holding lock:
weblogic/net/http/HttpURLConnection@0x15331728[thin lock]
at
weblogic/rjvm/http/HTTPClientJVMConnection.receiveAndDispatch(HTTPClientJVMConnection.java:404)
^-- Holding lock:
weblogic/rjvm/http/HTTPClientJVMConnection@0x4eb80c58[recursive]
at
weblogic/rjvm/http/HTTPClientJVMConnection.run(HTTPClientJVMConnection.java:295)
^-- Holding lock:
weblogic/rjvm/http/HTTPClientJVMConnection@0x4eb80c58[thin lock]
at
weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
at
weblogic/work/ExecuteThread.execute(ExecuteThread.java:200)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:172)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-11" id=38 idx=0x9c tid=8412 prio=5 alive, in
native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at
weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)
at
weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)
at
weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-12" id=39 idx=0xa0 tid=8919 prio=5 alive, in
native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at
weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)
at
weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)
at
weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-13" id=40 idx=0xa4 tid=8920 prio=5 alive, in
native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at
weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)
at
weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)
at
weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Thread-14" id=41 idx=0xa8 tid=8921 prio=5 alive, in
native, parked, daemon
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at
weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:562)
at
weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:610)
at
weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:378)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '3' for queue:
'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=8949
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x2f086fd8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x2f086fd8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '4' for queue:
'weblogic.kernel.Default (self-tuning)'" id=43 idx=0xb0 tid=8950
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x2f087ba8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x2f087ba8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-1-thread-1" id=44 idx=0xb4 tid=8963 prio=5 alive, in
native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-3-thread-1" id=45 idx=0xb8 tid=8972 prio=5 alive, in
native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-5-thread-1" id=46 idx=0xbc tid=8973 prio=5 alive, in
native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-7-thread-1" id=47 idx=0xc0 tid=8974 prio=5 alive, in
native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-9-thread-1" id=48 idx=0xc4 tid=8979 prio=5 alive, in
native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-11-thread-1" id=49 idx=0xc8 tid=8980 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-13-thread-1" id=50 idx=0xcc tid=8985 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-15-thread-1" id=51 idx=0xd0 tid=8987 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-17-thread-1" id=52 idx=0xd4 tid=8992 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-19-thread-1" id=53 idx=0xd8 tid=8997 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-21-thread-1" id=54 idx=0xdc tid=9004 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-23-thread-1" id=55 idx=0xe0 tid=9011 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-25-thread-1" id=56 idx=0xe4 tid=9017 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-27-thread-1" id=57 idx=0xe8 tid=9024 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-29-thread-1" id=58 idx=0xec tid=9026 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-31-thread-1" id=59 idx=0xf0 tid=9032 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-33-thread-1" id=60 idx=0xf4 tid=9033 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-35-thread-1" id=61 idx=0xf8 tid=9035 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-37-thread-1" id=62 idx=0xfc tid=9037 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-39-thread-1" id=63 idx=0x100 tid=9046 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-41-thread-1" id=64 idx=0x104 tid=9058 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-43-thread-1" id=65 idx=0x108 tid=9059 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-45-thread-1" id=66 idx=0x10c tid=9065 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-47-thread-1" id=67 idx=0x110 tid=9071 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-49-thread-1" id=68 idx=0x114 tid=9077 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-51-thread-1" id=69 idx=0x118 tid=9083 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-53-thread-1" id=70 idx=0x11c tid=9089 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"pool-55-thread-1" id=71 idx=0x120 tid=9095 prio=5 alive,
in native, parked
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2490)
at sun/misc/Unsafe.park(ZJ)V(Native Method)
at
java/util/concurrent/locks/LockSupport.park(LockSupport.java:118)
at
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"weblogic.cluster.MessageReceiver" id=72 idx=0x124
tid=9099 prio=5 alive, in native, daemon
at
java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native
Method)
^-- Holding lock:
java/net/PlainDatagramSocketImpl@0x4eb8ee38[recursive]
at
java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
^-- Holding lock:
java/net/PlainDatagramSocketImpl@0x4eb8ee38[thin lock]
at java/net/DatagramSocket.receive(DatagramSocket.java:712)
^-- Holding lock: java/net/DatagramPacket@0x7d95b500[thin
lock]
^-- Holding lock: java/net/MulticastSocket@0x4eb8ee88[thin
lock]
at
weblogic/cluster/MulticastFragmentSocket.receive(MulticastFragmentSocket.java:211)
at
weblogic/cluster/FragmentSocketWrapper.receive(FragmentSocketWrapper.java:98)
at
weblogic/cluster/MulticastManager.run(MulticastManager.java:405)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"DynamicListenThread[Default]" id=73 idx=0x128 tid=9121
prio=9 alive, in native, daemon
at
java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native
Method)
at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
^-- Holding lock: java/net/SocksSocketImpl@0x4eb8f8d8[thin
lock]
at java/net/ServerSocket.implAccept(ServerSocket.java:450)
at java/net/ServerSocket.accept(ServerSocket.java:421)
at
weblogic/socket/WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
at
weblogic/server/channels/DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:523)
at
weblogic/server/channels/DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:420)
at
weblogic/server/channels/DynamicListenThread.run(DynamicListenThread.java:166)
at java/lang/Thread.run(Thread.java:595)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '5' for queue:
'weblogic.kernel.Default (self-tuning)'" id=74 idx=0x12c tid=9130
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb8d858[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb8d858[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[STANDBY] ExecuteThread: '6' for queue:
'weblogic.kernel.Default (self-tuning)'" id=75 idx=0x130 tid=9133
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb8fd50[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb8fd50[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '7' for queue:
'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x134 tid=9180
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb908b8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb908b8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '8' for queue:
'weblogic.kernel.Default (self-tuning)'" id=77 idx=0x138 tid=9181
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb91988[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb91988[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '9' for queue:
'weblogic.kernel.Default (self-tuning)'" id=78 idx=0x13c tid=9182
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb92558[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb92558[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '10' for queue:
'weblogic.kernel.Default (self-tuning)'" id=79 idx=0x140 tid=9183
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb93138[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb93138[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '11' for queue:
'weblogic.kernel.Default (self-tuning)'" id=80 idx=0x144 tid=9184
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb95c98[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb95c98[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '12' for queue:
'weblogic.kernel.Default (self-tuning)'" id=81 idx=0x148 tid=9185
prio=5 alive, in native, daemon
at
bea/jmapi/DiagnosticCommandImpl.execute(Ljava/lang/String;Ljava/io/FileDescriptor;)V(Native
Method)
at
bea/jmapi/DiagnosticCommandImpl.execute(DiagnosticCommandImpl.java:50)
at
com/bea/jvm/DiagnosticCommand.execute(DiagnosticCommand.java:242)
at
com/bea/jvm/DiagnosticCommand$Command.execute(DiagnosticCommand.java:394)
at
bea/jmapi/ThreadSystemImpl.getThreadStackDump(ThreadSystemImpl.java:98)
at
weblogic/t3/srvr/JRockitRuntime.getThreadStackDump(JRockitRuntime.java:83)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
at
jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native
Method)
at
sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native
Method)
at
sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java/lang/reflect/Method.invoke(Method.java:585)
at
weblogic/management/jmx/modelmbean/WLSModelMBean.getAttribute(WLSModelMBean.java:531)
at
com/sun/jmx/mbeanserver/DynamicMetaDataImpl.getAttribute(DynamicMetaDataImpl.java:96)
at
com/sun/jmx/mbeanserver/MetaDataImpl.getAttribute(MetaDataImpl.java:181)
at
com/sun/jmx/interceptor/DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:638)
at
com/sun/jmx/mbeanserver/JmxMBeanServer.getAttribute(JmxMBeanServer.java:659)
at
weblogic/management/jmx/mbeanserver/WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:116)
at
weblogic/management/jmx/mbeanserver/WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:116)
at
weblogic/management/mbeanservers/internal/SecurityInterceptor.getAttribute(SecurityInterceptor.java:298)
at
weblogic/management/mbeanservers/internal/AuthenticatedSubjectInterceptor$5.run(AuthenticatedSubjectInterceptor.java:192)
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at
weblogic/management/mbeanservers/internal/AuthenticatedSubjectInterceptor.getAttribute(AuthenticatedSubjectInterceptor.java:190)
at
weblogic/management/jmx/mbeanserver/WLSMBeanServer.getAttribute(WLSMBeanServer.java:269)
at
javax/management/remote/rmi/RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1387)
at
javax/management/remote/rmi/RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
at
javax/management/remote/rmi/RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
at
jrockit/vm/AccessController.doPrivileged(AccessController.java:255)
at
javax/management/remote/rmi/RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1350)
at
javax/management/remote/rmi/RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:599)
at
javax/management/remote/rmi/RMIConnectionImpl_WLSkel.invoke(ILweblogic/rmi/spi/InboundRequest;Lweblogic/rmi/spi/OutboundResponse;Ljava/lang/Object;)Lweblogic/rmi/spi/OutboundResponse;(Unknown
Source)
at
weblogic/rmi/internal/BasicServerRef.invoke(BasicServerRef.java:589)
at
weblogic/rmi/internal/BasicServerRef$1.run(BasicServerRef.java:479)
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at
weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedExceptionAction;)Ljava/lang/Object;(Unknown
Source)
at
weblogic/rmi/internal/BasicServerRef.handleRequest(BasicServerRef.java:475)
at
weblogic/rmi/internal/BasicServerRef.access$300(BasicServerRef.java:59)
at
weblogic/rmi/internal/BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:1016)
at
weblogic/work/ExecuteThread.execute(ExecuteThread.java:200)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:172)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '13' for queue:
'weblogic.kernel.Default (self-tuning)'" id=82 idx=0x14c tid=9186
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb96878[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb96878[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '14' for queue:
'weblogic.kernel.Default (self-tuning)'" id=83 idx=0x150 tid=9187
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f819c38[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f819c38[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '15' for queue:
'weblogic.kernel.Default (self-tuning)'" id=84 idx=0x154 tid=9188
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f799c00[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f799c00[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '16' for queue:
'weblogic.kernel.Default (self-tuning)'" id=85 idx=0x158 tid=9189
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb97710[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb97710[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '17' for queue:
'weblogic.kernel.Default (self-tuning)'" id=86 idx=0x15c tid=9190
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f899440[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f899440[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '18' for queue:
'weblogic.kernel.Default (self-tuning)'" id=87 idx=0x160 tid=9192
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb98298[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb98298[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '19' for queue:
'weblogic.kernel.Default (self-tuning)'" id=88 idx=0x164 tid=9194
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f899fb8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f899fb8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '20' for queue:
'weblogic.kernel.Default (self-tuning)'" id=89 idx=0x168 tid=9204
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4eb98f30[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4eb98f30[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '21' for queue:
'weblogic.kernel.Default (self-tuning)'" id=90 idx=0x16c tid=9205
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f919150[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f919150[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '22' for queue:
'weblogic.kernel.Default (self-tuning)'" id=91 idx=0x170 tid=9206
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f919de0[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f919de0[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '23' for queue:
'weblogic.kernel.Default (self-tuning)'" id=92 idx=0x174 tid=9207
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4fd98cc8[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4fd98cc8[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '24' for queue:
'weblogic.kernel.Default (self-tuning)'" id=93 idx=0x178 tid=9208
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4fd99950[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4fd99950[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '25' for queue:
'weblogic.kernel.Default (self-tuning)'" id=94 idx=0x17c tid=9209
prio=5 alive, in native, waiting, daemon
-- Waiting for notification on:
weblogic/work/ExecuteThread@0x4f99a518[fat lock]
at
jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native
Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:474)
at
weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)
^-- Lock released while waiting:
weblogic/work/ExecuteThread@0x4f99a518[fat lock]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
Blocked lock chains
===================
Chain 2:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'"
id=23 idx=0x6c tid=8367 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Chain 3:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'"
id=24 idx=0x70 tid=8368 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Chain 4:
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'"
id=25 idx=0x74 tid=8369 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Chain 5:
"ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'"
id=26 idx=0x78 tid=8370 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Chain 6:
"ExecuteThread: '6' for queue: 'weblogic.socket.Muxer'"
id=28 idx=0x80 tid=8372 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Chain 7:
"ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'"
id=29 idx=0x84 tid=8373 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Chain 8:
"ExecuteThread: '8' for queue: 'weblogic.socket.Muxer'"
id=30 idx=0x88 tid=8374 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 in chain 1
Open lock chains
================
Chain 1:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'"
id=22 idx=0x68 tid=8366 waiting for java/lang/String@0x196625c0 held
by:
"ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'"
id=27 idx=0x7c tid=8371 (active)
===== END OF THREAD DUMP ===============
2008/11/28 Alexander Ramos Jardim <al...@gmail.com>
> Thanks for the answering,
>
> I isolated the problem and discovered it's not Solr's fault. It has
> something to do with the way I am manipulating the data, as the thread stays
> more than 10 minutes solely executing the same loop in some situations.
>
> 2008/11/28 Bill Au <bi...@gmail.com>
>
> Take a thread dump of the JVM next time it is stuck. That will tell you
>> where and why the threads are stuck.
>> Bill
>>
>> On Tue, Nov 25, 2008 at 4:15 PM, Alexander Ramos Jardim <
>> alexander.ramos.jardim@gmail.com> wrote:
>>
>> > Hello guys,
>> >
>> > I am getting some stuck threads on my application when it connects to
>> Solr.
>> > The stuck threads occur in an even time, in such a way that each 3 days
>> the
>> > app is online it hangs up the entire cluster.
>> >
>> > I don't know if there's any direct relation to Solr, but I get the
>> > following
>> > exception on some sparse connections the application does to Solr.
>> >
>> > Is there any know bug about Solr writing wrong responses?
>> >
>> > ####<Nov 25, 2008 6:14:35 PM BRST> <Error> <HTTP> <localhost> <cluster0>
>> > <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (s
>> > elf-tuning)'> <<WLS Kernel>> <> <> <1227644075142> <BEA-101083>
>> <Connection
>> > failure.
>> > java.net.ProtocolException: Didn't meet stated Content-Length, wrote:
>> '259'
>> > bytes instead of stated: '258' bytes.
>> > at weblogic.servlet.internal.
>> >
>> >
>> ServletOutputStreamImpl.ensureContentLength(ServletOutputStreamImpl.java:410)
>> > at
>> >
>> >
>> weblogic.servlet.internal.ServletResponseImpl.ensureContentLength(ServletResponseImpl.java:1358)
>> > at
>> >
>> >
>> weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1400)
>> > at
>> >
>> >
>> weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1375)
>> > at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
>> > at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
>> >
>> >
>> > --
>> > Alexander Ramos Jardim
>> >
>>
>
>
>
> --
> Alexander Ramos Jardim
>
--
Alexander Ramos Jardim
Re: Stuck threads on Weblogic
Posted by Alexander Ramos Jardim <al...@gmail.com>.
Thanks for the answering,
I isolated the problem and discovered it's not Solr's fault. It has
something to do with the way I am manipulating the data, as the thread stays
more than 10 minutes solely executing the same loop in some situations.
2008/11/28 Bill Au <bi...@gmail.com>
> Take a thread dump of the JVM next time it is stuck. That will tell you
> where and why the threads are stuck.
> Bill
>
> On Tue, Nov 25, 2008 at 4:15 PM, Alexander Ramos Jardim <
> alexander.ramos.jardim@gmail.com> wrote:
>
> > Hello guys,
> >
> > I am getting some stuck threads on my application when it connects to
> Solr.
> > The stuck threads occur in an even time, in such a way that each 3 days
> the
> > app is online it hangs up the entire cluster.
> >
> > I don't know if there's any direct relation to Solr, but I get the
> > following
> > exception on some sparse connections the application does to Solr.
> >
> > Is there any know bug about Solr writing wrong responses?
> >
> > ####<Nov 25, 2008 6:14:35 PM BRST> <Error> <HTTP> <localhost> <cluster0>
> > <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (s
> > elf-tuning)'> <<WLS Kernel>> <> <> <1227644075142> <BEA-101083>
> <Connection
> > failure.
> > java.net.ProtocolException: Didn't meet stated Content-Length, wrote:
> '259'
> > bytes instead of stated: '258' bytes.
> > at weblogic.servlet.internal.
> >
> >
> ServletOutputStreamImpl.ensureContentLength(ServletOutputStreamImpl.java:410)
> > at
> >
> >
> weblogic.servlet.internal.ServletResponseImpl.ensureContentLength(ServletResponseImpl.java:1358)
> > at
> >
> >
> weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1400)
> > at
> >
> >
> weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1375)
> > at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
> > at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
> >
> >
> > --
> > Alexander Ramos Jardim
> >
>
--
Alexander Ramos Jardim
Re: Stuck threads on Weblogic
Posted by Bill Au <bi...@gmail.com>.
Take a thread dump of the JVM next time it is stuck. That will tell you
where and why the threads are stuck.
Bill
On Tue, Nov 25, 2008 at 4:15 PM, Alexander Ramos Jardim <
alexander.ramos.jardim@gmail.com> wrote:
> Hello guys,
>
> I am getting some stuck threads on my application when it connects to Solr.
> The stuck threads occur in an even time, in such a way that each 3 days the
> app is online it hangs up the entire cluster.
>
> I don't know if there's any direct relation to Solr, but I get the
> following
> exception on some sparse connections the application does to Solr.
>
> Is there any know bug about Solr writing wrong responses?
>
> ####<Nov 25, 2008 6:14:35 PM BRST> <Error> <HTTP> <localhost> <cluster0>
> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (s
> elf-tuning)'> <<WLS Kernel>> <> <> <1227644075142> <BEA-101083> <Connection
> failure.
> java.net.ProtocolException: Didn't meet stated Content-Length, wrote: '259'
> bytes instead of stated: '258' bytes.
> at weblogic.servlet.internal.
>
> ServletOutputStreamImpl.ensureContentLength(ServletOutputStreamImpl.java:410)
> at
>
> weblogic.servlet.internal.ServletResponseImpl.ensureContentLength(ServletResponseImpl.java:1358)
> at
>
> weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1400)
> at
>
> weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1375)
> at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
> at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
>
>
> --
> Alexander Ramos Jardim
>