You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "irodionov (JIRA)" <ji...@apache.org> on 2016/05/16 10:53:12 UTC
[jira] [Commented] (AMQ-5815) Frequency IOException cast dead lock
while using kahadb in high concurrency environment
[ https://issues.apache.org/jira/browse/AMQ-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284379#comment-15284379 ]
irodionov commented on AMQ-5815:
--------------------------------
We had similar issue with KahaDB. AMQ version is 5.12.1.
Sometimes IOException occurs (we use windows net drive and we had slow disk on our VM and a lot of persistent messages) which blocks the broker.
KahaDB property enableIndexWriteAsync is setted to TRUE. In case of IOException "KahaDB Page Writer" thread dies so it blocks ""ActiveMQ Journal Checkpoint Worker" thread which waits for CountDownLatch (in PageFile.flush) which will never countDown.
Full thread dump:
{code}
2016-05-12 06:56:44
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode):
"ActiveMQ BrokerService[dcrd] Task-135" #464 daemon prio=5 os_prio=0 tid=0x0000000020a6e000 nid=0xfe8 waiting on condition [0x000000004e5ae000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081dc1450> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"ActiveMQ VMTransport: vm://localhost#1-20" #428 daemon prio=5 os_prio=0 tid=0x0000000020a66800 nid=0xce4 waiting on condition [0x000000005314d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081e52c50> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:994)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:980)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:482)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAsyncMessage(KahaDBStore.java:417)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.removeAsyncMessage(KahaDBTransactionStore.java:506)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.removeAsyncMessage(KahaDBTransactionStore.java:184)
at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:915)
at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1719)
at org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:55)
at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:234)
- locked <0x0000000081e60f08> (a java.lang.Object)
at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:526)
at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:484)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)
at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:276)
at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:97)
at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:555)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:338)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:270)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000def1aa98> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"ActiveMQ InactivityMonitor WriteCheckTimer" #427 daemon prio=5 os_prio=0 tid=0x0000000020a6f800 nid=0x1918 in Object.wait() [0x0000000052a1f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000da73e8d8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"ActiveMQ Transport: ssl:///127.0.0.1:60001" #425 daemon prio=5 os_prio=0 tid=0x0000000020a6d000 nid=0x9b4 waiting on condition [0x000000002016e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081e52c50> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:994)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:980)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:434)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.asyncAddQueueMessage(KahaDBTransactionStore.java:405)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.asyncAddQueueMessage(KahaDBTransactionStore.java:174)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:831)
at org.apache.activemq.broker.region.Queue.send(Queue.java:724)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:503)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:546)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:338)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:102)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000000081e5a148> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"ActiveMQ InactivityMonitor ReadCheckTimer" #424 daemon prio=5 os_prio=0 tid=0x0000000020a68800 nid=0xab0 in Object.wait() [0x000000001c60e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000d930c2c0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"ActiveMQ InactivityMonitor Worker" #382 daemon prio=5 os_prio=0 tid=0x000000001dae3800 nid=0xe70 waiting on condition [0x0000000052b1e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000825203f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"105542689@qtp-1219161283-10" #181 prio=5 os_prio=0 tid=0x000000001dae1000 nid=0x1720 in Object.wait() [0x000000004fb4e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x0000000082aea590> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"MultiThreadedHttpConnectionManager cleanup" #180 daemon prio=5 os_prio=0 tid=0x000000001dadd800 nid=0x221c in Object.wait() [0x000000004fa4e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000082af2600> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
Locked ownable synchronizers:
- None
"ActiveMQ BrokerService.worker.1" #81 daemon prio=5 os_prio=0 tid=0x0000000020a6a000 nid=0x1308 waiting on condition [0x000000002006e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081e561f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Connection(4)-127.0.0.1" #70 daemon prio=5 os_prio=0 tid=0x0000000020a6b800 nid=0x1074 runnable [0x000000004f16e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x0000000082178a50> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:722)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$256(TCPTransport.java:683)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/711930559.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000000081e756a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"DestroyJavaVM" #68 prio=5 os_prio=0 tid=0x0000000020a69800 nid=0x111c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Timer-3" #67 daemon prio=5 os_prio=0 tid=0x0000000020a68000 nid=0x924 in Object.wait() [0x000000004f06f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000081f48498> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Zabbix-agent" #66 daemon prio=5 os_prio=0 tid=0x0000000020a65800 nid=0xad4 runnable [0x000000004ef6e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x0000000081e8f2c0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.kjkoster.zapcat.zabbix.ZabbixAgent.run(ZabbixAgent.java:171)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Timer-2" #65 daemon prio=5 os_prio=0 tid=0x0000000021020000 nid=0xd94 in Object.wait() [0x000000004e4af000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000081e95b30> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"pool-8-thread-1" #58 prio=5 os_prio=0 tid=0x0000000021019800 nid=0x23b4 waiting on condition [0x000000004e1ae000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081e66cd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-19" #57 prio=5 os_prio=0 tid=0x000000002101c800 nid=0xf7c runnable [0x000000004e0af000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x0000000081e4bb78> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at com.sun.jmx.remote.socket.SocketConnectionServer.accept(SocketConnectionServer.java:173)
at com.sun.jmx.remote.generic.SynchroMessageConnectionServerImpl.accept(SynchroMessageConnectionServerImpl.java:47)
at javax.management.remote.generic.GenericConnectorServer$Receiver.run(GenericConnectorServer.java:340)
Locked ownable synchronizers:
- None
"Timer-1" #56 daemon prio=5 os_prio=0 tid=0x000000001d157800 nid=0xccc in Object.wait() [0x000000004dfaf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x0000000081e17ce8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"ActiveMQ Transport Server: ssl://0.0.0.0:13701?needClientAuth=true&keepAlive=true" #55 daemon prio=9 os_prio=2 tid=0x000000001d156800 nid=0x1b08 runnable [0x000000004dcae000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x0000000081e204f8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"ActiveMQ Transport Server Thread Handler: ssl://0.0.0.0:13701?needClientAuth=true&keepAlive=true" #54 daemon prio=8 os_prio=1 tid=0x000000001d14e800 nid=0x2114 waiting on condition [0x000000004dbae000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081e3dcf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:427)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"ActiveMQ Data File Writer" #51 daemon prio=10 os_prio=2 tid=0x000000001d159800 nid=0xf94 in Object.wait() [0x000000004d8af000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:306)
- locked <0x0000000081e4d9c0> (a java.lang.Object)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:193)
Locked ownable synchronizers:
- None
"ActiveMQ Journal Checkpoint Worker" #50 daemon prio=5 os_prio=0 tid=0x000000001d158000 nid=0x3ec waiting on condition [0x000000004d7ae000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f847d368> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:521)
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1563)
at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1535)
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1532)
at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:951)
at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:359)
Locked ownable synchronizers:
- <0x0000000081e52c50> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0000000081e52cc8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
"KahaDB Scheduler" #48 daemon prio=5 os_prio=0 tid=0x000000001d159000 nid=0x149c in Object.wait() [0x000000004d5af000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000081e4e288> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"ActiveMQ Broker[dcrd] Scheduler" #47 daemon prio=5 os_prio=0 tid=0x000000001d154800 nid=0x22b4 in Object.wait() [0x000000004d4af000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000081e54e38> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.StandardQueryCache Expiry Thread" #46 daemon prio=1 os_prio=-2 tid=0x000000001d155000 nid=0xa18 waiting on condition [0x000000004d1af000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.expiryThreadMain(DiskStore.java:831)
at net.sf.ehcache.store.DiskStore.access$700(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$ExpiryThread.run(DiskStore.java:1008)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.StandardQueryCache Spool Thread" #45 daemon prio=2 os_prio=-2 tid=0x000000001d150800 nid=0x158c waiting on condition [0x000000004d0af000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.spoolThreadMain(DiskStore.java:556)
at net.sf.ehcache.store.DiskStore.access$600(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$SpoolThread.run(DiskStore.java:990)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.UpdateTimestampsCache Expiry Thread" #44 daemon prio=1 os_prio=-2 tid=0x000000001d153800 nid=0x1a2c waiting on condition [0x000000004bd1e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.expiryThreadMain(DiskStore.java:831)
at net.sf.ehcache.store.DiskStore.access$700(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$ExpiryThread.run(DiskStore.java:1008)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" #43 daemon prio=2 os_prio=-2 tid=0x000000001d14c000 nid=0x1394 waiting on condition [0x000000004b92f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.spoolThreadMain(DiskStore.java:556)
at net.sf.ehcache.store.DiskStore.access$600(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$SpoolThread.run(DiskStore.java:990)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.StandardQueryCache Expiry Thread" #42 daemon prio=1 os_prio=-2 tid=0x000000001d153000 nid=0xb60 waiting on condition [0x000000004a3df000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.expiryThreadMain(DiskStore.java:831)
at net.sf.ehcache.store.DiskStore.access$700(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$ExpiryThread.run(DiskStore.java:1008)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.StandardQueryCache Spool Thread" #41 daemon prio=2 os_prio=-2 tid=0x000000001d14f000 nid=0x1be0 waiting on condition [0x000000004a2df000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.spoolThreadMain(DiskStore.java:556)
at net.sf.ehcache.store.DiskStore.access$600(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$SpoolThread.run(DiskStore.java:990)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.UpdateTimestampsCache Expiry Thread" #40 daemon prio=1 os_prio=-2 tid=0x000000001d152000 nid=0x2328 waiting on condition [0x000000004a1df000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.expiryThreadMain(DiskStore.java:831)
at net.sf.ehcache.store.DiskStore.access$700(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$ExpiryThread.run(DiskStore.java:1008)
Locked ownable synchronizers:
- None
"Store org.hibernate.cache.UpdateTimestampsCache Spool Thread" #39 daemon prio=2 os_prio=-2 tid=0x000000001d151800 nid=0x1180 waiting on condition [0x0000000049a6f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.spoolThreadMain(DiskStore.java:556)
at net.sf.ehcache.store.DiskStore.access$600(DiskStore.java:59)
at net.sf.ehcache.store.DiskStore$SpoolThread.run(DiskStore.java:990)
Locked ownable synchronizers:
- None
"pool-1-thread-1" #38 prio=5 os_prio=0 tid=0x000000001d14a800 nid=0x1ab4 waiting on condition [0x000000001a90f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000008045d550> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Timer-0" #37 daemon prio=5 os_prio=0 tid=0x000000001d14b800 nid=0x2190 in Object.wait() [0x000000001a60f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000800f1b68> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"2012993836@qtp-1219161283-9 - Acceptor0 SelectChannelConnector@0.0.0.0:13801" #35 prio=5 os_prio=0 tid=0x000000001b630000 nid=0x1138 runnable [0x000000001ca0e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000f708c4a8> (a sun.nio.ch.Util$2)
- locked <0x00000000f708c498> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f703e890> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Locked ownable synchronizers:
- None
"1499867659@qtp-1219161283-8 - Acceptor1 SelectChannelConnector@0.0.0.0:13801" #34 prio=5 os_prio=0 tid=0x000000001b62e800 nid=0x1ccc runnable [0x000000001c90f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000f8482690> (a sun.nio.ch.Util$2)
- locked <0x00000000f8482680> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f8482600> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Locked ownable synchronizers:
- None
"461160828@qtp-1219161283-7 - Acceptor0 SslSocketConnector@0.0.0.0:13901" #33 prio=5 os_prio=0 tid=0x000000001b62f800 nid=0x1900 runnable [0x000000001c80e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x0000000081e8b228> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.mortbay.jetty.security.SslSocketConnector.accept(SslSocketConnector.java:197)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Locked ownable synchronizers:
- None
"1325124186@qtp-1219161283-6" #32 prio=5 os_prio=0 tid=0x000000001b631000 nid=0x208 in Object.wait() [0x000000001c70f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000000800eadb0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"156856360@qtp-1219161283-4" #30 prio=5 os_prio=0 tid=0x000000001b62e000 nid=0xe30 in Object.wait() [0x000000001c50f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000000800eaf30> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"1226020905@qtp-1219161283-3" #29 prio=5 os_prio=0 tid=0x000000001b62c800 nid=0x1590 in Object.wait() [0x000000001c40f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000000800eb0b0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"573673894@qtp-1219161283-2" #28 prio=5 os_prio=0 tid=0x000000001b62a000 nid=0xef8 in Object.wait() [0x000000001c30f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000000800eb230> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"1321530272@qtp-1219161283-1" #27 prio=5 os_prio=0 tid=0x000000001b62d000 nid=0x110c in Object.wait() [0x000000001c20e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000000800eb3b0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"1162918744@qtp-1219161283-0" #26 prio=5 os_prio=0 tid=0x000000001b62b000 nid=0x2068 in Object.wait() [0x000000001c10f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000000800eb530> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" #23 daemon prio=5 os_prio=0 tid=0x0000000019ab5000 nid=0x3c4 runnable [0x000000001b20f000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000000800f1c80> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Accept-13301" #22 daemon prio=5 os_prio=0 tid=0x0000000019a80000 nid=0x1fa0 runnable [0x000000001b10e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x000000008012b968> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" #21 daemon prio=5 os_prio=0 tid=0x0000000019a7f000 nid=0x16d4 runnable [0x000000001b00f000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000000800f7e18> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x0000000019991000 nid=0x5e0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread2" #19 daemon prio=9 os_prio=2 tid=0x0000000019990000 nid=0x1358 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #18 daemon prio=9 os_prio=2 tid=0x000000001998b800 nid=0x790 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #17 daemon prio=9 os_prio=2 tid=0x0000000019988800 nid=0x1934 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Event Helper Thread" #16 daemon prio=10 os_prio=0 tid=0x000000001997d800 nid=0x1498 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Transport Listener: dt_socket" #15 daemon prio=10 os_prio=0 tid=0x000000001994f000 nid=0xa44 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"RMI RenewClean-[localhost:13401]" #12 daemon prio=5 os_prio=0 tid=0x000000001993e000 nid=0x27c in Object.wait() [0x000000001a80f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000008012c090> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:536)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" #11 daemon prio=5 os_prio=0 tid=0x0000000019933000 nid=0x2170 waiting on condition [0x000000001a70e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080021cc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"GC Daemon" #9 daemon prio=2 os_prio=-2 tid=0x00000000198b8000 nid=0x1820 in Object.wait() [0x000000001a14f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x0000000080000d58> (a sun.misc.GC$LatencyLock)
Locked ownable synchronizers:
- None
"RMI Reaper" #8 prio=5 os_prio=0 tid=0x00000000198b5000 nid=0xcec in Object.wait() [0x000000001a04e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000080007788> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000080007788> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Accept-13401" #7 daemon prio=5 os_prio=0 tid=0x0000000019823800 nid=0x58c runnable [0x0000000019f4e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x000000008000f820> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000017b15000 nid=0x23a4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000017b14000 nid=0xc68 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000002caa000 nid=0x1a20 in Object.wait() [0x000000001948f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000800175e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000017adb000 nid=0x1af8 in Object.wait() [0x000000001938e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x000000008001ed98> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=2 tid=0x0000000017ad9000 nid=0x148c runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002bca800 nid=0xcdc runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002bcb800 nid=0x8a4 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002bcd000 nid=0x830 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002bcf000 nid=0x1d8c runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000000002bd1000 nid=0xb9c runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000000002bd2000 nid=0x12b4 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x0000000019abb000 nid=0x152c waiting on condition
JNI global references: 7802
{code}
> Frequency IOException cast dead lock while using kahadb in high concurrency environment
> ---------------------------------------------------------------------------------------
>
> Key: AMQ-5815
> URL: https://issues.apache.org/jira/browse/AMQ-5815
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.10.0
> Environment: Master/Slave AMQ + kahadb(NAS using NFS)
> Reporter: Andrew
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> When the concurrency is in high level, iops more than 3000/s, we find that most of the amq connections are blocked.We can't either get or push any data through them unless restart.This issue appears *frequency*.
> There is many IOExceptions in the amq logs.
> {code}
> 2015-05-28 00:37:08,544 | ERROR | Checkpoint failed | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
> java.io.IOException: Input/output error
> at java.io.RandomAccessFile.write(Native Method)[:1.6.0_30]
> at java.io.RandomAccessFile.writeLong(RandomAccessFile.java:1001)[:1.6.0_30]
> at org.apache.activemq.util.RecoverableRandomAccessFile.writeLong(RecoverableRandomAccessFile.java:305)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1062)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> 2015-05-28 00:37:08,547 | INFO | Ignoring IO exception, java.io.IOException: Input/output error | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Journal Checkpoint Worker
> java.io.IOException: Input/output error
> at java.io.RandomAccessFile.write(Native Method)[:1.6.0_30]
> at java.io.RandomAccessFile.writeLong(RandomAccessFile.java:1001)[:1.6.0_30]
> at org.apache.activemq.util.RecoverableRandomAccessFile.writeLong(RecoverableRandomAccessFile.java:305)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1062)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> 2015-05-28 00:37:08,580 | INFO | KahaDB: Recovering checkpoint thread after death | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport: tcp:///10.40.50.31:58171@61616
> 2015-05-28 00:37:17,285 | INFO | Journal failed while writing at: 2359276 | org.apache.activemq.store.kahadb.disk.journal.DataFileAppender | ActiveMQ Data File Writer
> 2015-05-28 00:37:17,287 | ERROR | KahaDB failed to store to Journal | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport: tcp:///10.40.50.17:4487@61616
> java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.force0(Native Method)[:1.6.0_30]
> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)[:1.6.0_30]
> at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:373)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:193)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> 2015-05-28 00:37:17,291 | ERROR | KahaDB failed to store to Journal | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport: tcp:///10.40.50.17:4442@61616
> java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.force0(Native Method)[:1.6.0_30]
> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)[:1.6.0_30]
> at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:373)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:193)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> 2015-05-28 00:37:17,293 | INFO | Ignoring IO exception, java.io.IOException: Input/output error | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Transport: tcp:///10.40.50.17:4442@61616
> java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.force0(Native Method)[:1.6.0_30]
> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)[:1.6.0_30]
> at org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:373)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:193)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> {code}
> With the aid of jstack ,we find dead lock(How can I upload the whole jtack log?)
> The blocked connections are all waiting for 0x00007f90500034e0
> {code}
> [01:08:01] "ActiveMQ Transport: tcp:///10.40.50.17:4479@61616" daemon prio=10 tid=0x00007f8f8c015000 nid=0x2ece waiting for monitor entry [0x00007f8f1f6e4000]
> [01:08:01] java.lang.Thread.State: BLOCKED (on object monitor)
> [01:08:01] at org.apache.activemq.transaction.LocalTransaction.rollback(LocalTransaction.java:94)
> [01:08:01] - waiting to lock <0x00007f90500034e0> (a org.apache.activemq.store.kahadb.KahaDBTransactionStore)
> [01:08:01] at org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:74)
> [01:08:01] at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253)
> [01:08:01] at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112)
> [01:08:01] at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:453)
> [01:08:01] at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
> [01:08:01] at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> [01:08:01] at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> [01:08:01] at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> [01:08:01] at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> [01:08:01] at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> [01:08:01] at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> [01:08:01] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> [01:08:01] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> [01:08:01] at java.lang.Thread.run(Thread.java:662)
> {code}
> Only one connection(Thread) get this lock.But it is waiting for another Read lock 0x00007f90530ed8d8
> {code}
> [01:08:01] "ActiveMQ Transport: tcp:///10.40.50.17:4484@61616" daemon prio=10 tid=0x00007f8fa8039800 nid=0x2ed2 waiting on condition [0x00007f8f1f2e0000]
> [01:08:01] java.lang.Thread.State: WAITING (parking)
> [01:08:01] at sun.misc.Unsafe.park(Native Method)
> [01:08:01] - parking to wait for <0x00007f90530ed8d8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> [01:08:01] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> [01:08:01] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> [01:08:01] at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
> [01:08:01] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
> [01:08:01] at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
> [01:08:01] at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:972)
> [01:08:01] at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:958)
> [01:08:01] at org.apache.activemq.store.kahadb.KahaDBTransactionStore.rollback(KahaDBTransactionStore.java:312)
> [01:08:01] at org.apache.activemq.transaction.LocalTransaction.rollback(LocalTransaction.java:94)
> [01:08:01] - locked <0x00007f90500034e0> (a org.apache.activemq.store.kahadb.KahaDBTransactionStore)
> [01:08:01] at org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:74)
> [01:08:01] at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:253)
> [01:08:01] at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:112)
> [01:08:01] at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:453)
> [01:08:01] at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
> [01:08:01] at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> [01:08:01] at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> [01:08:01] at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> [01:08:01] at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> [01:08:01] at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> [01:08:01] at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> [01:08:01] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> [01:08:01] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> [01:08:01] at java.lang.Thread.run(Thread.java:662)
> {code}
> At last we find 0x00007f90530ed8d8 is in many write lock waiting.But none write lock hodes it. The dead lock occurs.
> {code}
> [01:07:56] "ActiveMQ Transport: tcp:///10.40.50.47:60701@61616" daemon prio=10 tid=0x00007f8f7cbc5000 nid=0x3ec8 waiting on condition [0x00007f900d864000]
> [01:07:56] java.lang.Thread.State: WAITING (parking)
> [01:07:56] at sun.misc.Unsafe.park(Native Method)
> [01:07:56] - parking to wait for <0x00007f90530ed8d8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> [01:07:56] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> [01:07:56] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> [01:07:56] at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
> [01:07:56] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
> [01:07:56] at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
> [01:07:56] at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:972)
> [01:07:56] at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:958)
> [01:07:56] at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:453)
> [01:07:56] at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAsyncMessage(KahaDBStore.java:407)
> [01:07:56] at org.apache.activemq.store.kahadb.KahaDBTransactionStore.removeAsyncMessage(KahaDBTransactionStore.java:498)
> [01:07:56] at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.removeAsyncMessage(KahaDBTransactionStore.java:185)
> [01:07:56] at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:998)
> [01:07:56] at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1781)
> [01:07:56] at org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:63)
> [01:07:56] at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:235)
> [01:07:56] - locked <0x00007f905d10b1e8> (a java.lang.Object)
> [01:07:56] at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:446)
> [01:07:56] at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:461)
> [01:07:56] at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
> [01:07:56] at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)
> [01:07:56] at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:92)
> [01:07:56] at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:505)
> [01:07:56] at org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)
> [01:07:56] at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> [01:07:56] at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> [01:07:56] at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> [01:07:56] at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> [01:07:56] at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> [01:07:56] at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> [01:07:56] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> [01:07:56] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> [01:07:56] at java.lang.Thread.run(Thread.java:662)
> {code}
> In my opinion, IOExceptions may lead to losing persisten messages and so on ,but the broker shouldn't stop service.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)