You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Christopher L. Shannon (JIRA)" <ji...@apache.org> on 2015/11/18 18:44:10 UTC
[jira] [Closed] (AMQ-6041) Activemq have a deadLock in running for
a long time
[ https://issues.apache.org/jira/browse/AMQ-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christopher L. Shannon closed AMQ-6041.
---------------------------------------
Resolution: Not A Problem
> Activemq have a deadLock in running for a long time
> ---------------------------------------------------
>
> Key: AMQ-6041
> URL: https://issues.apache.org/jira/browse/AMQ-6041
> Project: ActiveMQ
> Issue Type: Bug
> Components: activemq-leveldb-store
> Affects Versions: 5.11.0
> Environment: development and product
> Reporter: lee
> Labels: performance
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> my activemq is a Cluster that has three activemq single instance and using replicated stored levelDB . when the cluster is running for a long time ( a few days), this situation is that JVM Old Generation is filled，and eden and one of survivor 0 or 1 is filled., but not privider service and all of port include 61616 is dead when producer is calling. this time three instance's process is running , about half an hour , zookeeper is selected and another instance start all port and providers services
> i see this stack trace in jstack, the content is follow:
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-138 Selector0" prio=10 tid=0x00007f5e3cccd800 nid=0xffb runnable [0x00007f5db3cfb000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b5628fe0> (a sun.nio.ch.Util$2)
> - locked <0x00000000b5628fc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b55fb490> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "HashSessionScavenger-2" daemon prio=10 tid=0x00007f5e3cc5a800 nid=0xff7 in Object.wait() [0x00007f5e08156000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b5621138> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000b5621138> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "HashSessionScavenger-1" daemon prio=10 tid=0x00007f5e3cc63000 nid=0xff6 in Object.wait() [0x00007f5e08257000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b5621168> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000b5621168> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "HashSessionScavenger-0" daemon prio=10 tid=0x00007f5e3cbac000 nid=0xff5 in Object.wait() [0x00007f5e08358000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b5621198> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000b5621198> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "qtp819231520-130" prio=10 tid=0x00007f5e3cb2a800 nid=0xff1 waiting on condition [0x00007f5e08459000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-129" prio=10 tid=0x00007f5e3cb28800 nid=0xff0 waiting on condition [0x00007f5e0855a000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-128" prio=10 tid=0x00007f5e3cb26800 nid=0xfef waiting on condition [0x00007f5e0865b000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-127" prio=10 tid=0x00007f5e3cb24800 nid=0xfee waiting on condition [0x00007f5e088f3000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-126" prio=10 tid=0x00007f5e3cb22000 nid=0xfed waiting on condition [0x00007f5e089f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-125 Acceptor0 SelectChannelConnector@0.0.0.0:61614" prio=10 tid=0x00007f5e3cb20800 nid=0xfec runnable [0x00007f5e08af5000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
> - locked <0x00000000b563cdb0> (a java.lang.Object)
> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-124" prio=10 tid=0x00007f5e3cb1d000 nid=0xfeb waiting on condition [0x00007f5e08bf6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-123 Selector0" prio=10 tid=0x00007f5e3cb1c800 nid=0xfea runnable [0x00007f5e08cf7000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b563d6e0> (a sun.nio.ch.Util$2)
> - locked <0x00000000b563d4e8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b5621958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c64c000 nid=0xfe9 runnable [0x00007f5e0900d000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server Thread Handler: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c754000 nid=0xfe8 waiting on condition [0x00007f5e0910e000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5651560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c745800 nid=0xfe7 runnable [0x00007f5e0920f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server Thread Handler: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c231800 nid=0xfe6 runnable [0x00007f5e09310000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.createEntry(HashMap.java:897)
> at java.util.HashMap.addEntry(HashMap.java:884)
> at java.util.HashMap.put(HashMap.java:505)
> at org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:488)
> at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:428)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c230000 nid=0xfe5 runnable [0x00007f5e09411000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server Thread Handler: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c76b000 nid=0xfe4 waiting on condition [0x00007f5e09512000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5657bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
> at java.lang.Thread.run(Thread.java:745)
> "hawtdispatch timer" daemon prio=10 tid=0x00007f5dc80da800 nid=0x3ac in Object.wait() [0x00007f5e28382000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b521a678> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:503)
> at org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:186)
> - locked <0x00000000b521a678> (a java.lang.Object)
> "hawtdispatch-DEFAULT-4" daemon prio=10 tid=0x00007f5dc8042800 nid=0x3ab runnable [0x00007f5e28483000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b5206048> (a sun.nio.ch.Util$2)
> - locked <0x00000000b5206030> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b5205ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "hawtdispatch-DEFAULT-3" daemon prio=10 tid=0x00007f5dc8041800 nid=0x3aa runnable [0x00007f5e28584000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b51f58f0> (a sun.nio.ch.Util$2)
> - locked <0x00000000b51f58d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b51f57a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "hawtdispatch-DEFAULT-2" daemon prio=10 tid=0x00007f5dc8049800 nid=0x3a9 runnable [0x00007f5e28685000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b51e5198> (a sun.nio.ch.Util$2)
> - locked <0x00000000b51e5180> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b51e5048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "hawtdispatch-DEFAULT-1" daemon prio=10 tid=0x00007f5dc803f000 nid=0x3a8 runnable [0x00007f5e28786000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b51d2aa8> (a sun.nio.ch.Util$2)
> - locked <0x00000000b51d2a90> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b51d2958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "ZooKeeper state change dispatcher thread" daemon prio=10 tid=0x00007f5e3ca77800 nid=0x3a5 waiting on condition [0x00007f5e29190000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b4fb04b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.activemq.leveldb.replicated.groups.ZKClient$StateChangeDispatcher.run(ZKClient.java:347)
> "RMI RenewClean-[172.30.0.21:57440]" daemon prio=10 tid=0x00007f5dd4003800 nid=0x3a3 in Object.wait() [0x00007f5e28b8a000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock)
> at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
> at java.lang.Thread.run(Thread.java:745)
> "RMI Scheduler(0)" daemon prio=10 tid=0x00007f5de4018000 nid=0x3a2 waiting on condition [0x00007f5e28c8b000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b4d5bc38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "GC Daemon" daemon prio=10 tid=0x00007f5de4004800 nid=0x3a0 in Object.wait() [0x00007f5e28e8d000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock)
> at sun.misc.GC$Daemon.run(GC.java:117)
> - locked <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock)
> "RMI Reaper" prio=10 tid=0x00007f5de4002800 nid=0x39f in Object.wait() [0x00007f5e28f8e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5de4001000 nid=0x39e runnable [0x00007f5e2908f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-1099" daemon prio=10 tid=0x00007f5e3ca22800 nid=0x39c runnable [0x00007f5e29291000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5e3c29e800 nid=0x39a runnable [0x00007f5e296ce000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
> at java.lang.Thread.run(Thread.java:745)
> "Service Thread" daemon prio=10 tid=0x00007f5e3c1ee000 nid=0x399 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f5e3c1eb800 nid=0x398 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f5e3c1e9800 nid=0x397 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007f5e3c1e7000 nid=0x396 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f5e3c1e5000 nid=0x395 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x00007f5e3c1bc000 nid=0x394 in Object.wait() [0x00007f5e386c2000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007f5e3c1ba000 nid=0x393 in Object.wait() [0x00007f5e387c3000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> "main" prio=10 tid=0x00007f5e3c009800 nid=0x38c waiting on condition [0x00007f5e4339d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5669f08> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:118)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> at org.apache.activemq.console.Main.main(Main.java:115)
> "VM Thread" prio=10 tid=0x00007f5e3c1b6000 nid=0x392 runnable
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01b000 nid=0x38d runnable
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01c800 nid=0x38e runnable
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01e800 nid=0x38f runnable
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c020800 nid=0x390 runnable
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f5e3c0a1000 nid=0x391 runnable
> "VM Periodic Task Thread" prio=10 tid=0x00007f5e3c2ae800 nid=0x39b waiting on condition
> JNI global references: 171
> [work@lx-dev00 tool]$ cat 21activemq.log | grep '0x00000000b4cc4358'
> - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> [work@lx-dev00 tool]$ cat 21activemq.log | grep '0x00000000b4cc4358' -C 10
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007f5e3c1ba000 nid=0x393 in Object.wait() [0x00007f5e387c3000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> "main" prio=10 tid=0x00007f5e3c009800 nid=0x38c waiting on condition [0x00007f5e4339d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5669f08> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> [work@lx-dev00 tool]$ sz
> sz: need at least one file to send
> Try `sz --help' for more information.
> [work@lx-dev00 tool]$ sz 21activemq.log
> [work@lx-dev00 tool]$
> ^C
> [work@lx-dev00 tool]$ cat 21activemq.log
> 2015-11-11 14:18:00
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b07 mixed mode):
> "main-EventThread" daemon prio=10 tid=0x00007f5dbc067000 nid=0x4b83 waiting on condition [0x00007f5db21f7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b0998d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> "main-SendThread(172.30.0.21:2181)" daemon prio=10 tid=0x00007f5dbc0ab000 nid=0x4b82 runnable [0x00007f5daf4c1000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b0988558> (a sun.nio.ch.Util$2)
> - locked <0x00000000b0988548> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b0988430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> "ActiveMQ BrokerService[test] Task-456" daemon prio=10 tid=0x00007f5dd400c000 nid=0x4b7f waiting on condition [0x00007f5db12df000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b526d470> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "Attach Listener" daemon prio=10 tid=0x00007f5e04003800 nid=0x4986 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "ActiveMQ BrokerService[test] Task-445" daemon prio=10 tid=0x00007f5dd402c800 nid=0x4901 waiting on condition [0x00007f5db0bd8000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b6849cf0> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
> at org.apache.activemq.broker.TransportConnection.stop(TransportConnection.java:1063)
> at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
> at org.apache.activemq.broker.TransportConnector.stop(TransportConnector.java:287)
> at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
> at org.apache.activemq.broker.BrokerService.stopAllConnectors(BrokerService.java:2116)
> at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:771)
> at org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)
> at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$stop_master$2.apply$mcV$sp(ElectingLevelDBStore.scala:258)
> at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ BrokerService[test] Task-435" daemon prio=10 tid=0x00007f5dbc037000 nid=0x48f3 runnable [0x00007f5db05d1000]
> java.lang.Thread.State: RUNNABLE
> at javax.management.ObjectName.getSerializedNameString(ObjectName.java:1666)
> at javax.management.ObjectName.toString(ObjectName.java:1757)
> at org.apache.activemq.broker.jmx.BrokerMBeanSupport.createProducerName(BrokerMBeanSupport.java:90)
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.removeProducer(ManagedRegionBroker.java:277)
> at org.apache.activemq.broker.BrokerFilter.removeProducer(BrokerFilter.java:142)
> at org.apache.activemq.advisory.AdvisoryBroker.removeProducer(AdvisoryBroker.java:332)
> at org.apache.activemq.broker.CompositeDestinationBroker.removeProducer(CompositeDestinationBroker.java:75)
> at org.apache.activemq.broker.BrokerFilter.removeProducer(BrokerFilter.java:142)
> at org.apache.activemq.broker.MutableBrokerFilter.removeProducer(MutableBrokerFilter.java:147)
> at org.apache.activemq.broker.TransportConnection.processRemoveProducer(TransportConnection.java:636)
> at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:741)
> at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:846)
> - locked <0x00000000b5661c80> (a org.apache.activemq.broker.jmx.ManagedTransportConnection)
> at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1187)
> at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1117)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport: tcp:///172.30.13.76:38807@61616" daemon prio=10 tid=0x00007f5dd001e000 nid=0x1040 waiting on condition [0x00007f5db1be8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b6890fc8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
> at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:181)
> at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-145" prio=10 tid=0x00007f5e3ccdc000 nid=0x1002 waiting on condition [0x00007f5db35f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-144" prio=10 tid=0x00007f5e3ccda000 nid=0x1001 waiting on condition [0x00007f5db36f5000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-143" prio=10 tid=0x00007f5e3ccd6800 nid=0x1000 waiting on condition [0x00007f5db37f6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-142" prio=10 tid=0x00007f5e3ccd4800 nid=0xfff waiting on condition [0x00007f5db38f7000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-141" prio=10 tid=0x00007f5e3ccd2800 nid=0xffe waiting on condition [0x00007f5db39f8000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-140 Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10 tid=0x00007f5e3cccf000 nid=0xffd runnable [0x00007f5db3af9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
> - locked <0x00000000b56285b8> (a java.lang.Object)
> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-139" prio=10 tid=0x00007f5e3ccce000 nid=0xffc waiting on condition [0x00007f5db3bfa000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp2054301670-138 Selector0" prio=10 tid=0x00007f5e3cccd800 nid=0xffb runnable [0x00007f5db3cfb000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b5628fe0> (a sun.nio.ch.Util$2)
> - locked <0x00000000b5628fc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b55fb490> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "HashSessionScavenger-2" daemon prio=10 tid=0x00007f5e3cc5a800 nid=0xff7 in Object.wait() [0x00007f5e08156000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b5621138> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000b5621138> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "HashSessionScavenger-1" daemon prio=10 tid=0x00007f5e3cc63000 nid=0xff6 in Object.wait() [0x00007f5e08257000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b5621168> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000b5621168> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "HashSessionScavenger-0" daemon prio=10 tid=0x00007f5e3cbac000 nid=0xff5 in Object.wait() [0x00007f5e08358000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b5621198> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x00000000b5621198> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> "qtp819231520-130" prio=10 tid=0x00007f5e3cb2a800 nid=0xff1 waiting on condition [0x00007f5e08459000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-129" prio=10 tid=0x00007f5e3cb28800 nid=0xff0 waiting on condition [0x00007f5e0855a000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-128" prio=10 tid=0x00007f5e3cb26800 nid=0xfef waiting on condition [0x00007f5e0865b000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-127" prio=10 tid=0x00007f5e3cb24800 nid=0xfee waiting on condition [0x00007f5e088f3000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-126" prio=10 tid=0x00007f5e3cb22000 nid=0xfed waiting on condition [0x00007f5e089f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-125 Acceptor0 SelectChannelConnector@0.0.0.0:61614" prio=10 tid=0x00007f5e3cb20800 nid=0xfec runnable [0x00007f5e08af5000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
> - locked <0x00000000b563cdb0> (a java.lang.Object)
> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-124" prio=10 tid=0x00007f5e3cb1d000 nid=0xfeb waiting on condition [0x00007f5e08bf6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> "qtp819231520-123 Selector0" prio=10 tid=0x00007f5e3cb1c800 nid=0xfea runnable [0x00007f5e08cf7000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b563d6e0> (a sun.nio.ch.Util$2)
> - locked <0x00000000b563d4e8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b5621958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c64c000 nid=0xfe9 runnable [0x00007f5e0900d000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server Thread Handler: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c754000 nid=0xfe8 waiting on condition [0x00007f5e0910e000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5651560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c745800 nid=0xfe7 runnable [0x00007f5e0920f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server Thread Handler: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c231800 nid=0xfe6 runnable [0x00007f5e09310000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.createEntry(HashMap.java:897)
> at java.util.HashMap.addEntry(HashMap.java:884)
> at java.util.HashMap.put(HashMap.java:505)
> at org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:488)
> at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:428)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c230000 nid=0xfe5 runnable [0x00007f5e09411000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
> at java.lang.Thread.run(Thread.java:745)
> "ActiveMQ Transport Server Thread Handler: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c76b000 nid=0xfe4 waiting on condition [0x00007f5e09512000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5657bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
> at java.lang.Thread.run(Thread.java:745)
> "hawtdispatch timer" daemon prio=10 tid=0x00007f5dc80da800 nid=0x3ac in Object.wait() [0x00007f5e28382000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b521a678> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:503)
> at org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:186)
> - locked <0x00000000b521a678> (a java.lang.Object)
> "hawtdispatch-DEFAULT-4" daemon prio=10 tid=0x00007f5dc8042800 nid=0x3ab runnable [0x00007f5e28483000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b5206048> (a sun.nio.ch.Util$2)
> - locked <0x00000000b5206030> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b5205ef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "hawtdispatch-DEFAULT-3" daemon prio=10 tid=0x00007f5dc8041800 nid=0x3aa runnable [0x00007f5e28584000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b51f58f0> (a sun.nio.ch.Util$2)
> - locked <0x00000000b51f58d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b51f57a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "hawtdispatch-DEFAULT-2" daemon prio=10 tid=0x00007f5dc8049800 nid=0x3a9 runnable [0x00007f5e28685000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b51e5198> (a sun.nio.ch.Util$2)
> - locked <0x00000000b51e5180> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b51e5048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "hawtdispatch-DEFAULT-1" daemon prio=10 tid=0x00007f5dc803f000 nid=0x3a8 runnable [0x00007f5e28786000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000b51d2aa8> (a sun.nio.ch.Util$2)
> - locked <0x00000000b51d2a90> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000b51d2958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
> at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
> at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
> at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
> "ZooKeeper state change dispatcher thread" daemon prio=10 tid=0x00007f5e3ca77800 nid=0x3a5 waiting on condition [0x00007f5e29190000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b4fb04b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.activemq.leveldb.replicated.groups.ZKClient$StateChangeDispatcher.run(ZKClient.java:347)
> "RMI RenewClean-[172.30.0.21:57440]" daemon prio=10 tid=0x00007f5dd4003800 nid=0x3a3 in Object.wait() [0x00007f5e28b8a000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock)
> at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
> at java.lang.Thread.run(Thread.java:745)
> "RMI Scheduler(0)" daemon prio=10 tid=0x00007f5de4018000 nid=0x3a2 waiting on condition [0x00007f5e28c8b000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b4d5bc38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "GC Daemon" daemon prio=10 tid=0x00007f5de4004800 nid=0x3a0 in Object.wait() [0x00007f5e28e8d000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock)
> at sun.misc.GC$Daemon.run(GC.java:117)
> - locked <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock)
> "RMI Reaper" prio=10 tid=0x00007f5de4002800 nid=0x39f in Object.wait() [0x00007f5e28f8e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5de4001000 nid=0x39e runnable [0x00007f5e2908f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-1099" daemon prio=10 tid=0x00007f5e3ca22800 nid=0x39c runnable [0x00007f5e29291000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5e3c29e800 nid=0x39a runnable [0x00007f5e296ce000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
> at java.lang.Thread.run(Thread.java:745)
> "Service Thread" daemon prio=10 tid=0x00007f5e3c1ee000 nid=0x399 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f5e3c1eb800 nid=0x398 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f5e3c1e9800 nid=0x397 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007f5e3c1e7000 nid=0x396 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f5e3c1e5000 nid=0x395 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x00007f5e3c1bc000 nid=0x394 in Object.wait() [0x00007f5e386c2000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007f5e3c1ba000 nid=0x393 in Object.wait() [0x00007f5e387c3000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock)
> "main" prio=10 tid=0x00007f5e3c009800 nid=0x38c waiting on condition [0x00007f5e4339d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000b5669f08> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:118)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> at org.apache.activemq.console.Main.main(Main.java:115)
> "VM Thread" prio=10 tid=0x00007f5e3c1b6000 nid=0x392 runnable
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01b000 nid=0x38d runnable
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01c800 nid=0x38e runnable
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01e800 nid=0x38f runnable
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c020800 nid=0x390 runnable
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f5e3c0a1000 nid=0x391 runnable
> "VM Periodic Task Thread" prio=10 tid=0x00007f5e3c2ae800 nid=0x39b waiting on condition
> JNI global references: 171
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)