You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Enno Shioji (JIRA)" <ji...@apache.org> on 2015/05/28 17:24:17 UTC
[jira] [Created] (STORM-839) Deadlock hazard in
backtype.storm.messaging.netty.Client
Enno Shioji created STORM-839:
---------------------------------
Summary: Deadlock hazard in backtype.storm.messaging.netty.Client
Key: STORM-839
URL: https://issues.apache.org/jira/browse/STORM-839
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.4
Reporter: Enno Shioji
Priority: Critical
See the thread dump below that shows the deadlock. client-worker-1 is holding 7b5a7fa5 and waiting on 1446a1e9. Thread-10-disruptor-worker-transfer-queue is holding 1446a1e9 and is waiting on 7b5a7fa5.
2015-05-28 15:37:15
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):
"JMX server connection timeout 61" - Thread t@61
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <131b6fcd> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Connection(5)-10.2.99.116" - Thread t@60
java.lang.Thread.State: RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.reflect.misc.Trampoline.invoke(Unknown Source)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at javax.management.StandardMBean.invoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <a1122e0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Connection(4)-10.2.99.116" - Thread t@59
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <35745453> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <5a7a53c9> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Connection(6)-10.2.99.116" - Thread t@58
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <7fc3219> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <1df8e296> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"metrics-graphite-reporter-2-thread-1" - Thread t@57
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <6e490587> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"metrics-graphite-reporter-1-thread-1" - Thread t@56
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <7c0a1aa1> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"DestroyJavaVM" - Thread t@55
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-10-disruptor-worker-transfer-queue" - Thread t@52
java.lang.Thread.State: BLOCKED
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:398)
- waiting to lock <7b5a7fa5> (a java.lang.Object) owned by "client-worker-1" t@25
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:128)
at org.apache.storm.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84)
at org.apache.storm.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
at org.apache.storm.netty.channel.Channels.write(Channels.java:725)
at org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.apache.storm.netty.channel.Channels.write(Channels.java:704)
at org.apache.storm.netty.channel.Channels.write(Channels.java:671)
at org.apache.storm.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at backtype.storm.messaging.netty.Client.flushMessages(Client.java:480)
- locked <1446a1e9> (a backtype.storm.messaging.netty.Client)
at backtype.storm.messaging.netty.Client.send(Client.java:412)
- locked <1446a1e9> (a backtype.storm.messaging.netty.Client)
at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54)
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__5014$fn__5015.invoke(worker.clj:334)
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__5014.invoke(worker.clj:332)
at backtype.storm.disruptor$clojure_handler$reify__1446.onEvent(disruptor.clj:58)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1459.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-9-__system" - Thread t@51
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <727268e7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:87)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:97)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.daemon.executor$fn__4722$fn__4734$fn__4781.invoke(executor.clj:748)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-8-disruptor-executor[-1 -1]-send-queue" - Thread t@50
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <35455064> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:87)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:97)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1459.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-7-parent" - Thread t@49
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.SingleThreadedClaimStrategy.waitForFreeSlotAt(SingleThreadedClaimStrategy.java:129)
at com.lmax.disruptor.SingleThreadedClaimStrategy.incrementAndGet(SingleThreadedClaimStrategy.java:81)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:174)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:167)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4506.invoke(executor.clj:190)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4506.invoke(executor.clj:197)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4506.invoke(executor.clj:199)
at backtype.storm.daemon.executor$fn__4722$fn__4734$bolt_emit__4761.invoke(executor.clj:679)
at backtype.storm.daemon.executor$fn__4722$fn$reify__4767.emit(executor.clj:698)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:203)
at backtype.storm.task.OutputCollector.emit(OutputCollector.java:49)
at backtype.storm.topology.BasicOutputCollector.emit(BasicOutputCollector.java:36)
at backtype.storm.topology.BasicOutputCollector.emit(BasicOutputCollector.java:40)
at com.adform.experiment.RequestFieldExtractBolt.execute(RequestFieldExtractBolt.scala:44)
at backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
at backtype.storm.daemon.executor$fn__4722$tuple_action_fn__4724.invoke(executor.clj:633)
at backtype.storm.daemon.executor$mk_task_receiver$fn__4645.invoke(executor.clj:401)
at backtype.storm.disruptor$clojure_handler$reify__1446.onEvent(disruptor.clj:58)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.daemon.executor$fn__4722$fn__4734$fn__4781.invoke(executor.clj:748)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-6-disruptor-executor[43 43]-send-queue" - Thread t@48
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(AbstractMultithreadedClaimStrategy.java:99)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(AbstractMultithreadedClaimStrategy.java:49)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:174)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:167)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:68)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4830.invoke(worker.clj:133)
at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__4564.invoke(executor.clj:258)
at backtype.storm.disruptor$clojure_handler$reify__1446.onEvent(disruptor.clj:58)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1459.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-5-child" - Thread t@47
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.SingleThreadedClaimStrategy.waitForFreeSlotAt(SingleThreadedClaimStrategy.java:129)
at com.lmax.disruptor.SingleThreadedClaimStrategy.incrementAndGet(SingleThreadedClaimStrategy.java:81)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:174)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:167)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4506.invoke(executor.clj:190)
at backtype.storm.daemon.executor$mk_executor_transfer_fn$this__4506.invoke(executor.clj:197)
at backtype.storm.daemon.task$send_unanchored.invoke(task.clj:112)
at backtype.storm.daemon.task$send_unanchored.invoke(task.clj:117)
at backtype.storm.daemon.executor$fn__4722$fn$reify__4767.ack(executor.clj:707)
at backtype.storm.task.OutputCollector.ack(OutputCollector.java:213)
at backtype.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:51)
at backtype.storm.daemon.executor$fn__4722$tuple_action_fn__4724.invoke(executor.clj:633)
at backtype.storm.daemon.executor$mk_task_receiver$fn__4645.invoke(executor.clj:401)
at backtype.storm.disruptor$clojure_handler$reify__1446.onEvent(disruptor.clj:58)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.daemon.executor$fn__4722$fn__4734$fn__4781.invoke(executor.clj:748)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-4-disruptor-executor[9 9]-send-queue" - Thread t@46
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(AbstractMultithreadedClaimStrategy.java:99)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(AbstractMultithreadedClaimStrategy.java:49)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:174)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:167)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:68)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4830.invoke(worker.clj:133)
at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__4564.invoke(executor.clj:258)
at backtype.storm.disruptor$clojure_handler$reify__1446.onEvent(disruptor.clj:58)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1459.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-10" - Thread t@45
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-9" - Thread t@44
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-8" - Thread t@43
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-7" - Thread t@42
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-6" - Thread t@41
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-5" - Thread t@40
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <5511289a> (a org.apache.storm.netty.channel.DefaultChannelFuture)
at java.lang.Object.wait(Object.java:503)
at org.apache.storm.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:259)
at backtype.storm.messaging.netty.Client$Connector.call(Client.java:613)
at backtype.storm.messaging.netty.Client$Connector.call(Client.java:595)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <40c7bbb1> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"pool-5-thread-1" - Thread t@26
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.storm.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
at org.apache.storm.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
at org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-4" - Thread t@39
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-3" - Thread t@38
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-2" - Thread t@37
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-schedule-service-1" - Thread t@36
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <59a0f08e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Thread-3-worker-receiver-thread-0" - Thread t@35
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(AbstractMultithreadedClaimStrategy.java:99)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(AbstractMultithreadedClaimStrategy.java:49)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:174)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:167)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:68)
at backtype.storm.daemon.worker$mk_transfer_local_fn$fn__4823.invoke(worker.clj:108)
at backtype.storm.messaging.loader$mk_receive_thread$fn__1467$fn__1468.invoke(loader.clj:49)
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Netty-server-localhost-6703-boss-1" - Thread t@34
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <6c56de1b> (a sun.nio.ch.Util$2)
- locked <14eb644d> (a java.util.Collections$UnmodifiableSet)
- locked <31714e34> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.storm.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.apache.storm.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <34fd48bb> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Netty-server-localhost-6703-worker-1" - Thread t@33
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <161e277d> (a sun.nio.ch.Util$2)
- locked <7cbc4835> (a java.util.Collections$UnmodifiableSet)
- locked <52524924> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <26041cb3> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"user-timer" - Thread t@32
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(AbstractMultithreadedClaimStrategy.java:99)
at com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(AbstractMultithreadedClaimStrategy.java:49)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:174)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:167)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:68)
at backtype.storm.daemon.executor$setup_metrics_BANG_$fn__4574.invoke(executor.clj:274)
at backtype.storm.timer$schedule_recurring$this__1538.invoke(timer.clj:99)
at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:50)
at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"executor-heartbeat-timer" - Thread t@31
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"refresh-active-timer" - Thread t@30
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"refresh-connections-timer" - Thread t@29
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <5980c217> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(Unknown Source)
at backtype.storm.daemon.worker$mk_refresh_connections$this__4973$fn__4995.invoke(worker.clj:292)
at backtype.storm.daemon.worker$mk_refresh_connections$this__4973.invoke(worker.clj:292)
at backtype.storm.daemon.worker$mk_refresh_connections$this__4973.invoke(worker.clj:257)
at backtype.storm.timer$schedule_recurring$this__1538.invoke(timer.clj:99)
at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:50)
at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"heartbeat-timer" - Thread t@28
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"client-boss-1" - Thread t@27
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <12f51383> (a sun.nio.ch.Util$2)
- locked <6123693a> (a java.util.Collections$UnmodifiableSet)
- locked <63931ce2> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.apache.storm.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <1d363114> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"client-worker-1" - Thread t@25
java.lang.Thread.State: BLOCKED
at backtype.storm.messaging.netty.Client.closeChannelAndReconnect(Client.java:501)
- waiting to lock <1446a1e9> (a backtype.storm.messaging.netty.Client) owned by "Thread-10-disruptor-worker-transfer-queue" t@52
at backtype.storm.messaging.netty.Client.access$1400(Client.java:78)
at backtype.storm.messaging.netty.Client$3.operationComplete(Client.java:492)
at org.apache.storm.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:427)
at org.apache.storm.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:413)
at org.apache.storm.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:380)
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:437)
- locked <7b5a7fa5> (a java.lang.Object)
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373)
at org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <75e528fd> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"CuratorFramework-0" - Thread t@24
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <292fd265> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:781)
at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:57)
at org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:275)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <66e1a48> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" - Thread t@23
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <29dfa03> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.storm.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Locked ownable synchronizers:
- None
"main-SendThread(10.3.14.10:2181)" - Thread t@22
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <6fe68c05> (a sun.nio.ch.Util$2)
- locked <7ea90d2e> (a java.util.Collections$UnmodifiableSet)
- locked <7f3ef2e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Locked ownable synchronizers:
- None
"ConnectionStateManager-0" - Thread t@21
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <5188dc4c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)
at org.apache.storm.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:208)
at org.apache.storm.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:42)
at org.apache.storm.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:110)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <119fc7a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"JMX server connection timeout 16" - Thread t@16
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <70372b65> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" - Thread t@15
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <36022d71> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Connection(2)-10.2.14.142" - Thread t@14
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <7dbd9e9b> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <39ef77c2> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"logback-async-disruptor-appender-172.22.12.55:4560-1" - Thread t@12
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler.openSocket(AbstractLogstashTcpSocketAppender.java:244)
- locked <10977258> (a net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler)
at net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler.onStart(AbstractLogstashTcpSocketAppender.java:191)
at net.logstash.logback.encoder.com.lmax.disruptor.BatchEventProcessor.notifyStart(BatchEventProcessor.java:184)
at net.logstash.logback.encoder.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <6a22c2c3> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Accept-0" - Thread t@11
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Accept-16703" - Thread t@10
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" - Thread t@9
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Signal Dispatcher" - Thread t@4
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" - Thread t@3
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <48f5716a> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Reference Handler" - Thread t@2
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <69e90b91> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Locked ownable synchronizers:
- None
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)