You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by ? ?? <wa...@outlook.com> on 2018/02/05 03:50:17 UTC
Idle Worker

Hi,

The KafkaSpout keeps running, but no tuples are emitted. There are no errors in the log. I dumped all the workers and found there seems dead locks within them. This is all I find. But I do not know how to identify the root cause. Any help would be appreciated. Below is one of the dump:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.141-b15 mixed mode):

"Attach Listener" #547 daemon prio=9 os_prio=0 tid=0x00007f230c012800 nid=0x7cfe waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"disruptor-flush-task-pool" #545 daemon prio=5 os_prio=0 tid=0x00007f22c4098800 nid=0x3f4a waiting on condition [0x00007f2137b60000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000701a2dc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"disruptor-flush-task-pool" #544 daemon prio=5 os_prio=0 tid=0x00007f22c4096800 nid=0x75bd waiting on condition [0x00007f2137c61000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000701a2dc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"disruptor-flush-task-pool" #543 daemon prio=5 os_prio=0 tid=0x00007f22c4095000 nid=0x75bc waiting on condition [0x00007f2137d62000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000701a2dc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"disruptor-flush-task-pool" #542 daemon prio=5 os_prio=0 tid=0x00007f22c4093000 nid=0x5b5d waiting on condition [0x00007f2137e63000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000701a2dc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"disruptor-flush-task-pool" #541 daemon prio=5 os_prio=0 tid=0x00007f22c4091000 nid=0x5241 waiting on condition [0x00007f2137f64000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000701a2dc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"disruptor-flush-task-pool" #540 daemon prio=5 os_prio=0 tid=0x00007f22c408f800 nid=0xab3 waiting on condition [0x00007f2138065000]
   java.lang.Thread.State: TIMED_WAITING (parking)

......


"main-SendThread(132.98.16.184:2181)" #26 daemon prio=5 os_prio=0 tid=0x00007f2371d4f800 nid=0x284c runnable [0x00007f2355494000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007019b3e40> (a sun.nio.ch.Util$3)
        - locked <0x00000007019b3e30> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007019b3d18> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

"Curator-ConnectionStateManager-0" #25 daemon prio=5 os_prio=0 tid=0x00007f2371d39800 nid=0x284b waiting on condition [0x00007f2355625000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007019b2470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at org.apache.storm.shade.org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
        at org.apache.storm.shade.org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
        at org.apache.storm.shade.org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"Log4j2-TF-6-Scheduled-1" #19 daemon prio=5 os_prio=0 tid=0x00007f2370c7f000 nid=0x2846 waiting on condition [0x00007f2356449000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000700cf9270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f23700f9800 nid=0x2841 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f23700ee000 nid=0x2840 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f23700ec000 nid=0x283f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f23700ea000 nid=0x283e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f23700e7000 nid=0x283d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f23700e5000 nid=0x283c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f23700e2800 nid=0x283b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f23700e0800 nid=0x283a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f23700d6800 nid=0x2839 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f23700d4000 nid=0x2838 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f23700d2000 nid=0x2837 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f23700cf800 nid=0x2836 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f23700cd800 nid=0x2835 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f23700cc000 nid=0x2834 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f2370099000 nid=0x2833 in Object.wait() [0x00007f2358c08000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x0000000700007e70> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f2370094800 nid=0x2832 in Object.wait() [0x00007f2358d09000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x0000000700006480> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"VM Thread" os_prio=0 tid=0x00007f237008c800 nid=0x2831 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f237001f800 nid=0x2822 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370021800 nid=0x2823 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370023000 nid=0x2824 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370025000 nid=0x2825 runnable

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370026800 nid=0x2826 runnable

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370028800 nid=0x2827 runnable

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f237002a800 nid=0x2828 runnable

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f237002c000 nid=0x2829 runnable

"Gang worker#8 (Parallel GC Threads)" os_prio=0 tid=0x00007f237002e000 nid=0x282b runnable

"Gang worker#9 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370030000 nid=0x282c runnable

"Gang worker#10 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370031800 nid=0x282d runnable

"Gang worker#11 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370033800 nid=0x282f runnable

"Gang worker#12 (Parallel GC Threads)" os_prio=0 tid=0x00007f2370035000 nid=0x2830 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f23700fc800 nid=0x2842 waiting on condition

JNI global references: 1570


________________________________
wangchunchao@outlook.com