You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Alex Sobrino (JIRA)" <ji...@apache.org> on 2015/08/06 13:03:06 UTC
[jira] [Comment Edited] (STORM-963) Frozen topology (KafkaSpout + Multilang bolt)

    [ https://issues.apache.org/jira/browse/STORM-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659799#comment-14659799 ] 

Alex Sobrino edited comment on STORM-963 at 8/6/15 11:02 AM:
-------------------------------------------------------------

Done. This is the result:

{code}
Attaching to process ID 7636, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.45-b02
Deadlock Detection:

No deadlocks found.

Thread 7761: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=100, line=1088 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7760: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7759: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7758: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7757: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.zookeeper.ClientCnxn$SendThread.run() @bci=604, line=1081 (Interpreted frame)


Thread 7756: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.access$000(org.apache.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7755: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.Reader.read(char[]) @bci=5, line=140 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer, char[]) @bci=8, line=2001 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer) @bci=7, line=1980 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.Reader, java.io.Writer) @bci=2, line=1957 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.InputStream, java.io.Writer, java.nio.charset.Charset) @bci=15, line=1907 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream, java.nio.charset.Charset) @bci=11, line=778 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream) @bci=4, line=759 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getErrorsString() @bci=11, line=138 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getProcessTerminationInfoString() @bci=19, line=180 (Interpreted frame)
 - backtype.storm.task.ShellBolt.die(java.lang.Throwable) @bci=21, line=280 (Interpreted frame)
 - backtype.storm.task.ShellBolt.access$400(backtype.storm.task.ShellBolt, java.lang.Throwable) @bci=2, line=69 (Interpreted frame)
 - backtype.storm.task.ShellBolt$BoltHeartbeatTimerTask.run() @bci=81, line=305 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
 - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7754: (state = BLOCKED)
 - java.io.BufferedInputStream.available() @bci=0, line=409 (Compiled frame)
 - backtype.storm.utils.ShellProcess.logErrorStream() @bci=4, line=125 (Compiled frame)
 - backtype.storm.utils.ShellProcess.writeBoltMsg(backtype.storm.multilang.BoltMsg) @bci=11, line=108 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=108, line=361 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7753: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7752: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.Reader.read(char[]) @bci=5, line=140 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer, char[]) @bci=8, line=2001 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer) @bci=7, line=1980 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.Reader, java.io.Writer) @bci=2, line=1957 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.InputStream, java.io.Writer, java.nio.charset.Charset) @bci=15, line=1907 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream, java.nio.charset.Charset) @bci=11, line=778 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream) @bci=4, line=759 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getErrorsString() @bci=11, line=138 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getProcessTerminationInfoString() @bci=19, line=180 (Interpreted frame)
 - backtype.storm.task.ShellBolt.die(java.lang.Throwable) @bci=21, line=280 (Interpreted frame)
 - backtype.storm.task.ShellBolt.access$400(backtype.storm.task.ShellBolt, java.lang.Throwable) @bci=2, line=69 (Interpreted frame)
 - backtype.storm.task.ShellBolt$BoltHeartbeatTimerTask.run() @bci=81, line=305 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
 - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7751: (state = BLOCKED)
 - java.io.BufferedInputStream.available() @bci=0, line=409 (Compiled frame)
 - backtype.storm.utils.ShellProcess.logErrorStream() @bci=4, line=125 (Compiled frame)
 - backtype.storm.utils.ShellProcess.writeBoltMsg(backtype.storm.multilang.BoltMsg) @bci=11, line=108 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=108, line=361 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7750: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7749: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7748: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7747: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.zookeeper.ClientCnxn$SendThread.run() @bci=604, line=1081 (Interpreted frame)


Thread 7746: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.access$000(org.apache.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7745: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7744: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7743: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.zookeeper.ClientCnxn$SendThread.run() @bci=604, line=1081 (Interpreted frame)


Thread 7738: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.access$000(org.apache.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7737: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7736: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7735: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7733: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=86, line=359 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7732: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=86, line=359 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7731: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7730: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7729: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7723: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7722: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7637: (state = BLOCKED)


Thread 7719: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7718: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7717: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7716: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.spout.SleepSpoutWaitStrategy.emptyEmit(long) @bci=4, line=36 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke() @bci=1143, line=575 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7715: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(long, com.lmax.disruptor.Sequence[], com.lmax.disruptor.util.MutableLong) @bci=34, line=99 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(com.lmax.disruptor.Sequence[]) @bci=21, line=49 (Compiled frame)
 - com.lmax.disruptor.Sequencer.next() @bci=26, line=127 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publishDirect(java.lang.Object, boolean) @bci=8, line=174 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publish(java.lang.Object, boolean) @bci=69, line=167 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=18, line=66 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object) @bci=18, line=68 (Compiled frame)
 - backtype.storm.daemon.worker$mk_transfer_local_fn$fn__6748.invoke(java.lang.Object) @bci=180, line=108 (Compiled frame)
 - backtype.storm.daemon.worker$mk_transfer_fn$fn__6755.invoke(java.lang.Object, java.lang.Object) @bci=604, line=134 (Compiled frame)
 - backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__6489.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=65, line=258 (Compiled frame)
 - backtype.storm.disruptor$clojure_handler$reify__1605.onEvent(java.lang.Object, long, boolean) @bci=28, line=58 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(long, com.lmax.disruptor.EventHandler) @bci=125, line=125 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=39, line=99 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7714: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7713: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7712: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7711: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7710: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.put(java.lang.Object) @bci=56, line=350 (Compiled frame)
 - backtype.storm.task.ShellBolt.execute(backtype.storm.tuple.Tuple) @bci=54, line=145 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$tuple_action_fn__6649.invoke(java.lang.Object, java.lang.Object) @bci=354, line=633 (Compiled frame)
 - backtype.storm.daemon.executor$mk_task_receiver$fn__6570.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=357, line=401 (Compiled frame)
 - backtype.storm.disruptor$clojure_handler$reify__1605.onEvent(java.lang.Object, long, boolean) @bci=28, line=58 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(long, com.lmax.disruptor.EventHandler) @bci=125, line=125 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=39, line=99 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7709: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7708: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.put(java.lang.Object) @bci=56, line=350 (Compiled frame)
 - backtype.storm.task.ShellBolt.execute(backtype.storm.tuple.Tuple) @bci=54, line=145 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$tuple_action_fn__6649.invoke(java.lang.Object, java.lang.Object) @bci=354, line=633 (Compiled frame)
 - backtype.storm.daemon.executor$mk_task_receiver$fn__6570.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=357, line=401 (Compiled frame)
 - backtype.storm.disruptor$clojure_handler$reify__1605.onEvent(java.lang.Object, long, boolean) @bci=28, line=58 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(long, com.lmax.disruptor.EventHandler) @bci=125, line=125 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=39, line=99 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7707: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7706: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7705: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - org.apache.storm.netty.util.HashedWheelTimer$Worker.waitForNextTick() @bci=84, line=483 (Compiled frame)
 - org.apache.storm.netty.util.HashedWheelTimer$Worker.run() @bci=49, line=392 (Compiled frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7704: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=100, line=1088 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7703: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(long, com.lmax.disruptor.Sequence[], com.lmax.disruptor.util.MutableLong) @bci=34, line=99 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(com.lmax.disruptor.Sequence[]) @bci=21, line=49 (Interpreted frame)
 - com.lmax.disruptor.Sequencer.next() @bci=26, line=127 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publishDirect(java.lang.Object, boolean) @bci=8, line=174 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publish(java.lang.Object, boolean) @bci=69, line=167 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=18, line=66 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object) @bci=18, line=68 (Compiled frame)
 - backtype.storm.daemon.worker$mk_transfer_local_fn$fn__6748.invoke(java.lang.Object) @bci=180, line=108 (Compiled frame)
 - backtype.storm.messaging.loader$mk_receive_thread$fn__1626$fn__1627.invoke() @bci=600, line=49 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7702: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select() @bci=2, line=101 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioServerBoss.select(java.nio.channels.Selector) @bci=1, line=163 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioServerBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7701: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7700: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(long, com.lmax.disruptor.Sequence[], com.lmax.disruptor.util.MutableLong) @bci=34, line=99 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(com.lmax.disruptor.Sequence[]) @bci=21, line=49 (Compiled frame)
 - com.lmax.disruptor.Sequencer.next() @bci=26, line=127 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publishDirect(java.lang.Object, boolean) @bci=8, line=174 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publish(java.lang.Object, boolean) @bci=69, line=167 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=18, line=66 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object) @bci=18, line=68 (Compiled frame)
 - backtype.storm.daemon.executor$setup_metrics_BANG_$fn__6499.invoke() @bci=80, line=274 (Compiled frame)
 - backtype.storm.timer$schedule_recurring$this__1807.invoke() @bci=7, line=99 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=174, line=50 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7699: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7698: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7697: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7696: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7695: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.NioClientBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7694: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7693: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.storm.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7692: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.storm.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7691: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.storm.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.storm.zookeeper.ClientCnxn$SendThread.run() @bci=615, line=1081 (Interpreted frame)


Thread 7690: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.storm.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.storm.curator.framework.state.ConnectionStateManager.access$000(org.apache.storm.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.storm.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7647: (state = BLOCKED)


Thread 7646: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)


Thread 7645: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157 (Compiled frame)
{code}

It's from a worker process.

Nothing is written inside the logs :-\ And {{kafkaSpout}} offset is not being updated in Zookeeper.


was (Author: alexsobrino):
Done. This is the result:

{code}
Attaching to process ID 7636, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.45-b02
Deadlock Detection:

No deadlocks found.

Thread 7761: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=100, line=1088 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7760: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7759: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7758: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7757: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.zookeeper.ClientCnxn$SendThread.run() @bci=604, line=1081 (Interpreted frame)


Thread 7756: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.access$000(org.apache.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7755: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.Reader.read(char[]) @bci=5, line=140 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer, char[]) @bci=8, line=2001 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer) @bci=7, line=1980 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.Reader, java.io.Writer) @bci=2, line=1957 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.InputStream, java.io.Writer, java.nio.charset.Charset) @bci=15, line=1907 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream, java.nio.charset.Charset) @bci=11, line=778 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream) @bci=4, line=759 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getErrorsString() @bci=11, line=138 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getProcessTerminationInfoString() @bci=19, line=180 (Interpreted frame)
 - backtype.storm.task.ShellBolt.die(java.lang.Throwable) @bci=21, line=280 (Interpreted frame)
 - backtype.storm.task.ShellBolt.access$400(backtype.storm.task.ShellBolt, java.lang.Throwable) @bci=2, line=69 (Interpreted frame)
 - backtype.storm.task.ShellBolt$BoltHeartbeatTimerTask.run() @bci=81, line=305 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
 - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7754: (state = BLOCKED)
 - java.io.BufferedInputStream.available() @bci=0, line=409 (Compiled frame)
 - backtype.storm.utils.ShellProcess.logErrorStream() @bci=4, line=125 (Compiled frame)
 - backtype.storm.utils.ShellProcess.writeBoltMsg(backtype.storm.multilang.BoltMsg) @bci=11, line=108 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=108, line=361 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7753: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7752: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.Reader.read(char[]) @bci=5, line=140 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer, char[]) @bci=8, line=2001 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copyLarge(java.io.Reader, java.io.Writer) @bci=7, line=1980 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.Reader, java.io.Writer) @bci=2, line=1957 (Interpreted frame)
 - org.apache.commons.io.IOUtils.copy(java.io.InputStream, java.io.Writer, java.nio.charset.Charset) @bci=15, line=1907 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream, java.nio.charset.Charset) @bci=11, line=778 (Interpreted frame)
 - org.apache.commons.io.IOUtils.toString(java.io.InputStream) @bci=4, line=759 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getErrorsString() @bci=11, line=138 (Interpreted frame)
 - backtype.storm.utils.ShellProcess.getProcessTerminationInfoString() @bci=19, line=180 (Interpreted frame)
 - backtype.storm.task.ShellBolt.die(java.lang.Throwable) @bci=21, line=280 (Interpreted frame)
 - backtype.storm.task.ShellBolt.access$400(backtype.storm.task.ShellBolt, java.lang.Throwable) @bci=2, line=69 (Interpreted frame)
 - backtype.storm.task.ShellBolt$BoltHeartbeatTimerTask.run() @bci=81, line=305 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
 - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7751: (state = BLOCKED)
 - java.io.BufferedInputStream.available() @bci=0, line=409 (Compiled frame)
 - backtype.storm.utils.ShellProcess.logErrorStream() @bci=4, line=125 (Compiled frame)
 - backtype.storm.utils.ShellProcess.writeBoltMsg(backtype.storm.multilang.BoltMsg) @bci=11, line=108 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=108, line=361 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7750: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7749: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7748: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7747: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.zookeeper.ClientCnxn$SendThread.run() @bci=604, line=1081 (Interpreted frame)


Thread 7746: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.access$000(org.apache.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7745: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7744: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7743: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.zookeeper.ClientCnxn$SendThread.run() @bci=604, line=1081 (Interpreted frame)


Thread 7738: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager.access$000(org.apache.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7737: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7736: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7735: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7733: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=86, line=359 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7732: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltWriterRunnable.run() @bci=86, line=359 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7731: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7730: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=284 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.nio.cs.StreamDecoder.readBytes() @bci=135, line=284 (Compiled frame)
 - sun.nio.cs.StreamDecoder.implRead(char[], int, int) @bci=112, line=326 (Compiled frame)
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=178 (Compiled frame)
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=184 (Compiled frame)
 - java.io.BufferedReader.fill() @bci=145, line=161 (Compiled frame)
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=324 (Compiled frame)
 - java.io.BufferedReader.readLine() @bci=2, line=389 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readString() @bci=12, line=179 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readMessage() @bci=1, line=166 (Compiled frame)
 - backtype.storm.multilang.JsonSerializer.readShellMsg() @bci=1, line=100 (Compiled frame)
 - backtype.storm.utils.ShellProcess.readShellMsg() @bci=4, line=99 (Compiled frame)
 - backtype.storm.task.ShellBolt$BoltReaderRunnable.run() @bci=17, line=318 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7729: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7723: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7722: (state = IN_NATIVE)
 - java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
 - java.lang.UNIXProcess.lambda$initStreams$266() @bci=5, line=290 (Interpreted frame)
 - java.lang.UNIXProcess$$Lambda$9.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7637: (state = BLOCKED)


Thread 7719: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7718: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7717: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7716: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.spout.SleepSpoutWaitStrategy.emptyEmit(long) @bci=4, line=36 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke() @bci=1143, line=575 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7715: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(long, com.lmax.disruptor.Sequence[], com.lmax.disruptor.util.MutableLong) @bci=34, line=99 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(com.lmax.disruptor.Sequence[]) @bci=21, line=49 (Compiled frame)
 - com.lmax.disruptor.Sequencer.next() @bci=26, line=127 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publishDirect(java.lang.Object, boolean) @bci=8, line=174 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publish(java.lang.Object, boolean) @bci=69, line=167 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=18, line=66 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object) @bci=18, line=68 (Compiled frame)
 - backtype.storm.daemon.worker$mk_transfer_local_fn$fn__6748.invoke(java.lang.Object) @bci=180, line=108 (Compiled frame)
 - backtype.storm.daemon.worker$mk_transfer_fn$fn__6755.invoke(java.lang.Object, java.lang.Object) @bci=604, line=134 (Compiled frame)
 - backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__6489.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=65, line=258 (Compiled frame)
 - backtype.storm.disruptor$clojure_handler$reify__1605.onEvent(java.lang.Object, long, boolean) @bci=28, line=58 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(long, com.lmax.disruptor.EventHandler) @bci=125, line=125 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=39, line=99 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7714: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7713: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7712: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7711: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7710: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.put(java.lang.Object) @bci=56, line=350 (Compiled frame)
 - backtype.storm.task.ShellBolt.execute(backtype.storm.tuple.Tuple) @bci=54, line=145 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$tuple_action_fn__6649.invoke(java.lang.Object, java.lang.Object) @bci=354, line=633 (Compiled frame)
 - backtype.storm.daemon.executor$mk_task_receiver$fn__6570.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=357, line=401 (Compiled frame)
 - backtype.storm.disruptor$clojure_handler$reify__1605.onEvent(java.lang.Object, long, boolean) @bci=28, line=58 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(long, com.lmax.disruptor.EventHandler) @bci=125, line=125 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=39, line=99 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7709: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7708: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.put(java.lang.Object) @bci=56, line=350 (Compiled frame)
 - backtype.storm.task.ShellBolt.execute(backtype.storm.tuple.Tuple) @bci=54, line=145 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$tuple_action_fn__6649.invoke(java.lang.Object, java.lang.Object) @bci=354, line=633 (Compiled frame)
 - backtype.storm.daemon.executor$mk_task_receiver$fn__6570.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=357, line=401 (Compiled frame)
 - backtype.storm.disruptor$clojure_handler$reify__1605.onEvent(java.lang.Object, long, boolean) @bci=28, line=58 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(long, com.lmax.disruptor.EventHandler) @bci=125, line=125 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=39, line=99 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.daemon.executor$fn__6647$fn__6659$fn__6706.invoke() @bci=17, line=748 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7707: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Compiled frame)
 - com.lmax.disruptor.BlockingWaitStrategy.waitFor(long, com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence[], com.lmax.disruptor.SequenceBarrier, long, java.util.concurrent.TimeUnit) @bci=58, line=87 (Compiled frame)
 - com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long, long, java.util.concurrent.TimeUnit) @bci=21, line=54 (Compiled frame)
 - backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(com.lmax.disruptor.EventHandler) @bci=21, line=97 (Compiled frame)
 - backtype.storm.disruptor$consume_batch_when_available.invoke(java.lang.Object, java.lang.Object) @bci=12, line=80 (Compiled frame)
 - backtype.storm.disruptor$consume_loop_STAR_$fn__1618.invoke() @bci=17, line=94 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7706: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7705: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - org.apache.storm.netty.util.HashedWheelTimer$Worker.waitForNextTick() @bci=84, line=483 (Compiled frame)
 - org.apache.storm.netty.util.HashedWheelTimer$Worker.run() @bci=49, line=392 (Compiled frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7704: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=100, line=1088 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7703: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(long, com.lmax.disruptor.Sequence[], com.lmax.disruptor.util.MutableLong) @bci=34, line=99 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(com.lmax.disruptor.Sequence[]) @bci=21, line=49 (Interpreted frame)
 - com.lmax.disruptor.Sequencer.next() @bci=26, line=127 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publishDirect(java.lang.Object, boolean) @bci=8, line=174 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publish(java.lang.Object, boolean) @bci=69, line=167 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=18, line=66 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object) @bci=18, line=68 (Compiled frame)
 - backtype.storm.daemon.worker$mk_transfer_local_fn$fn__6748.invoke(java.lang.Object) @bci=180, line=108 (Compiled frame)
 - backtype.storm.messaging.loader$mk_receive_thread$fn__1626$fn__1627.invoke() @bci=600, line=49 (Compiled frame)
 - backtype.storm.util$async_loop$fn__459.invoke() @bci=39, line=463 (Compiled frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7702: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select() @bci=2, line=101 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioServerBoss.select(java.nio.channels.Selector) @bci=1, line=163 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioServerBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7701: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7700: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(long, com.lmax.disruptor.Sequence[], com.lmax.disruptor.util.MutableLong) @bci=34, line=99 (Compiled frame)
 - com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(com.lmax.disruptor.Sequence[]) @bci=21, line=49 (Compiled frame)
 - com.lmax.disruptor.Sequencer.next() @bci=26, line=127 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publishDirect(java.lang.Object, boolean) @bci=8, line=174 (Interpreted frame)
 - backtype.storm.utils.DisruptorQueue.publish(java.lang.Object, boolean) @bci=69, line=167 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object, java.lang.Object) @bci=18, line=66 (Compiled frame)
 - backtype.storm.disruptor$publish.invoke(java.lang.Object, java.lang.Object) @bci=18, line=68 (Compiled frame)
 - backtype.storm.daemon.executor$setup_metrics_BANG_$fn__6499.invoke() @bci=80, line=274 (Compiled frame)
 - backtype.storm.timer$schedule_recurring$this__1807.invoke() @bci=7, line=99 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=174, line=50 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7699: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7698: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7697: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7696: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - backtype.storm.utils.Time.sleepUntil(long) @bci=173, line=76 (Compiled frame)
 - backtype.storm.utils.Time.sleep(long) @bci=5, line=81 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke() @bci=220, line=57 (Compiled frame)
 - backtype.storm.timer$mk_timer$fn__1790.invoke() @bci=54, line=42 (Interpreted frame)
 - clojure.lang.AFn.run() @bci=1, line=24 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7695: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.NioClientBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7694: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Compiled frame)
 - org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.apache.storm.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.apache.storm.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7693: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=28, line=211 (Interpreted frame)
 - java.util.concurrent.DelayQueue.take() @bci=1, line=70 (Interpreted frame)
 - org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop() @bci=10, line=781 (Interpreted frame)
 - org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.access$400(org.apache.storm.curator.framework.imps.CuratorFrameworkImpl) @bci=1, line=57 (Interpreted frame)
 - org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$4.call() @bci=4, line=275 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7692: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - org.apache.storm.zookeeper.ClientCnxn$EventThread.run() @bci=9, line=494 (Interpreted frame)


Thread 7691: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(int, java.util.List, java.util.LinkedList, org.apache.storm.zookeeper.ClientCnxn) @bci=6, line=349 (Compiled frame)
 - org.apache.storm.zookeeper.ClientCnxn$SendThread.run() @bci=615, line=1081 (Interpreted frame)


Thread 7690: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=403 (Interpreted frame)
 - org.apache.storm.curator.framework.state.ConnectionStateManager.processEvents() @bci=13, line=208 (Interpreted frame)
 - org.apache.storm.curator.framework.state.ConnectionStateManager.access$000(org.apache.storm.curator.framework.state.ConnectionStateManager) @bci=1, line=42 (Interpreted frame)
 - org.apache.storm.curator.framework.state.ConnectionStateManager$1.call() @bci=4, line=110 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 7647: (state = BLOCKED)


Thread 7646: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)


Thread 7645: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157 (Compiled frame)
{code}

Nothing is written inside the logs :-\ And {{kafkaSpout}} offset is not being updated in Zookeeper.

> Frozen topology (KafkaSpout + Multilang bolt)
> ---------------------------------------------
>
>                 Key: STORM-963
>                 URL: https://issues.apache.org/jira/browse/STORM-963
>             Project: Apache Storm
>          Issue Type: Bug
>    Affects Versions: 0.9.4, 0.9.5, 0.9.6
>         Environment: - VMware ESX 5.5 
> - Ubuntu Server 14.04 LTS (kernel 3.16.0-41-generic)
> - Java (TM) SE Runtime Environment (build 1.8.0_45-b14)
> - Python 2.7.6 (default, Jun 22 2015, 17:58:13)
> - Zookeeper 3.4.6
>            Reporter: Alex Sobrino
>              Labels: multilang
>
> Hi,
> We've got a pretty simple topology running with Storm 0.9.5 (tried also with 0.9.4 and 0.9.6-INCUBATING) in a 3 machine cluster:
> {code}kafkaSpout (3) -----> processBolt (12){code}
> Some info:
> - kafkaSpout reads from a topic with 3 partitions and 2 replications
> - processBolt iterates throught the message and saves the results in MongoDB
> - processBolt is implemented in Python and has a storm.log("I'm doing something") just to add a simple debug message in the logs
> - The messages can be quite big (~25-40 MB) and are in JSON format
> - The kafka topic has a retention of 2 hours
> - We use the same ZooKeeper cluster to both Kafka and Storm
> The topology gets frozen after several hours (not days) running. We don't see any message in the logs... In fact, the periodic message from s.k.KafkaUtils and s.k.ZkCoordinator disapears. As you can imagine, the message from the Bolt also dissapears. Logs are copy/pasted further on. If we redeploy the topology everything starts to work again until it becomes frozen again.
> Our kafkaSpout config is:
> {code}
> ZkHosts zkHosts = new ZkHosts("zkhost01:2181,zkhost02:2181,zkhost03:2181");
> SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, "topic", "/topic/ourclientid", "ourclientid");
> kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
> kafkaConfig.fetchSizeBytes = 50*1024*1024;
> kafkaConfig.bufferSizeBytes = 50*1024*1024;
> {code}
> We've also tried setting the following options
> {code}
> kafkaConfig.forceFromStart = true;
> kafkaConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime(); // Also with kafka.api.OffsetRequest.LatestTime();
> kafkaConfig.useStartOffsetTimeIfOffsetOutOfRange = true;
> {code}
> Right now the topology is running without acking the messages since there's a bug in kafkaSpout with failed messages and deleted offsets in Kafka.
> This is what can be seen in the logs in one of the workers:
> {code}
> 2015-07-23T12:37:38.008+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364, name:processBolt I'm doing something
> 2015-07-23T12:37:39.079+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364, name:processBolt I'm doing something
> 2015-07-23T12:37:51.013+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364, name:processBolt I'm doing something
> 2015-07-23T12:37:51.091+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364, name:processBolt I'm doing something
> 2015-07-23T12:38:02.684+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing partition manager connections
> 2015-07-23T12:38:02.687+0200 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092, 1=kafka2:9092, 2=kafka3:9092}}
> 2015-07-23T12:38:02.687+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned [Partition{host=kafka2, partition=1}]
> 2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted partition managers: []
> 2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] New partition managers: []
> 2015-07-23T12:38:02.687+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished refreshing
> 2015-07-23T12:38:09.012+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364, name:processBolt I'm doing something
> 2015-07-23T12:38:41.878+0200 b.s.t.ShellBolt [INFO] ShellLog pid:28364, name:processBolt I'm doing something
> 2015-07-23T12:39:02.688+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing partition manager connections
> 2015-07-23T12:39:02.691+0200 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092, 1=kafka2:9092, 2=kafka3:9092}}
> 2015-07-23T12:39:02.691+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned [Partition{host=kafka2:9092, partition=1}]
> 2015-07-23T12:39:02.691+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted partition managers: []
> 2015-07-23T12:39:02.691+0200 s.k.ZkCoordinator [INFO] Task [2/3] New partition managers: []
> 2015-07-23T12:39:02.691+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished refreshing
> 2015-07-23T12:40:02.692+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing partition manager connections
> 2015-07-23T12:40:02.695+0200 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092, 1=kafka2:9092, 2=kafka3:9092}}
> 2015-07-23T12:40:02.695+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned [Partition{host=kafka2:9092, partition=1}]
> 2015-07-23T12:40:02.695+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted partition managers: []
> 2015-07-23T12:40:02.695+0200 s.k.ZkCoordinator [INFO] Task [2/3] New partition managers: []
> 2015-07-23T12:40:02.695+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished refreshing
> 2015-07-23T12:41:02.696+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing partition manager connections
> 2015-07-23T12:41:02.699+0200 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092, 1=kafka2:9092, 2=kafka3:9092}}
> 2015-07-23T12:41:02.699+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned [Partition{host=kafka2:9092, partition=1}]
> 2015-07-23T12:41:02.699+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted partition managers: []
> 2015-07-23T12:41:02.699+0200 s.k.ZkCoordinator [INFO] Task [2/3] New partition managers: []
> 2015-07-23T12:41:02.699+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished refreshing
> 2015-07-23T12:42:02.735+0200 s.k.ZkCoordinator [INFO] Task [2/3] Refreshing partition manager connections
> 2015-07-23T12:42:02.737+0200 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=kafka1:9092, 1=kafka2:9092, 2=kafka3:9092}}
> 2015-07-23T12:42:02.737+0200 s.k.KafkaUtils [INFO] Task [2/3] assigned [Partition{host=kafka2:9092, partition=1}]
> 2015-07-23T12:42:02.737+0200 s.k.ZkCoordinator [INFO] Task [2/3] Deleted partition managers: []
> 2015-07-23T12:42:02.737+0200 s.k.ZkCoordinator [INFO] Task [2/3] New partition managers: []
> 2015-07-23T12:42:02.737+0200 s.k.ZkCoordinator [INFO] Task [2/3] Finished refreshing
> {code}
> and then it becomes frozen. Nothing is written into the nimbus log. We've checked the offsets in ZooKeeper and they're not updated:
> {code}
> {"topology":{"id":"218e58a5-6bfb-4b32-ae89-f3afa19306e1","name":"our-topology"},"offset":12047144,"partition":1,"broker":{"host":"kafka2","port":9092},"topic":"topic"}
> cZxid = 0x100028958
> ctime = Wed Jul 01 12:22:36 CEST 2015
> mZxid = 0x100518527
> mtime = Thu Jul 23 12:42:41 CEST 2015
> pZxid = 0x100028958
> cversion = 0
> dataVersion = 446913
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 183
> numChildren = 0
> {code}
> Any ideas of what we could be missing?
> PS: This was sent to the Storm user's mailing list and got 0 replies :\



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)