You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by gzc <18...@126.com> on 2016/03/16 09:03:47 UTC

ShellBolt died

Hi, guys


    My ShellBolt died.


    2016-03-15T14:02:03.028+0800 b.s.t.ShellBolt [ERROR] Halting process: ShellBolt died.
    java.lang.RuntimeException: subprocess heartbeat timeout
      at backtype.storm.task.ShellBolt$BoltHeartbeatTimerTask.run(ShellBolt.java:305) [storm-core-0.9.6.jar:0.9.6]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_31]
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_31]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_31]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_31]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]


Why did it happen? CPU too busy? I cannot find any more clue.


--

gzc

Re: ShellBolt died

Posted by Jungtaek Lim <ka...@gmail.com>.
Hi,

Normally, multi-lang heartbeat is done with below way.

- ShellBolt sends heartbeat tuple periodically
- Subprocess reads tuple from queue and responses it to heartbeat message
- ShellBolt periodically checks the time of last heartbeat message and
suicide if it's too old

It has a serious downside: when there're so many tuples flowing through the
subprocess so that subprocess is alive but can't read heartbeat tuple in
time.

STORM-742 <http://issues.apache.org/jira/browse/STORM-742> addressed this,
and that patch became part of Storm 0.10.0. You may want to give it a try
with Storm 0.10.0.

Thanks,
Jungtaek Lim (HeartSaVioR)


2016년 3월 16일 (수) 오후 5:04, gzc <18...@126.com>님이 작성:

> Hi, guys
>
>     My ShellBolt died.
>
>     2016-03-15T14:02:03.028+0800 b.s.t.ShellBolt [ERROR] Halting process:
> ShellBolt died.
>     java.lang.RuntimeException: subprocess heartbeat timeout
>       at
> backtype.storm.task.ShellBolt$BoltHeartbeatTimerTask.run(ShellBolt.java:305)
> [storm-core-0.9.6.jar:0.9.6]
>       at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_31]
>       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_31]
>       at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_31]
>       at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_31]
>       at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_31]
>       at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_31]
>       at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
>
> Why did it happen? CPU too busy? I cannot find any more clue.
>
> --
> gzc
>
>
>
>
>
>