You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2014/10/08 16:09:33 UTC

[jira] [Commented] (STORM-513) ShellBolt keeps sending heartbeats even when child process is hung

    [ https://issues.apache.org/jira/browse/STORM-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163526#comment-14163526 ] 

ASF GitHub Bot commented on STORM-513:
--------------------------------------

GitHub user HeartSaVioR opened a pull request:

    https://github.com/apache/storm/pull/286

    STORM-513 check heartbeat from multilang subprocess

    Related issue link : https://issues.apache.org/jira/browse/STORM-513
    
    It seems that ShellSpout and ShellBolt doesn't check subprocess, and set heartbeat with their only states.
    Subprocess could hang, but it doesn't affect ShellSpout / ShellBolt. It just stops working on tuple.
    It's better to check heartbeat from subprocess, and suicide if subprocess stops working.
    
    * Spout
      * ShellSpout sends "next" to subprocess continuously
      * subprocess sends "sync" to ShellSpout when "next" is received
      * so we can treat "sync", or any messages to heartbeat
    * Bolt
      * ShellBolt sends tuples to subprocess if it's available
      * so we need to send "heartbeat" tuple
       * subprocess sends "sync" to ShellBolt when "heartbeat" tuple is received

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HeartSaVioR/storm STORM-513

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #286
    
----
commit ca5874cdf11af8d835335d228b643f28aeb3f9c3
Author: Jungtaek Lim <ka...@gmail.com>
Date:   2014-10-08T13:48:01Z

    STORM-513 check heartbeat from multilang subprocess
    
    * Spout
    ** ShellSpout sends "next" to subprocess continuously
    ** subprocess sends "sync" to ShellSpout when "next" is received
    ** so we can treat "sync", or any messages to heartbeat
    * Bolt
    ** ShellBolt sends tuples to subprocess if it's available
    ** so we need to send "heartbeat" tuple
    ** subprocess sends "sync" to ShellBolt when "heartbeat" tuple is
    received

commit 1a0d4bdd735ba0ade42f6777a4c47affec931557
Author: Jungtaek Lim <ka...@gmail.com>
Date:   2014-10-08T14:06:24Z

    Fix mixed tab / space, remove FIXME

----


> ShellBolt keeps sending heartbeats even when child process is hung
> ------------------------------------------------------------------
>
>                 Key: STORM-513
>                 URL: https://issues.apache.org/jira/browse/STORM-513
>             Project: Apache Storm
>          Issue Type: Bug
>         Environment: Linux: 2.6.32-431.11.2.el6.x86_64 (RHEL 6.5)
>            Reporter: Dan Blanchard
>            Priority: Blocker
>
> If I'm understanding everything correctly with how ShellBolts work, the Java ShellBolt executor is the part of the topology that sends heartbeats back to Nimbus to let it know that a particular multilang bolt is still alive.  The problem with this is that if the multilang subprocess/bolt severely hangs (i.e., it will not even respond to {{SIGALRM}} and the like), the Java ShellBolt does not seem to notice or care. Simply having the tuple get replayed when it times out will not suffice either, because the subprocess will still be stuck.
> The most obvious way to handle this seem to be to add heartbeating to the multilang protocol itself, so that the ShellBolt expects a message of some kind every {{timeout}} seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)