[jira] [Created] (STORM-75) Dead lock between ShellBolt and ShellProcess

["James Xu (JIRA)" <ji...@apache.org>]

James Xu created STORM-75:
-----------------------------

             Summary: Dead lock between ShellBolt and ShellProcess
                 Key: STORM-75
                 URL: https://issues.apache.org/jira/browse/STORM-75
             Project: Apache Storm (Incubating)
          Issue Type: Bug
            Reporter: James Xu
            Priority: Minor


https://github.com/nathanmarz/storm/issues/423

The ShellBolt creates shell process and read data from output stream and error stream. The current implementation only read error stream when the output stream is closed. So messages in error stream will be put into the buffer of error stream. When the buffer is fully filled, the output in shell process would be blocked waiting for the error stream buffer to become available. While in ShellBolt it will also block there wait for the output in output stream from shell process. So it's a dead lock.

This behavior seems dangerous as the issue can be hidden, it can hardly be seen in normal tests. And normally the error output won't be too big to fill up the error stream buffer, but after the system have been running for a while on production, the error stream can be accumulated to full, and then dead lock would happen. There's no any error in log, hard to debug.

Here in Yahoo we are using many native libraries which is built long time ago, which sometimes writes to error stream when there's some error. It's impossible for us to inspect all the direct and indirect native library dependencies and rebuild all to remove all the error stream writing.

Now we used a workaround to redirect error stream into /dev/null at the beginning of our shell process. But I think in long term it should be fixed in ShellBolt and ShellProcess.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)