You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by HeartSaVioR <gi...@git.apache.org> on 2016/06/27 15:21:04 UTC
[GitHub] storm pull request #1526: STORM-1928 ShellSpout should check heartbeat while...
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1526
STORM-1928 ShellSpout should check heartbeat while ShellSpout is waiting for subprocess to sync
When nextTuple is not calling at any chances (max spout pending, backpressure introduced at 1.0.0, and so on) multi-lang spout don't receive any message from ShellSpout, thus no messages from multi-lang spout side. (because multilang spout and ShellSpout communicates like ping-pong, unlike multilang bolt and ShellBolt)
In result, no heartbeat is being marked, and finally raises subprocess heartbeat timeout.
This PR changes ShellSpout to only check heartbeat while ShellSpout is waiting for subprocess to sync.
And this PR also changes ShellSpout.die logic similar to ShellBolt, since I found ShellSpout's die message was logged but worker itself was not died after logging. Only that task was died.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-1928
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1526.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1526
----
commit 1c94bc7a820f90dff97f191cf30b485eba7b8c38
Author: Jungtaek Lim <ka...@gmail.com>
Date: 2016-06-27T15:07:59Z
STORM-1928 ShellSpout should only check heartbeat while ShellSpout is waiting for subprocess to sync
* only check heartbeat while ShellSpout is waiting for subprocess to sync
* change ShellSpout.die logic similar to ShellBolt
* ShellSpout's die message is logged but worker is not died after logging
* Confirmed that worker becomes died when die has been called
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1526: STORM-1928 ShellSpout should check heartbeat while ShellS...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1526
one minor nit, but overall I am +1 with or without the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1526: STORM-1928 ShellSpout should check heartbeat while...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1526
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1526: STORM-1928 ShellSpout should check heartbeat while ShellS...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1526
Could anyone review this? This is critical issue on multilang spout.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1526: STORM-1928 ShellSpout should check heartbeat while ShellS...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1526
OK. Only one of build which is run by `!storm-core` failed. Merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1526: STORM-1928 ShellSpout should check heartbeat while...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1526#discussion_r70251081
--- Diff: storm-core/src/jvm/org/apache/storm/spout/ShellSpout.java ---
@@ -47,6 +50,8 @@
private String[] _command;
private Map<String, String> env = new HashMap<>();
private ShellProcess _process;
+ private volatile boolean _running = true;
+ private volatile Throwable _exception;
--- End diff --
nit: this is only ever set to a RuntimeException already, and then it is wrapped in a RuntimeException. It might be nice to just have it be a RuntimeException to begin with and then we can skip the wrapping.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1526: STORM-1928 ShellSpout should check heartbeat while ShellS...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1526
+1 but all 4 CI jobs failed. They look unrelated to this, as some of them are failing because of a RAT problem. Lets fix the build ASAP before we merge in much more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1526: STORM-1928 ShellSpout should check heartbeat while...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1526#discussion_r70263885
--- Diff: storm-core/src/jvm/org/apache/storm/spout/ShellSpout.java ---
@@ -47,6 +50,8 @@
private String[] _command;
private Map<String, String> env = new HashMap<>();
private ShellProcess _process;
+ private volatile boolean _running = true;
+ private volatile Throwable _exception;
--- End diff --
Yes it would be better to fix this. Thanks for pointing out. Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1526: STORM-1928 ShellSpout should check heartbeat while ShellS...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1526
The file without a license is external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaPartitionOffsetLag.java
I'll file a separate JIRA for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1526: STORM-1928 ShellSpout should check heartbeat while ShellS...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1526
Just rebased with master which fixes RAT issue. I'll merge after build shows green or at least issues are all unrelated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---