You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2014/05/20 19:47:24 UTC
[1/5] git commit: Removed inline heartbeats to avoid being killed
when under heavy ZK load.
Repository: incubator-storm
Updated Branches:
refs/heads/master 52fe66e9f -> fe7ac2910
Removed inline heartbeats to avoid being killed when under heavy ZK load.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/0666c413
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/0666c413
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/0666c413
Branch: refs/heads/master
Commit: 0666c41387fc11c0422b26ab27ebc38c30fe26af
Parents: dc4de42
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Tue Apr 29 17:56:53 2014 +0000
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Tue Apr 29 17:56:53 2014 +0000
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/worker.clj | 7 -------
1 file changed, 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/0666c413/storm-core/src/clj/backtype/storm/daemon/worker.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/worker.clj b/storm-core/src/clj/backtype/storm/daemon/worker.clj
index 0d1f6c6..b7fe1cf 100644
--- a/storm-core/src/clj/backtype/storm/daemon/worker.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/worker.clj
@@ -352,13 +352,6 @@
(touch (worker-pid-path conf worker-id (process-pid))))
(let [worker (worker-data conf shared-mq-context storm-id assignment-id port worker-id)
heartbeat-fn #(do-heartbeat worker)
- ;; do this here so that the worker process dies if this fails
- ;; it's important that worker heartbeat to supervisor ASAP when launching so that the supervisor knows it's running (and can move on)
- _ (heartbeat-fn)
-
- ;; heartbeat immediately to nimbus so that it knows that the worker has been started
- _ (do-executor-heartbeats worker)
-
executors (atom nil)
;; launch heartbeat threads immediately so that slow-loading tasks don't cause the worker to timeout
[3/5] git commit: Addressed review comments.
Posted by pt...@apache.org.
Addressed review comments.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/a043f669
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/a043f669
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/a043f669
Branch: refs/heads/master
Commit: a043f669210607c8e33840898ba3c182ad267c15
Parents: 865756d
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Thu May 15 17:50:22 2014 +0000
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Thu May 15 17:50:22 2014 +0000
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/worker.clj | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a043f669/storm-core/src/clj/backtype/storm/daemon/worker.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/worker.clj b/storm-core/src/clj/backtype/storm/daemon/worker.clj
index b7fe1cf..921c259 100644
--- a/storm-core/src/clj/backtype/storm/daemon/worker.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/worker.clj
@@ -352,7 +352,11 @@
(touch (worker-pid-path conf worker-id (process-pid))))
(let [worker (worker-data conf shared-mq-context storm-id assignment-id port worker-id)
heartbeat-fn #(do-heartbeat worker)
-
+
+ ;; do this here so that the worker process dies if this fails
+ ;; it's important that worker heartbeat to supervisor ASAP when launching so that the supervisor knows it's running (and can move on)
+ _ (heartbeat-fn)
+
executors (atom nil)
;; launch heartbeat threads immediately so that slow-loading tasks don't cause the worker to timeout
;; to the supervisor
[4/5] git commit: Merge branch 'zk-hb-blocks' of
github.com:revans2/incubator-storm
Posted by pt...@apache.org.
Merge branch 'zk-hb-blocks' of github.com:revans2/incubator-storm
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/9e1dc591
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/9e1dc591
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/9e1dc591
Branch: refs/heads/master
Commit: 9e1dc5918618e9a23e7bc6c1b2e41cb002206230
Parents: 52fe66e a043f66
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Tue May 20 13:45:01 2014 -0400
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Tue May 20 13:45:01 2014 -0400
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/worker.clj | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[2/5] git commit: Merge branch 'apache-master' into zk-hb-blocks
Posted by pt...@apache.org.
Merge branch 'apache-master' into zk-hb-blocks
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/865756d9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/865756d9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/865756d9
Branch: refs/heads/master
Commit: 865756d969dfbda02925bc9d650a526f17c3bbbc
Parents: 0666c41 222c725
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Thu May 15 17:47:56 2014 +0000
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Thu May 15 17:47:56 2014 +0000
----------------------------------------------------------------------
CHANGELOG.md | 12 +
DEVELOPER.md | 313 +++++++++++++++++++
README.markdown | 3 +
SECURITY.md | 74 +++++
examples/storm-starter/pom.xml | 5 +
.../jvm/storm/starter/BasicDRPCTopology.java | 2 +-
.../jvm/storm/starter/ExclamationTopology.java | 2 +-
.../jvm/storm/starter/PrintSampleStream.java | 24 +-
.../src/jvm/storm/starter/ReachTopology.java | 2 +-
.../jvm/storm/starter/WordCountTopology.java | 2 +-
.../storm/starter/spout/TwitterSampleSpout.java | 222 +++++++------
.../storm/starter/trident/TridentWordCount.java | 2 +-
.../src/test/storm/kafka/KafkaTestBroker.java | 41 ++-
.../src/test/storm/kafka/KafkaUtilsTest.java | 8 +-
.../src/test/storm/kafka/ZkCoordinatorTest.java | 2 +-
logback/cluster.xml | 2 +-
pom.xml | 2 +-
.../clj/backtype/storm/command/rebalance.clj | 7 +-
.../src/clj/backtype/storm/daemon/logviewer.clj | 29 +-
storm-core/src/clj/backtype/storm/testing.clj | 5 +-
.../src/jvm/backtype/storm/StormSubmitter.java | 152 ++++++++-
.../backtype/storm/task/TopologyContext.java | 11 +-
.../jvm/storm/trident/util/TridentUtils.java | 14 +-
storm-dist/binary/pom.xml | 40 +--
storm-dist/binary/src/main/assembly/binary.xml | 5 +
storm-dist/source/pom.xml | 38 +--
26 files changed, 824 insertions(+), 195 deletions(-)
----------------------------------------------------------------------
[5/5] git commit: STORM-233: update changelog
Posted by pt...@apache.org.
STORM-233: update changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/fe7ac291
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/fe7ac291
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/fe7ac291
Branch: refs/heads/master
Commit: fe7ac29105e0425d91cb211b724d1db90a64c363
Parents: 9e1dc59
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Tue May 20 13:47:15 2014 -0400
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Tue May 20 13:47:15 2014 -0400
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/fe7ac291/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b3deae5..c6d0816 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
## 0.9.2-incubating (unreleased)
+ * STORM-233: Removed inline heartbeat to nimbus to avoid workers being killed when under heavy ZK load
* STORM-267: fix package name of LoggingMetricsConsumer in storm.yaml.example
* STORM-265: upgrade to clojure 1.5.1
* STORM-232: ship JNI dependencies with the topology jar