You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2015/03/11 21:10:12 UTC
[1/6] storm git commit: Merge pull request #4 from apache/master
Repository: storm
Updated Branches:
refs/heads/master bbc052ebb -> 3a3b7cea7
Merge pull request #4 from apache/master
Merger from apache/storm to caofangkun/apache-storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6c04e1be
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6c04e1be
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6c04e1be
Branch: refs/heads/master
Commit: 6c04e1bea802d28ae92a234ddbaad2c1ed0a6835
Parents: 0b9bc2e 01e75ec
Author: caofangkun <ca...@gmail.com>
Authored: Fri Jan 23 09:28:46 2015 +0800
Committer: caofangkun <ca...@gmail.com>
Committed: Fri Jan 23 09:28:46 2015 +0800
----------------------------------------------------------------------
.gitignore | 1 +
CHANGELOG.md | 13 +-
README.markdown | 3 +
SECURITY.md | 14 +-
bin/storm | 36 ++-
bin/storm-config.cmd | 32 +-
bin/storm.cmd | 15 +-
conf/defaults.yaml | 3 +-
docs/documentation/Powered-By.md | 11 +
.../hbase/trident/state/HBaseMapState.java | 2 +-
.../storm/hbase/trident/state/HBaseState.java | 2 +-
external/storm-kafka/README.md | 24 +-
external/storm-kafka/pom.xml | 21 +-
.../ExponentialBackoffMsgRetryManager.java | 167 ++++++++++
.../jvm/storm/kafka/FailedMsgRetryManager.java | 26 ++
.../src/jvm/storm/kafka/PartitionManager.java | 42 ++-
.../src/jvm/storm/kafka/SpoutConfig.java | 8 +
.../ExponentialBackoffMsgRetryManagerTest.java | 194 ++++++++++++
pom.xml | 6 +-
storm-buildtools/storm-maven-plugins/pom.xml | 81 +++++
.../storm/maven/plugin/util/CommandExec.java | 89 ++++++
.../plugin/versioninfo/VersionInfoMojo.java | 304 +++++++++++++++++++
storm-core/pom.xml | 39 ++-
.../src/clj/backtype/storm/daemon/nimbus.clj | 2 +-
.../clj/backtype/storm/daemon/supervisor.clj | 5 +-
storm-core/src/clj/backtype/storm/ui/core.clj | 15 +-
storm-core/src/clj/backtype/storm/util.clj | 5 +-
storm-core/src/jvm/backtype/storm/Config.java | 6 +
.../backtype/storm/messaging/netty/Server.java | 2 +
.../jvm/backtype/storm/utils/VersionInfo.java | 131 ++++++++
.../storm-core-version-info.properties | 24 ++
.../templates/topology-page-template.html | 8 +-
.../scheduler/multitenant_scheduler_test.clj | 2 +
.../test/clj/backtype/storm/supervisor_test.clj | 30 +-
storm-dist/binary/src/main/assembly/binary.xml | 7 -
35 files changed, 1264 insertions(+), 106 deletions(-)
----------------------------------------------------------------------
[2/6] storm git commit: Merge pull request #5 from apache/master
Posted by bo...@apache.org.
Merge pull request #5 from apache/master
Merge from apache/storm to caofangkun/apache-storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/d8c98cb1
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/d8c98cb1
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/d8c98cb1
Branch: refs/heads/master
Commit: d8c98cb1846bd07aef558bde410607b923aa5c1e
Parents: 6c04e1b 8491151
Author: caofangkun <ca...@gmail.com>
Authored: Wed Feb 4 09:09:22 2015 +0800
Committer: caofangkun <ca...@gmail.com>
Committed: Wed Feb 4 09:09:22 2015 +0800
----------------------------------------------------------------------
CHANGELOG.md | 8 +
README.markdown | 3 +
STORM-UI-REST-API.md | 21 +
docs/README.md | 9 +
docs/documentation/Contributing-to-Storm.md | 2 +-
docs/documentation/Home.md | 2 +-
docs/documentation/Powered-By.md | 4 +-
docs/downloads.html | 3 +
pom.xml | 13 +-
storm-core/pom.xml | 2 +-
storm-core/src/clj/backtype/storm/cluster.clj | 38 +-
.../src/clj/backtype/storm/daemon/logviewer.clj | 5 +-
.../src/clj/backtype/storm/daemon/nimbus.clj | 35 +-
.../src/clj/backtype/storm/daemon/worker.clj | 6 +-
storm-core/src/clj/backtype/storm/ui/core.clj | 34 +-
storm-core/src/clj/backtype/storm/util.clj | 6 +-
.../storm/generated/GetInfoOptions.java | 350 +++++++
.../jvm/backtype/storm/generated/Nimbus.java | 974 +++++++++++++++++++
.../storm/generated/NumErrorsChoice.java | 64 ++
.../storm/grouping/PartialKeyGrouping.java | 57 ++
.../src/jvm/backtype/storm/utils/Monitor.java | 8 +-
storm-core/src/py/storm/DistributedRPC-remote | 0
.../py/storm/DistributedRPCInvocations-remote | 0
storm-core/src/py/storm/Nimbus-remote | 7 +
storm-core/src/py/storm/Nimbus.py | 226 +++++
storm-core/src/py/storm/ttypes.py | 80 ++
storm-core/src/storm.thrift | 11 +
.../clj/backtype/storm/integration_test.clj | 10 +-
.../test/clj/backtype/storm/supervisor_test.clj | 4 +-
.../storm/grouping/PartialKeyGroupingTest.java | 46 +
30 files changed, 1985 insertions(+), 43 deletions(-)
----------------------------------------------------------------------
[4/6] storm git commit: STORM-657:use when-not in
supervisor.shutdown-worker
Posted by bo...@apache.org.
STORM-657:use when-not in supervisor.shutdown-worker
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/1d9b2837
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/1d9b2837
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/1d9b2837
Branch: refs/heads/master
Commit: 1d9b28373faeb4a4d7b9017d8ffd130f3397b052
Parents: b1c993b
Author: caofangkun <ca...@gmail.com>
Authored: Thu Feb 12 09:25:05 2015 +0800
Committer: caofangkun <ca...@gmail.com>
Committed: Thu Feb 12 09:25:05 2015 +0800
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/supervisor.clj | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/1d9b2837/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 7764310..36f7a8a 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -258,9 +258,9 @@
(if as-user
(worker-launcher-and-wait conf user ["signal" pid "9"] :log-prefix (str "kill -15 " pid))
(kill-process-with-sig-term pid)))
- (if-not (empty? pids) (do
- (log-message "Sleep " shutdown-sleep-secs " seconds for execution of cleanup threads on worker.")
- (sleep-secs shutdown-sleep-secs)))
+ (when-not (empty? pids)
+ (log-message "Sleep " shutdown-sleep-secs " seconds for execution of cleanup threads on worker.")
+ (sleep-secs shutdown-sleep-secs))
(doseq [pid pids]
(if as-user
(worker-launcher-and-wait conf user ["signal" pid "9"] :log-prefix (str "kill -9 " pid))
[3/6] storm git commit: STORM-657:make the shutdown-worker sleep time
before kill -9 configurable
Posted by bo...@apache.org.
STORM-657:make the shutdown-worker sleep time before kill -9 configurable
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/b1c993b2
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/b1c993b2
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/b1c993b2
Branch: refs/heads/master
Commit: b1c993b288385f637fa26b840a11532c7b60a6c7
Parents: d8c98cb
Author: caofangkun <ca...@gmail.com>
Authored: Mon Feb 9 13:23:23 2015 +0800
Committer: caofangkun <ca...@gmail.com>
Committed: Mon Feb 9 13:23:23 2015 +0800
----------------------------------------------------------------------
conf/defaults.yaml | 2 ++
storm-core/src/clj/backtype/storm/daemon/supervisor.clj | 5 ++++-
storm-core/src/jvm/backtype/storm/Config.java | 6 ++++++
3 files changed, 12 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/b1c993b2/conf/defaults.yaml
----------------------------------------------------------------------
diff --git a/conf/defaults.yaml b/conf/defaults.yaml
index 35d20ff..b5de229 100644
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@ -117,6 +117,8 @@ supervisor.run.worker.as.user: false
supervisor.worker.start.timeout.secs: 120
#how long between heartbeats until supervisor considers that worker dead and tries to restart it
supervisor.worker.timeout.secs: 30
+#how many seconds to sleep for before shutting down threads on worker
+supervisor.worker.shutdown.sleep.secs: 1
#how frequently the supervisor checks on the status of the processes it's monitoring and restarts if necessary
supervisor.monitor.frequency.secs: 3
#how frequently the supervisor heartbeats to the cluster state (for nimbus)
http://git-wip-us.apache.org/repos/asf/storm/blob/b1c993b2/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index a5d5aef..7764310 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -249,6 +249,7 @@
(let [conf (:conf supervisor)
pids (read-dir-contents (worker-pids-root conf id))
thread-pid (@(:worker-thread-pids-atom supervisor) id)
+ shutdown-sleep-secs (conf SUPERVISOR-WORKER-SHUTDOWN-SLEEP-SECS)
as-user (conf SUPERVISOR-RUN-WORKER-AS-USER)
user (get-worker-user conf id)]
(when thread-pid
@@ -257,7 +258,9 @@
(if as-user
(worker-launcher-and-wait conf user ["signal" pid "9"] :log-prefix (str "kill -15 " pid))
(kill-process-with-sig-term pid)))
- (if-not (empty? pids) (sleep-secs 1)) ;; allow 1 second for execution of cleanup threads on worker.
+ (if-not (empty? pids) (do
+ (log-message "Sleep " shutdown-sleep-secs " seconds for execution of cleanup threads on worker.")
+ (sleep-secs shutdown-sleep-secs)))
(doseq [pid pids]
(if as-user
(worker-launcher-and-wait conf user ["signal" pid "9"] :log-prefix (str "kill -9 " pid))
http://git-wip-us.apache.org/repos/asf/storm/blob/b1c993b2/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/Config.java b/storm-core/src/jvm/backtype/storm/Config.java
index 73a1976..2462d4e 100644
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@ -734,6 +734,12 @@ public class Config extends HashMap<String, Object> {
public static final Object SUPERVISOR_WORKER_TIMEOUT_SECS_SCHEMA = ConfigValidation.IntegerValidator;
/**
+ * How many seconds to sleep for before shutting down threads on worker
+ */
+ public static final String SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS = "supervisor.worker.shutdown.sleep.secs";
+ public static final Object SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS_SCHEMA = ConfigValidation.IntegerValidator;
+
+ /**
* How long a worker can go without heartbeating during the initial launch before
* the supervisor tries to restart the worker process. This value override
* supervisor.worker.timeout.secs during launch because there is additional
[5/6] storm git commit: Merge branch 'storm-657' of
https://github.com/caofangkun/apache-storm into STORM-657
Posted by bo...@apache.org.
Merge branch 'storm-657' of https://github.com/caofangkun/apache-storm into STORM-657
STORM-657:make the shutdown-worker sleep time before kill -9 configurable
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c8ac509b
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c8ac509b
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c8ac509b
Branch: refs/heads/master
Commit: c8ac509b026668f672e185b9f91ca11ce78881a8
Parents: bbc052e 1d9b283
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Mar 11 15:01:57 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Wed Mar 11 15:01:57 2015 -0500
----------------------------------------------------------------------
conf/defaults.yaml | 2 ++
storm-core/src/clj/backtype/storm/daemon/supervisor.clj | 5 ++++-
storm-core/src/jvm/backtype/storm/Config.java | 6 ++++++
3 files changed, 12 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/c8ac509b/conf/defaults.yaml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/c8ac509b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
[6/6] storm git commit: Added STORM-657 to Changelog
Posted by bo...@apache.org.
Added STORM-657 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/3a3b7cea
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/3a3b7cea
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/3a3b7cea
Branch: refs/heads/master
Commit: 3a3b7cea7b99abea28502fe55c4b50a5ad5f4456
Parents: c8ac509
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Mar 11 15:09:04 2015 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Wed Mar 11 15:09:04 2015 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/3a3b7cea/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4bda4f5..53ce8f2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -70,6 +70,7 @@
* STORM-554: the type of first param "topology" should be ^StormTopology not ^TopologyContext
* STORM-469: Storm UI Last Error Detail Insufficient for debugging
* STORM-656: Document "external" modules and "Committer Sponsors"
+ * STORM-657:make the shutdown-worker sleep time before kill -9 configurable
## 0.9.3-rc2
* STORM-558: change "swap!" to "reset!" to fix assignment-versions in supervisor