You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2014/08/28 22:03:13 UTC
[01/50] git commit: 1. fix test case passed on original code;
2. use count-by
Repository: incubator-storm
Updated Branches:
refs/heads/security 7a06ef722 -> b19fdbb74
1. fix test case passed on original code; 2. use count-by
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/d6a71142
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/d6a71142
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/d6a71142
Branch: refs/heads/security
Commit: d6a71142eec6f669f335312f418134960143fdbc
Parents: 3a20181
Author: Kang Xiao <kx...@gmail.com>
Authored: Thu Aug 14 00:34:06 2014 +0800
Committer: Kang Xiao <kx...@gmail.com>
Committed: Thu Aug 14 00:34:06 2014 +0800
----------------------------------------------------------------------
.../src/clj/backtype/storm/scheduler/EvenScheduler.clj | 2 +-
storm-core/test/clj/backtype/storm/scheduler_test.clj | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/d6a71142/storm-core/src/clj/backtype/storm/scheduler/EvenScheduler.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/scheduler/EvenScheduler.clj b/storm-core/src/clj/backtype/storm/scheduler/EvenScheduler.clj
index e377c13..25ba03b 100644
--- a/storm-core/src/clj/backtype/storm/scheduler/EvenScheduler.clj
+++ b/storm-core/src/clj/backtype/storm/scheduler/EvenScheduler.clj
@@ -22,7 +22,7 @@
:implements [backtype.storm.scheduler.IScheduler]))
(defn sort-slots [all-slots]
- (let [split-up (sort #(> (count %1) (count %2)) (vals (group-by first all-slots)))]
+ (let [split-up (sort-by count > (vals (group-by first all-slots)))]
(apply interleave-all split-up)
))
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/d6a71142/storm-core/test/clj/backtype/storm/scheduler_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/scheduler_test.clj b/storm-core/test/clj/backtype/storm/scheduler_test.clj
index a11e6b1..9744ac6 100644
--- a/storm-core/test/clj/backtype/storm/scheduler_test.clj
+++ b/storm-core/test/clj/backtype/storm/scheduler_test.clj
@@ -262,6 +262,10 @@
))
(deftest test-sort-slots
+ ;; test supervisor1 and supervisor2 has the same free slot
+ (is (= '(["supervisor1" 6700] ["supervisor2" 6700]
+ ["supervisor1" 6701] ["supervisor2" 6701])
+ (sort-slots [["supervisor1" 6700] ["supervisor1" 6701] ["supervisor2" 6700] ["supervisor2" 6701]])))
;; test supervisor2 has more free slots
(is (= '(["supervisor2" 6700] ["supervisor1" 6700]
["supervisor2" 6701] ["supervisor1" 6701]
@@ -278,9 +282,5 @@
["supervisor2" 6700] ["supervisor2" 6701] ["supervisor2" 6702]
["supervisor3" 6700] ["supervisor3" 6701] ["supervisor3" 6702] ["supervisor3" 6703]
])))
- ;; test supervisor1 and supervisor2 has the same free slot
- (is (= '(["supervisor1" 6700] ["supervisor2" 6700]
- ["supervisor1" 6701] ["supervisor2" 6701])
- (sort-slots [["supervisor1" 6700] ["supervisor1" 6701] ["supervisor2" 6700] ["supervisor2" 6701]])))
)
[25/50] git commit: Merged master fixed conflict.
Posted by bo...@apache.org.
Merged master fixed conflict.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/cdaa90b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/cdaa90b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/cdaa90b1
Branch: refs/heads/security
Commit: cdaa90b11601f99dc2d7bb2147dd05f48e072941
Parents: 2db902e b0f002d
Author: Sriharsha Chintalapani <ma...@harsha.io>
Authored: Wed Aug 20 15:27:36 2014 -0700
Committer: Sriharsha Chintalapani <ma...@harsha.io>
Committed: Wed Aug 20 15:27:36 2014 -0700
----------------------------------------------------------------------
----------------------------------------------------------------------
[28/50] git commit: Added STORM-449 to Changelog
Posted by bo...@apache.org.
Added STORM-449 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/0ee66316
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/0ee66316
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/0ee66316
Branch: refs/heads/security
Commit: 0ee6631635222af94719551376480de7c4d761d0
Parents: 27400ee
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:14:26 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:14:26 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/0ee66316/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0d6e8c3..f99986b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -50,6 +50,7 @@
* STORM-458: sample spout uses incorrect name when connecting bolt
* STORM-455: Report error-level messages from ShellBolt children
* STORM-443: multilang log's loglevel protocol can cause hang
+ * STORM-449: Updated ShellBolt to not exit when shutting down.
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[12/50] git commit: Added wurstmeister as contributor
Posted by bo...@apache.org.
Added wurstmeister as contributor
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/50ab15b7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/50ab15b7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/50ab15b7
Branch: refs/heads/security
Commit: 50ab15b73aaeaa5e15c9341755f20152ad3316e6
Parents: e62d3d7
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 09:53:51 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 09:53:51 2014 -0500
----------------------------------------------------------------------
README.markdown | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/50ab15b7/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 1341906..11ca1de 100644
--- a/README.markdown
+++ b/README.markdown
@@ -164,7 +164,7 @@ under the License.
* Naresh Kosgi ([@NareshKosgi](https://github.com/NareshKosgi))
* Sean T Allen ([@SeanTAllen](https://github.com/SeanTAllen))
* Gabriel Ciuloaica ([@devsprint](https://github.com/devsprint))
-
+* wurstmeister ([@wurstmeister](https://github.com/wurstmeister))
## Acknowledgements
[07/50] git commit: Merge branch 'master' of
https://github.com/SeanTAllen/incubator-storm
Posted by bo...@apache.org.
Merge branch 'master' of https://github.com/SeanTAllen/incubator-storm
Remove out of date comment
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/ab2cf9a6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/ab2cf9a6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/ab2cf9a6
Branch: refs/heads/security
Commit: ab2cf9a63bc772bb8399320a2141c20c40543a2b
Parents: 7a58c2d a1cf23d
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 18 15:36:45 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 18 15:36:45 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/worker.clj | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
[29/50] git commit: Merge branch 'STORM-464' of
https://github.com/revans2/incubator-storm into STORM-464
Posted by bo...@apache.org.
Merge branch 'STORM-464' of https://github.com/revans2/incubator-storm into STORM-464
STORM-464: Simulated time advanced after test cluster exits causes intermittent test failures
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/5171c89a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/5171c89a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/5171c89a
Branch: refs/heads/security
Commit: 5171c89a66dde90cac1a652e942801d862be6bbb
Parents: 0ee6631 8604e90
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:23:26 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:23:26 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/testing.clj | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
[31/50] git commit: Merge branch 'STORM-463' of
https://github.com/pershyn/incubator-storm into STORM-463
Posted by bo...@apache.org.
Merge branch 'STORM-463' of https://github.com/pershyn/incubator-storm into STORM-463
STORM-463: added static version of metrics helpers for Config
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/05a83c8e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/05a83c8e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/05a83c8e
Branch: refs/heads/security
Commit: 05a83c8e542642b4fa403a13ac697c2f91e47997
Parents: ee71dee acc1ae6
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:46:38 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:46:38 2014 -0500
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/Config.java | 24 ++++++++++++++++------
1 file changed, 18 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
[05/50] git commit: emoved Author tags
Posted by bo...@apache.org.
emoved Author tags
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/a0ca23ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/a0ca23ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/a0ca23ca
Branch: refs/heads/security
Commit: a0ca23ca8862500a59b4f4029f05386e3be2ced8
Parents: 0587bee
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Mon Aug 18 10:31:27 2014 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Mon Aug 18 10:31:27 2014 -0500
----------------------------------------------------------------------
.../storm/serialization/DefaultSerializationDelegate.java | 3 ---
.../storm/serialization/GzipBridgeSerializationDelegate.java | 1 -
.../backtype/storm/serialization/GzipSerializationDelegate.java | 1 -
.../jvm/backtype/storm/serialization/SerializationDelegate.java | 1 -
4 files changed, 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a0ca23ca/storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java b/storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java
index 0ad36a1..b09bbe8 100644
--- a/storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java
+++ b/storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java
@@ -24,9 +24,6 @@ import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.util.Map;
-/**
- * @author danehammer
- */
public class DefaultSerializationDelegate implements SerializationDelegate {
@Override
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a0ca23ca/storm-core/src/jvm/backtype/storm/serialization/GzipBridgeSerializationDelegate.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/serialization/GzipBridgeSerializationDelegate.java b/storm-core/src/jvm/backtype/storm/serialization/GzipBridgeSerializationDelegate.java
index 305cff7..6a7552f 100644
--- a/storm-core/src/jvm/backtype/storm/serialization/GzipBridgeSerializationDelegate.java
+++ b/storm-core/src/jvm/backtype/storm/serialization/GzipBridgeSerializationDelegate.java
@@ -24,7 +24,6 @@ import java.util.zip.GZIPInputStream;
* Always writes gzip out, but tests incoming to see if it's gzipped. If it is, deserializes with gzip. If not, uses
* {@link backtype.storm.serialization.DefaultSerializationDelegate} to deserialize. Any logic needing to be enabled
* via {@link #prepare(java.util.Map)} is passed through to both delegates.
- * @author danehammer
*/
public class GzipBridgeSerializationDelegate implements SerializationDelegate {
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a0ca23ca/storm-core/src/jvm/backtype/storm/serialization/GzipSerializationDelegate.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/serialization/GzipSerializationDelegate.java b/storm-core/src/jvm/backtype/storm/serialization/GzipSerializationDelegate.java
index e7ddd0e..e5a230f 100644
--- a/storm-core/src/jvm/backtype/storm/serialization/GzipSerializationDelegate.java
+++ b/storm-core/src/jvm/backtype/storm/serialization/GzipSerializationDelegate.java
@@ -28,7 +28,6 @@ import java.util.zip.GZIPOutputStream;
/**
* Note, this assumes it's deserializing a gzip byte stream, and will err if it encounters any other serialization.
- * @author danehammer
*/
public class GzipSerializationDelegate implements SerializationDelegate {
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a0ca23ca/storm-core/src/jvm/backtype/storm/serialization/SerializationDelegate.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/serialization/SerializationDelegate.java b/storm-core/src/jvm/backtype/storm/serialization/SerializationDelegate.java
index feb2c66..b9059b8 100644
--- a/storm-core/src/jvm/backtype/storm/serialization/SerializationDelegate.java
+++ b/storm-core/src/jvm/backtype/storm/serialization/SerializationDelegate.java
@@ -21,7 +21,6 @@ import java.util.Map;
/**
* Allow {@link backtype.storm.utils.Utils} to delegate meta serialization.
- * @author danehammer
*/
public interface SerializationDelegate {
[36/50] git commit: Added STORM-437 to Changelog
Posted by bo...@apache.org.
Added STORM-437 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/8dd170d4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/8dd170d4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/8dd170d4
Branch: refs/heads/security
Commit: 8dd170d41c8ecf935f4e7a8d263bbe3cee57d3e4
Parents: f49aa1c
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 11:17:55 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 11:17:55 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/8dd170d4/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fc5554a..d0ef90f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -54,6 +54,7 @@
* STORM-464: Simulated time advanced after test cluster exits causes intermittent test failures
* STORM-463: added static version of metrics helpers for Config
* STORM-376: Add compression to serialization
+ * STORM-437: Enforce utf-8 when multilang reads from stdin
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[42/50] git commit: Added STORM-274 to Changelog
Posted by bo...@apache.org.
Added STORM-274 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/5ddf308a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/5ddf308a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/5ddf308a
Branch: refs/heads/security
Commit: 5ddf308a8df765885410bd116017bd3132fea089
Parents: 4042b9d
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 12:51:57 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 12:51:57 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5ddf308a/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f36ebe8..8f769d8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -57,6 +57,7 @@
* STORM-437: Enforce utf-8 when multilang reads from stdin
* STORM-361: Add JSON-P support to Storm UI API
* STORM-373: Provide Additional String substitutions for *.worker.childopts
+ * STORM-274: Add support for command remoteconfvalue in storm.cmd
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[43/50] git commit: Merge branch 'STORM-132' of
https://github.com/xiaokang/incubator-storm into STORM-132
Posted by bo...@apache.org.
Merge branch 'STORM-132' of https://github.com/xiaokang/incubator-storm into STORM-132
STORM-132: sort supervisor by free slot in desending order
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/6393d31b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/6393d31b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/6393d31b
Branch: refs/heads/security
Commit: 6393d31b0be99f93213405543c8af8d6e8cd6f63
Parents: 5ddf308 a2484f3
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 13:15:57 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 13:15:57 2014 -0500
----------------------------------------------------------------------
.../backtype/storm/scheduler/EvenScheduler.clj | 2 +-
.../test/clj/backtype/storm/scheduler_test.clj | 21 ++++++++++++++++++++
2 files changed, 22 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
[19/50] git commit: STORM-463: added static version of metrics
helpers for Config
Posted by bo...@apache.org.
STORM-463: added static version of metrics helpers for Config
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/acc1ae6a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/acc1ae6a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/acc1ae6a
Branch: refs/heads/security
Commit: acc1ae6a757cf426e5917dfdd910c746442b2bc2
Parents: d4ebb6d
Author: Michael Pershyn <mi...@adition.com>
Authored: Wed Aug 20 15:06:24 2014 +0200
Committer: Michael Pershyn <mi...@adition.com>
Committed: Wed Aug 20 15:06:24 2014 +0200
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/Config.java | 24 ++++++++++++++++------
1 file changed, 18 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/acc1ae6a/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/Config.java b/storm-core/src/jvm/backtype/storm/Config.java
index ac8b6b6..e09e673 100644
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@ -921,24 +921,36 @@ public class Config extends HashMap<String, Object> {
registerSerialization(this, klass, serializerClass);
}
- public void registerMetricsConsumer(Class klass, Object argument, long parallelismHint) {
+ public static void registerMetricsConsumer(Map conf, Class klass, Object argument, long parallelismHint) {
HashMap m = new HashMap();
m.put("class", klass.getCanonicalName());
m.put("parallelism.hint", parallelismHint);
m.put("argument", argument);
- List l = (List)this.get(TOPOLOGY_METRICS_CONSUMER_REGISTER);
- if(l == null) { l = new ArrayList(); }
+ List l = (List)conf.get(TOPOLOGY_METRICS_CONSUMER_REGISTER);
+ if (l == null) { l = new ArrayList(); }
l.add(m);
- this.put(TOPOLOGY_METRICS_CONSUMER_REGISTER, l);
+ conf.put(TOPOLOGY_METRICS_CONSUMER_REGISTER, l);
+ }
+
+ public void registerMetricsConsumer(Class klass, Object argument, long parallelismHint) {
+ registerMetricsConsumer(this, klass, argument, parallelismHint);
+ }
+
+ public static void registerMetricsConsumer(Map conf, Class klass, long parallelismHint) {
+ registerMetricsConsumer(conf, klass, null, parallelismHint);
}
public void registerMetricsConsumer(Class klass, long parallelismHint) {
- registerMetricsConsumer(klass, null, parallelismHint);
+ registerMetricsConsumer(this, klass, parallelismHint);
+ }
+
+ public static void registerMetricsConsumer(Map conf, Class klass) {
+ registerMetricsConsumer(conf, klass, null, 1L);
}
public void registerMetricsConsumer(Class klass) {
- registerMetricsConsumer(klass, null, 1L);
+ registerMetricsConsumer(this, klass);
}
public static void registerDecorator(Map conf, Class<? extends IKryoDecorator> klass) {
[35/50] git commit: Merge branch 'STORM-437-multilang-utf8' of
https://github.com/forter/incubator-storm into STORM-437
Posted by bo...@apache.org.
Merge branch 'STORM-437-multilang-utf8' of https://github.com/forter/incubator-storm into STORM-437
STORM-437: Enforce utf-8 when multilang reads from stdin
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/f49aa1c3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/f49aa1c3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/f49aa1c3
Branch: refs/heads/security
Commit: f49aa1c3dc45e0ee2c74b5b86c13fa135a55b8fe
Parents: 93e712b c54539b
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 11:17:07 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 11:17:07 2014 -0500
----------------------------------------------------------------------
.../src/jvm/backtype/storm/multilang/JsonSerializer.java | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/f49aa1c3/storm-core/src/jvm/backtype/storm/multilang/JsonSerializer.java
----------------------------------------------------------------------
[44/50] git commit: Added STORM-132 to Changelog
Posted by bo...@apache.org.
Added STORM-132 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/15757694
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/15757694
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/15757694
Branch: refs/heads/security
Commit: 15757694512319540d385257af3af6d19c8b1aaa
Parents: 6393d31
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 13:16:47 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 13:16:47 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/15757694/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8f769d8..2962c47 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -58,6 +58,7 @@
* STORM-361: Add JSON-P support to Storm UI API
* STORM-373: Provide Additional String substitutions for *.worker.childopts
* STORM-274: Add support for command remoteconfvalue in storm.cmd
+ * STORM-132: sort supervisor by free slot in desending order
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[09/50] git commit: STORM-453: migrated to curator 2.5.0
Posted by bo...@apache.org.
STORM-453: migrated to curator 2.5.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/4196e5fe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/4196e5fe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/4196e5fe
Branch: refs/heads/security
Commit: 4196e5fe1757ea169a35b305f1f51a05c3dec00c
Parents: 18f283e
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 18 15:47:16 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 18 15:47:16 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
pom.xml | 4 ++--
3 files changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/4196e5fe/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c256e31..1a60c9a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -46,6 +46,7 @@
* STORM-417: Storm UI lost table sort style when tablesorter was updated
* STORM-396: Replace NullPointerException with IllegalArgumentExeption
* STORM-451: Latest storm does not build due to a pom.xml error in storm-hdfs pom.xml
+ * STORM-453: migrated to curator 2.5.0
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/4196e5fe/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index d5c3b23..47edd8e 100644
--- a/README.markdown
+++ b/README.markdown
@@ -162,6 +162,7 @@ under the License.
* Boris Aksenov ([@miofthena](https://github.com/miofthena))
* Itai Frenkel ([@itaifrenkel](https://github.com/itaifrenkel))
* Naresh Kosgi ([@NareshKosgi](https://github.com/NareshKosgi))
+* Gabriel Ciuloaica ([@devsprint](https://github.com/devsprint))
## Acknowledgements
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/4196e5fe/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 771a450..3db8b41 100644
--- a/pom.xml
+++ b/pom.xml
@@ -197,10 +197,10 @@
<clojure.tools.cli.version>0.2.4</clojure.tools.cli.version>
<disruptor.version>3.2.1</disruptor.version>
<jgrapht.version>0.9.0</jgrapht.version>
- <guava.version>14.0.1</guava.version>
+ <guava.version>16.0.1</guava.version>
<logback-classic.version>1.0.6</logback-classic.version>
<log4j-over-slf4j.version>1.6.6</log4j-over-slf4j.version>
- <netty.version>3.7.0.Final</netty.version>
+ <netty.version>3.9.0.Final</netty.version>
<clojure.tools.nrepl.version>0.2.3</clojure.tools.nrepl.version>
<clojure-complete.version>0.2.3</clojure-complete.version>
<mockito.version>1.9.5</mockito.version>
[11/50] git commit: Merge branch 'STORM-213' of
https://github.com/wurstmeister/incubator-storm into STORM-213-fix
Posted by bo...@apache.org.
Merge branch 'STORM-213' of https://github.com/wurstmeister/incubator-storm into STORM-213-fix
STORM-213: fixed NullPointerException
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/e62d3d7f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/e62d3d7f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/e62d3d7f
Branch: refs/heads/security
Commit: e62d3d7fcb34258a65f54bb2f02fa6602b38e75b
Parents: 393a21a 8391205
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 09:52:44 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 09:52:44 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/LocalCluster.clj | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
[38/50] git commit: Added STORM-361 to Changelog.
Posted by bo...@apache.org.
Added STORM-361 to Changelog.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/14d0ee5c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/14d0ee5c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/14d0ee5c
Branch: refs/heads/security
Commit: 14d0ee5c5a66e4955112b6186d564a0abf16e13e
Parents: 103e2c1
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 11:25:55 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 11:25:55 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/14d0ee5c/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d0ef90f..7a02dbd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -55,6 +55,7 @@
* STORM-463: added static version of metrics helpers for Config
* STORM-376: Add compression to serialization
* STORM-437: Enforce utf-8 when multilang reads from stdin
+ * STORM-361: Add JSON-P support to Storm UI API
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[47/50] git commit: Merge branch 'STORM-401' of
https://github.com/iwasakims/incubator-storm into STORM-401
Posted by bo...@apache.org.
Merge branch 'STORM-401' of https://github.com/iwasakims/incubator-storm into STORM-401
STORM-401: supervisor-test sometimes crashes.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/bfc7ce19
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/bfc7ce19
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/bfc7ce19
Branch: refs/heads/security
Commit: bfc7ce193da1313fb656c859ec253cb6d98d0f3e
Parents: 9760867 8081e76
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 26 16:11:37 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 26 16:11:37 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/common.clj | 3 +++
storm-core/src/clj/backtype/storm/event.clj | 3 +++
2 files changed, 6 insertions(+)
----------------------------------------------------------------------
[16/50] git commit: Added STORM-455 to Changelog and README
Posted by bo...@apache.org.
Added STORM-455 to Changelog and README
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/aa33ecd3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/aa33ecd3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/aa33ecd3
Branch: refs/heads/security
Commit: aa33ecd318a423f28fe14fde0269ee4b43dd7240
Parents: 9657468
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 11:25:17 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 11:25:17 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/aa33ecd3/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6e5e8db..680dc94 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -48,6 +48,7 @@
* STORM-451: Latest storm does not build due to a pom.xml error in storm-hdfs pom.xml
* STORM-453: migrated to curator 2.5.0
* STORM-458: sample spout uses incorrect name when connecting bolt
+ * STORM-455: Report error-level messages from ShellBolt children
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/aa33ecd3/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 8b4f544..0c7ae80 100644
--- a/README.markdown
+++ b/README.markdown
@@ -166,6 +166,7 @@ under the License.
* Gabriel Ciuloaica ([@devsprint](https://github.com/devsprint))
* wurstmeister ([@wurstmeister](https://github.com/wurstmeister))
* Marc Beitchman ([@mbeitchman](https://github.com/mbeitchman))
+* Dan Blanchard ([@dan-blanchard](https://github.com/dan-blanchard))
## Acknowledgements
[10/50] git commit: Merge branch 'STORM-453'
Posted by bo...@apache.org.
Merge branch 'STORM-453'
Conflicts:
README.markdown
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/393a21ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/393a21ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/393a21ab
Branch: refs/heads/security
Commit: 393a21aba625c1700befe209945abe09105037c2
Parents: 9d8a6f0 4196e5f
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 18 15:49:09 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 18 15:49:09 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
pom.xml | 8 ++++----
3 files changed, 6 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/393a21ab/README.markdown
----------------------------------------------------------------------
diff --cc README.markdown
index 8977354,47edd8e..1341906
--- a/README.markdown
+++ b/README.markdown
@@@ -162,7 -162,7 +162,8 @@@ under the License
* Boris Aksenov ([@miofthena](https://github.com/miofthena))
* Itai Frenkel ([@itaifrenkel](https://github.com/itaifrenkel))
* Naresh Kosgi ([@NareshKosgi](https://github.com/NareshKosgi))
+* Sean T Allen ([@SeanTAllen](https://github.com/SeanTAllen))
+ * Gabriel Ciuloaica ([@devsprint](https://github.com/devsprint))
## Acknowledgements
[40/50] git commit: Added STORM-373 to Changelog
Posted by bo...@apache.org.
Added STORM-373 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/2b4cbe91
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/2b4cbe91
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/2b4cbe91
Branch: refs/heads/security
Commit: 2b4cbe91cd4f42649c06fb75293397c8eb5de669
Parents: b7fb68c
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 11:34:51 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 11:34:51 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2b4cbe91/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7a02dbd..f36ebe8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -56,6 +56,7 @@
* STORM-376: Add compression to serialization
* STORM-437: Enforce utf-8 when multilang reads from stdin
* STORM-361: Add JSON-P support to Storm UI API
+ * STORM-373: Provide Additional String substitutions for *.worker.childopts
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[20/50] git commit: Addressed review comments
Posted by bo...@apache.org.
Addressed review comments
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/8abafb34
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/8abafb34
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/8abafb34
Branch: refs/heads/security
Commit: 8abafb34188edcc9e8845c0b24ccd3e13647fb18
Parents: ee86ad9
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Aug 20 10:56:46 2014 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Wed Aug 20 10:56:46 2014 -0500
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/task/ShellBolt.java | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/8abafb34/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/task/ShellBolt.java b/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
index 73ebfb1..c9dc980 100644
--- a/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
+++ b/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
@@ -295,13 +295,12 @@ public class ShellBolt implements IBolt {
}
private void die(Throwable exception) {
- if (_running) { //Don't exit if running tests
- String processInfo = _process.getProcessInfoString() + _process.getProcessTerminationInfoString();
- _exception = new RuntimeException(processInfo, exception);
- LOG.error("Halting process: ShellBolt died.", exception);
- _collector.reportError(exception);
+ String processInfo = _process.getProcessInfoString() + _process.getProcessTerminationInfoString();
+ _exception = new RuntimeException(processInfo, exception);
+ LOG.error("Halting process: ShellBolt died.", exception);
+ _collector.reportError(exception);
+ if (_running || (exception instanceof Error)) { //don't exit if not running, unless it is an Error
System.exit(11);
}
}
-
}
[18/50] git commit: Added STORM-443 to Changelog
Posted by bo...@apache.org.
Added STORM-443 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/d4ebb6de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/d4ebb6de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/d4ebb6de
Branch: refs/heads/security
Commit: d4ebb6de222a1f017c15104b5b8d463f6c232cd7
Parents: 0d82c2f
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 12:33:49 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 12:33:49 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/d4ebb6de/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 680dc94..0d6e8c3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -49,6 +49,7 @@
* STORM-453: migrated to curator 2.5.0
* STORM-458: sample spout uses incorrect name when connecting bolt
* STORM-455: Report error-level messages from ShellBolt children
+ * STORM-443: multilang log's loglevel protocol can cause hang
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[13/50] git commit: Merge branch 'master' of
https://github.com/mbeitchman/incubator-storm into STORM-458
Posted by bo...@apache.org.
Merge branch 'master' of https://github.com/mbeitchman/incubator-storm into STORM-458
STORM-458: sample spout uses incorrect name when connecting bolt
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/7c37b05e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/7c37b05e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/7c37b05e
Branch: refs/heads/security
Commit: 7c37b05e02c7d6aee3fd46e8fea888220f137084
Parents: 50ab15b 747765b
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 10:49:59 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 10:49:59 2014 -0500
----------------------------------------------------------------------
.../storm-starter/src/jvm/storm/starter/PrintSampleStream.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
[14/50] git commit: Updated Changelog and Readme for STORM-458
Posted by bo...@apache.org.
Updated Changelog and Readme for STORM-458
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/b607ca9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/b607ca9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/b607ca9b
Branch: refs/heads/security
Commit: b607ca9b9b641babab310144560bf253861579b2
Parents: 7c37b05
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 10:51:11 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 10:51:11 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b607ca9b/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1a60c9a..6e5e8db 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -47,6 +47,7 @@
* STORM-396: Replace NullPointerException with IllegalArgumentExeption
* STORM-451: Latest storm does not build due to a pom.xml error in storm-hdfs pom.xml
* STORM-453: migrated to curator 2.5.0
+ * STORM-458: sample spout uses incorrect name when connecting bolt
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b607ca9b/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 11ca1de..8b4f544 100644
--- a/README.markdown
+++ b/README.markdown
@@ -165,6 +165,7 @@ under the License.
* Sean T Allen ([@SeanTAllen](https://github.com/SeanTAllen))
* Gabriel Ciuloaica ([@devsprint](https://github.com/devsprint))
* wurstmeister ([@wurstmeister](https://github.com/wurstmeister))
+* Marc Beitchman ([@mbeitchman](https://github.com/mbeitchman))
## Acknowledgements
[32/50] git commit: Added STORM-463 to Changelog and Readme.
Posted by bo...@apache.org.
Added STORM-463 to Changelog and Readme.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/2c46e4c6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/2c46e4c6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/2c46e4c6
Branch: refs/heads/security
Commit: 2c46e4c6d76462100b746041c132fb30767b47e6
Parents: 05a83c8
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:48:15 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:48:15 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2c46e4c6/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9d2b0fe..9ed1a08 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -52,6 +52,7 @@
* STORM-443: multilang log's loglevel protocol can cause hang
* STORM-449: Updated ShellBolt to not exit when shutting down.
* STORM-464: Simulated time advanced after test cluster exits causes intermittent test failures
+ * STORM-463: added static version of metrics helpers for Config
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2c46e4c6/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 0c7ae80..341eb31 100644
--- a/README.markdown
+++ b/README.markdown
@@ -167,6 +167,7 @@ under the License.
* wurstmeister ([@wurstmeister](https://github.com/wurstmeister))
* Marc Beitchman ([@mbeitchman](https://github.com/mbeitchman))
* Dan Blanchard ([@dan-blanchard](https://github.com/dan-blanchard))
+* Mykhailo "Michael" Pershyn ([@pershyn](https://github.com/pershyn))
## Acknowledgements
[37/50] git commit: Merge branch 'STORM-361-V2' of
https://github.com/harshach/incubator-storm into STORM-361
Posted by bo...@apache.org.
Merge branch 'STORM-361-V2' of https://github.com/harshach/incubator-storm into STORM-361
STORM-361: Add JSON-P support to Storm UI API
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/103e2c19
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/103e2c19
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/103e2c19
Branch: refs/heads/security
Commit: 103e2c19ec10375649d536363dde6ad644bd8f44
Parents: 8dd170d cdaa90b
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 11:25:21 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 11:25:21 2014 -0500
----------------------------------------------------------------------
STORM-UI-REST-API.md | 1 +
storm-core/src/clj/backtype/storm/ui/core.clj | 42 +++++++++++++---------
2 files changed, 26 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
[27/50] git commit: Merge branch 'STORM-449' of
https://github.com/revans2/incubator-storm into STORM-449
Posted by bo...@apache.org.
Merge branch 'STORM-449' of https://github.com/revans2/incubator-storm into STORM-449
STORM-449: Updated ShellBolt to not exit when shutting down.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/27400ee3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/27400ee3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/27400ee3
Branch: refs/heads/security
Commit: 27400ee39d99909abb791eb732714ba4265d40be
Parents: d4ebb6d 8abafb3
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:13:57 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:13:57 2014 -0500
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/task/ShellBolt.java | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/27400ee3/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
----------------------------------------------------------------------
[04/50] git commit: Remove out of date comment
Posted by bo...@apache.org.
Remove out of date comment
per worker heartbeating was introduced in Storm 0.8.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/a1cf23d8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/a1cf23d8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/a1cf23d8
Branch: refs/heads/security
Commit: a1cf23d81eded0da5183c61912908761226732c9
Parents: 9968b8b
Author: Sean T Allen <se...@monkeysnatchbanana.com>
Authored: Sat Aug 16 10:45:31 2014 -0400
Committer: Sean T Allen <se...@monkeysnatchbanana.com>
Committed: Sat Aug 16 10:45:31 2014 -0400
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/worker.clj | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a1cf23d8/storm-core/src/clj/backtype/storm/daemon/worker.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/worker.clj b/storm-core/src/clj/backtype/storm/daemon/worker.clj
index aeabdf6..f1c4d3a 100644
--- a/storm-core/src/clj/backtype/storm/daemon/worker.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/worker.clj
@@ -351,7 +351,6 @@
;; deducable from cluster state (by searching through assignments)
;; what about if there's inconsistency in assignments? -> but nimbus
;; should guarantee this consistency
-;; TODO: consider doing worker heartbeating rather than task heartbeating to reduce the load on zookeeper
(defserverfn mk-worker [conf shared-mq-context storm-id assignment-id port worker-id]
(log-message "Launching worker for " storm-id " on " assignment-id ":" port " with id " worker-id
" and conf " conf)
[30/50] git commit: Added STORM-464 to Changelog
Posted by bo...@apache.org.
Added STORM-464 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/ee71deee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/ee71deee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/ee71deee
Branch: refs/heads/security
Commit: ee71deeea40159a77175e0668b6fb0b70e5d3237
Parents: 5171c89
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:24:08 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:24:08 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ee71deee/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f99986b..9d2b0fe 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -51,6 +51,7 @@
* STORM-455: Report error-level messages from ShellBolt children
* STORM-443: multilang log's loglevel protocol can cause hang
* STORM-449: Updated ShellBolt to not exit when shutting down.
+ * STORM-464: Simulated time advanced after test cluster exits causes intermittent test failures
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[45/50] git commit: Merge branch 'wip/improve_completeable_exception'
of https://github.com/dgolombek/incubator-storm into STORM-472
Posted by bo...@apache.org.
Merge branch 'wip/improve_completeable_exception' of https://github.com/dgolombek/incubator-storm into STORM-472
STORM-472: Improve error message for non-completeable testing spouts
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/3fcaf696
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/3fcaf696
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/3fcaf696
Branch: refs/heads/security
Commit: 3fcaf69637620173baddc13637bf0ba3289b254d
Parents: 1575769 2058f40
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 15:50:12 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 15:50:12 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/testing.clj | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/3fcaf696/storm-core/src/clj/backtype/storm/testing.clj
----------------------------------------------------------------------
[06/50] git commit: Merge branch 'curator-2.5.0' of
https://github.com/devsprint/incubator-storm into STORM-453
Posted by bo...@apache.org.
Merge branch 'curator-2.5.0' of https://github.com/devsprint/incubator-storm into STORM-453
STORM-453: migrated to curator 2.5.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/18f283e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/18f283e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/18f283e3
Branch: refs/heads/security
Commit: 18f283e329920696439c8766155f3291fc339dcc
Parents: 7a58c2d 99b914c
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 18 14:48:50 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 18 14:48:50 2014 -0500
----------------------------------------------------------------------
pom.xml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/18f283e3/pom.xml
----------------------------------------------------------------------
diff --cc pom.xml
index a827d1d,c7f8199..771a450
--- a/pom.xml
+++ b/pom.xml
@@@ -195,12 -194,12 +195,12 @@@
<snakeyaml.version>1.11</snakeyaml.version>
<httpclient.version>4.3.3</httpclient.version>
<clojure.tools.cli.version>0.2.4</clojure.tools.cli.version>
- <disruptor.version>2.10.1</disruptor.version>
+ <disruptor.version>3.2.1</disruptor.version>
<jgrapht.version>0.9.0</jgrapht.version>
- <guava.version>13.0</guava.version>
+ <guava.version>14.0.1</guava.version>
<logback-classic.version>1.0.6</logback-classic.version>
<log4j-over-slf4j.version>1.6.6</log4j-over-slf4j.version>
- <netty.version>3.6.3.Final</netty.version>
+ <netty.version>3.7.0.Final</netty.version>
<clojure.tools.nrepl.version>0.2.3</clojure.tools.nrepl.version>
<clojure-complete.version>0.2.3</clojure-complete.version>
<mockito.version>1.9.5</mockito.version>
[08/50] git commit: Added SeanTAllen to README
Posted by bo...@apache.org.
Added SeanTAllen to README
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/9d8a6f0d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/9d8a6f0d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/9d8a6f0d
Branch: refs/heads/security
Commit: 9d8a6f0d17bd15799e32a10844ade455d51d6aef
Parents: ab2cf9a
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 18 15:37:34 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 18 15:37:34 2014 -0500
----------------------------------------------------------------------
README.markdown | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/9d8a6f0d/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index d5c3b23..8977354 100644
--- a/README.markdown
+++ b/README.markdown
@@ -162,6 +162,7 @@ under the License.
* Boris Aksenov ([@miofthena](https://github.com/miofthena))
* Itai Frenkel ([@itaifrenkel](https://github.com/itaifrenkel))
* Naresh Kosgi ([@NareshKosgi](https://github.com/NareshKosgi))
+* Sean T Allen ([@SeanTAllen](https://github.com/SeanTAllen))
## Acknowledgements
[50/50] git commit: Fixed some merge issues.
Posted by bo...@apache.org.
Fixed some merge issues.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/b19fdbb7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/b19fdbb7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/b19fdbb7
Branch: refs/heads/security
Commit: b19fdbb7444e9d3ea38ede9d186055319df4751d
Parents: 84b7dea
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Thu Aug 28 15:02:27 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Thu Aug 28 15:02:27 2014 -0500
----------------------------------------------------------------------
pom.xml | 5 +++++
storm-core/pom.xml | 4 ++++
storm-core/src/clj/backtype/storm/daemon/logviewer.clj | 4 +---
storm-core/src/ui/public/templates/index-page-template.html | 2 +-
storm-core/src/ui/public/templates/topology-page-template.html | 2 --
5 files changed, 11 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b19fdbb7/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 7d31916..2ee56fc 100644
--- a/pom.xml
+++ b/pom.xml
@@ -397,6 +397,11 @@
<version>${jetty.version}</version>
</dependency>
<dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-servlets</artifactId>
+ <version>${jetty.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.clojure</groupId>
<artifactId>tools.logging</artifactId>
<version>${clojure.tools.logging.version}</version>
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b19fdbb7/storm-core/pom.xml
----------------------------------------------------------------------
diff --git a/storm-core/pom.xml b/storm-core/pom.xml
index 1a676f1..3785858 100644
--- a/storm-core/pom.xml
+++ b/storm-core/pom.xml
@@ -64,6 +64,10 @@
<artifactId>jetty-servlet</artifactId>
</dependency>
<dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-servlets</artifactId>
+ </dependency>
+ <dependency>
<groupId>org.clojure</groupId>
<artifactId>tools.logging</artifactId>
</dependency>
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b19fdbb7/storm-core/src/clj/backtype/storm/daemon/logviewer.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/logviewer.clj b/storm-core/src/clj/backtype/storm/daemon/logviewer.clj
index 36a737f..c3868eb 100644
--- a/storm-core/src/clj/backtype/storm/daemon/logviewer.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/logviewer.clj
@@ -348,15 +348,13 @@ Note that if anything goes wrong, this will throw an Error and exit."
:filter-params (or (conf UI-FILTER-PARAMS) {})}]
[])
filters-confs (concat filters-confs
- [{:filter-class "org.mortbay.servlet.GzipFilter"
+ [{:filter-class "org.eclipse.jetty.servlets.GzipFilter"
:filter-name "Gzipper"
:filter-params {}}])]
(run-jetty middle
{:port (int (conf LOGVIEWER-PORT))
:join? false
:configurator (fn [server]
- (doseq [connector (.getConnectors server)]
- (.setHeaderBufferSize connector header-buffer-size))
(config-filter server middle filters-confs))}))
(catch Exception ex
(log-error ex))))
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b19fdbb7/storm-core/src/ui/public/templates/index-page-template.html
----------------------------------------------------------------------
diff --git a/storm-core/src/ui/public/templates/index-page-template.html b/storm-core/src/ui/public/templates/index-page-template.html
index 8f1c29b..dfe9446 100644
--- a/storm-core/src/ui/public/templates/index-page-template.html
+++ b/storm-core/src/ui/public/templates/index-page-template.html
@@ -53,8 +53,8 @@
<td>{{uptime}}</td>
<td>{{workersTotal}}</td>
<td>{{executorsTotal}}</td>
- <td>{{schedulerInfo}}</td>
<td>{{tasksTotal}}</td>
+ <td>{{schedulerInfo}}</td>
</tr>
{{/topologies}}
</tbody>
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b19fdbb7/storm-core/src/ui/public/templates/topology-page-template.html
----------------------------------------------------------------------
diff --git a/storm-core/src/ui/public/templates/topology-page-template.html b/storm-core/src/ui/public/templates/topology-page-template.html
index ddf3c17..d442107 100644
--- a/storm-core/src/ui/public/templates/topology-page-template.html
+++ b/storm-core/src/ui/public/templates/topology-page-template.html
@@ -160,7 +160,6 @@
</script>
<script id="topology-actions-template" type="text/html">
- {{#uiActionsEnabled}}
<h2>Topology actions</h2>
<p id="topology-actions">
<input {{activateStatus}} onclick="confirmAction('{{id}}', '{{name}}', 'activate', false, 0)" type="button" value="Activate">
@@ -168,5 +167,4 @@
<input {{rebalanceStatus}} onclick="confirmAction('{{id}}', '{{name}}', 'rebalance', true, {{msgTimeout}})" type="button" value="Rebalance">
<input {{killStatus}} onclick="confirmAction('{{id}}', '{{name}}', 'kill', true, 30)" type="button" value="Kill">
</p>
- {{/uiActionsEnabled}}
</script>
[39/50] git commit: Merge branch 'childopts-replacements' of
https://github.com/kishorvpatil/incubator-storm into STORM-373
Posted by bo...@apache.org.
Merge branch 'childopts-replacements' of https://github.com/kishorvpatil/incubator-storm into STORM-373
STORM-373: Provide Additional String substitutions for *.worker.childopts
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/b7fb68c5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/b7fb68c5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/b7fb68c5
Branch: refs/heads/security
Commit: b7fb68c5cb7866c63af437a8a7d63217517393d0
Parents: 14d0ee5 1d84194
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 11:34:20 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 11:34:20 2014 -0500
----------------------------------------------------------------------
.../clj/backtype/storm/daemon/supervisor.clj | 24 +++++---
storm-core/src/jvm/backtype/storm/Config.java | 8 ++-
.../test/clj/backtype/storm/supervisor_test.clj | 60 ++++++++++++++++++++
3 files changed, 83 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b7fb68c5/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b7fb68c5/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b7fb68c5/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
[49/50] git commit: Merge branch 'master' into security-upmerge
Posted by bo...@apache.org.
Merge branch 'master' into security-upmerge
Conflicts:
README.markdown
conf/defaults.yaml
pom.xml
storm-core/src/clj/backtype/storm/daemon/common.clj
storm-core/src/clj/backtype/storm/daemon/supervisor.clj
storm-core/src/clj/backtype/storm/ui/core.clj
storm-core/src/jvm/backtype/storm/Config.java
storm-core/src/jvm/backtype/storm/utils/Utils.java
storm-core/test/clj/backtype/storm/supervisor_test.clj
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/84b7deaa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/84b7deaa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/84b7deaa
Branch: refs/heads/security
Commit: 84b7deaa5badc07b7ea8a88fef589577c837f8ee
Parents: 7a06ef7 d9b68ab
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Thu Aug 28 13:48:29 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Thu Aug 28 13:48:29 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 15 +++
README.markdown | 7 ++
STORM-UI-REST-API.md | 1 +
bin/storm.cmd | 10 +-
conf/defaults.yaml | 1 +
.../jvm/storm/starter/PrintSampleStream.java | 4 +-
pom.xml | 8 +-
.../src/clj/backtype/storm/LocalCluster.clj | 4 +-
.../src/clj/backtype/storm/daemon/common.clj | 3 +
.../clj/backtype/storm/daemon/supervisor.clj | 32 +++--
.../src/clj/backtype/storm/daemon/worker.clj | 1 -
storm-core/src/clj/backtype/storm/event.clj | 3 +
.../backtype/storm/scheduler/EvenScheduler.clj | 2 +-
storm-core/src/clj/backtype/storm/testing.clj | 9 +-
storm-core/src/clj/backtype/storm/ui/core.clj | 40 ++++---
storm-core/src/jvm/backtype/storm/Config.java | 42 +++++--
.../storm/multilang/JsonSerializer.java | 14 ++-
.../DefaultSerializationDelegate.java | 61 ++++++++++
.../GzipBridgeSerializationDelegate.java | 64 ++++++++++
.../GzipSerializationDelegate.java | 68 +++++++++++
.../serialization/SerializationDelegate.java | 35 ++++++
.../src/jvm/backtype/storm/task/ShellBolt.java | 9 +-
.../src/jvm/backtype/storm/utils/Utils.java | 60 ++++++----
.../test/clj/backtype/storm/scheduler_test.clj | 21 ++++
.../test/clj/backtype/storm/supervisor_test.clj | 118 ++++++++-----------
.../GzipBridgeSerializationDelegateTest.java | 82 +++++++++++++
26 files changed, 555 insertions(+), 159 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/README.markdown
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/conf/defaults.yaml
----------------------------------------------------------------------
diff --cc conf/defaults.yaml
index 3e00b3d,0050227..2601bee
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@@ -37,16 -35,8 +37,17 @@@ storm.zookeeper.auth.password: nul
storm.cluster.mode: "distributed" # can be distributed or local
storm.local.mode.zmq: false
storm.thrift.transport: "backtype.storm.security.auth.SimpleTransportPlugin"
+storm.principal.tolocal: "backtype.storm.security.auth.DefaultPrincipalToLocal"
+storm.group.mapping.service: "backtype.storm.security.auth.ShellBasedGroupsMapping"
storm.messaging.transport: "backtype.storm.messaging.netty.Context"
+storm.nimbus.retry.times: 5
+storm.nimbus.retry.interval.millis: 2000
+storm.nimbus.retry.intervalceiling.millis: 60000
+storm.auth.simple-white-list.users: []
+storm.auth.simple-acl.users: []
+storm.auth.simple-acl.users.commands: []
+storm.auth.simple-acl.admins: []
+ storm.meta.serialization.delegate: "backtype.storm.serialization.DefaultSerializationDelegate"
### nimbus.* configs are for the master
nimbus.host: "localhost"
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/pom.xml
----------------------------------------------------------------------
diff --cc pom.xml
index ba3d1ce,3db8b41..7d31916
--- a/pom.xml
+++ b/pom.xml
@@@ -186,10 -186,9 +186,10 @@@
<commons-lang.version>2.5</commons-lang.version>
<commons-exec.version>1.1</commons-exec.version>
<clj-time.version>0.4.1</clj-time.version>
- <curator.version>2.4.0</curator.version>
+ <curator.version>2.5.0</curator.version>
<json-simple.version>1.1</json-simple.version>
- <ring.version>0.3.11</ring.version>
+ <ring.version>1.3.0</ring.version>
+ <jetty.version>7.6.13.v20130916</jetty.version>
<clojure.tools.logging.version>0.2.3</clojure.tools.logging.version>
<clojure.math.numeric-tower.version>0.0.1</clojure.math.numeric-tower.version>
<carbonite.version>1.4.0</carbonite.version>
@@@ -207,10 -206,7 +207,10 @@@
<mockito.version>1.9.5</mockito.version>
<reply.version>0.3.0</reply.version>
<conjure.version>2.1.3</conjure.version>
- <zookeeper.version>3.4.5</zookeeper.version>
+ <zookeeper.version>3.4.6</zookeeper.version>
+ <conjure.version>2.1.3</conjure.version>
+ <clojure-data-codec.version>0.1.0</clojure-data-codec.version>
+ <clojure-contrib.version>1.2.0</clojure-contrib.version>
</properties>
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/clj/backtype/storm/LocalCluster.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/clj/backtype/storm/daemon/common.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/daemon/common.clj
index cf0549c,0b45535..f091dfb
--- a/storm-core/src/clj/backtype/storm/daemon/common.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/common.clj
@@@ -21,7 -21,7 +21,8 @@@
(:import [backtype.storm.task WorkerTopologyContext])
(:import [backtype.storm Constants])
(:import [backtype.storm.metric SystemBolt])
+ (:import [backtype.storm.security.auth IAuthorizer])
+ (:import [java.io InterruptedIOException])
(:require [clojure.set :as set])
(:require [backtype.storm.daemon.acker :as acker])
(:require [backtype.storm.thrift :as thrift])
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/clj/backtype/storm/daemon/worker.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/clj/backtype/storm/testing.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/clj/backtype/storm/ui/core.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/ui/core.clj
index 0aea7ec,9937607..d0d71c3
--- a/storm-core/src/clj/backtype/storm/ui/core.clj
+++ b/storm-core/src/clj/backtype/storm/ui/core.clj
@@@ -850,42 -816,38 +850,50 @@@
[sys?]
(if (or (nil? sys?) (= "false" sys?)) false true))
- (defn json-response [data & [status]]
- {:status (or status 200)
- :headers {"Content-Type" "application/json"}
- :body (to-json data)})
+ (defn wrap-json-in-callback [callback response]
+ (str callback "(" response ");"))
+
+ (defnk json-response
+ [data callback :serialize-fn to-json :status 200]
+ {:status status
+ :headers (if (not-nil? callback) {"Content-Type" "application/javascript"}
+ {"Content-Type" "application/json"})
+ :body (if (not-nil? callback)
+ (wrap-json-in-callback callback (serialize-fn data))
+ (serialize-fn data))})
+(def http-creds-handler (AuthUtils/GetUiHttpCredentialsPlugin *STORM-CONF*))
+
(defroutes main-routes
- (GET "/api/v1/cluster/configuration" []
- (cluster-configuration))
- (GET "/api/v1/cluster/summary" [:as {:keys [cookies servlet-request]}]
+ (GET "/api/v1/cluster/configuration" [& m]
+ (json-response (cluster-configuration)
+ (:callback m) :serialize-fn identity))
- (GET "/api/v1/cluster/summary" [& m]
- (json-response (cluster-summary) (:callback m)))
- (GET "/api/v1/supervisor/summary" [& m]
++ (GET "/api/v1/cluster/summary" [:as {:keys [cookies servlet-request]} & m]
+ (let [user (.getUserName http-creds-handler servlet-request)]
+ (assert-authorized-user servlet-request "getClusterInfo")
- (json-response (cluster-summary user))))
- (GET "/api/v1/supervisor/summary" [:as {:keys [cookies servlet-request]}]
++ (json-response (cluster-summary user) (:callback m))))
++ (GET "/api/v1/supervisor/summary" [:as {:keys [cookies servlet-request]} & m]
+ (assert-authorized-user servlet-request "getClusterInfo")
- (json-response (supervisor-summary)))
- (GET "/api/v1/topology/summary" [:as {:keys [cookies servlet-request]}]
+ (json-response (supervisor-summary) (:callback m)))
- (GET "/api/v1/topology/summary" [& m]
++ (GET "/api/v1/topology/summary" [:as {:keys [cookies servlet-request]} & m]
+ (assert-authorized-user servlet-request "getClusterInfo")
- (json-response (all-topologies-summary)))
+ (json-response (all-topologies-summary) (:callback m)))
- (GET "/api/v1/topology/:id" [id & m]
- (let [id (url-decode id)]
- (json-response (topology-page id (:window m) (check-include-sys? (:sys m))) (:callback m))))
+ (GET "/api/v1/topology/:id" [:as {:keys [cookies servlet-request]} id & m]
+ (let [id (url-decode id)
+ user (.getUserName http-creds-handler servlet-request)]
+ (assert-authorized-user servlet-request "getTopology" (topology-config id))
- (json-response (topology-page id (:window m) (check-include-sys? (:sys m)) user))))
++ (json-response (topology-page id (:window m) (check-include-sys? (:sys m)) user) (:callback m))))
(GET "/api/v1/topology/:id/visualization" [:as {:keys [cookies servlet-request]} id & m]
- (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys m))) (:callback m)))
- (GET "/api/v1/topology/:id/component/:component" [id component & m]
+ (let [id (url-decode id)]
+ (assert-authorized-user servlet-request "getTopology" (topology-config id))
- (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys m))))))
++ (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys m))) (:callback m))))
+ (GET "/api/v1/topology/:id/component/:component" [:as {:keys [cookies servlet-request]} id component & m]
(let [id (url-decode id)
- component (url-decode component)]
- (json-response (component-page id component (:window m) (check-include-sys? (:sys m))) (:callback m))))
- (POST "/api/v1/topology/:id/activate" [id]
+ component (url-decode component)
+ user (.getUserName http-creds-handler servlet-request)]
+ (assert-authorized-user servlet-request "getTopology" (topology-config id))
- (json-response (component-page id component (:window m) (check-include-sys? (:sys m)) user))))
++ (json-response (component-page id component (:window m) (check-include-sys? (:sys m)) user) (:callback m))))
+ (POST "/api/v1/topology/:id/activate" [:as {:keys [cookies servlet-request]} id]
(with-nimbus nimbus
(let [id (url-decode id)
tplg (.getTopologyInfo ^Nimbus$Client nimbus id)
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --cc storm-core/src/jvm/backtype/storm/Config.java
index c15c45d,8407010..a38b51e
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@@ -102,14 -95,15 +102,21 @@@ public class Config extends HashMap<Str
*/
public static final String STORM_NETTY_FLUSH_CHECK_INTERVAL_MS = "storm.messaging.netty.flush.check.interval.ms";
public static final Object STORM_NETTY_FLUSH_CHECK_INTERVAL_MS_SCHEMA = ConfigValidation.IntegerValidator;
-
+
+ /**
+ * Netty based messaging: Is authentication required for Netty messaging from client worker process to server worker process.
+ */
+ public static final String STORM_MESSAGING_NETTY_AUTHENTICATION = "storm.messaging.netty.authentication";
+ public static final Object STORM_MESSAGING_NETTY_AUTHENTICATION_SCHEMA = Boolean.class;
+
/**
+ * The delegate for serializing metadata, should be used for serialized objects stored in zookeeper and on disk.
+ * This is NOT used for compressing serialized tuples sent between topologies.
+ */
+ public static final String STORM_META_SERIALIZATION_DELEGATE = "storm.meta.serialization.delegate";
+ public static final Object STORM_META_SERIALIZATION_DELEGATE_SCHEMA = String.class;
+
+ /**
* A list of hosts of ZooKeeper servers used to manage the cluster.
*/
public static final String STORM_ZOOKEEPER_SERVERS = "storm.zookeeper.servers";
@@@ -746,22 -473,12 +753,25 @@@
public static final Object SUPERVISOR_MONITOR_FREQUENCY_SECS_SCHEMA = ConfigValidation.IntegerValidator;
/**
+ * Should the supervior try to run the worker as the lauching user or not. Defaults to false.
+ */
+ public static final String SUPERVISOR_RUN_WORKER_AS_USER = "supervisor.run.worker.as.user";
+ public static final Object SUPERVISOR_RUN_WORKER_AS_USER_SCHEMA = Boolean.class;
+
+ /**
+ * Full path to the worker-laucher executable that will be used to lauch workers when
+ * SUPERVISOR_RUN_WORKER_AS_USER is set to true.
+ */
+ public static final String SUPERVISOR_WORKER_LAUNCHER = "supervisor.worker.launcher";
+ public static final Object SUPERVISOR_WORKER_LAUNCHER_SCHEMA = String.class;
+
+ /**
- * The jvm opts provided to workers launched by this supervisor. All "%ID%" substrings are replaced
- * with an identifier for this worker. Also, "%WORKER-ID%", "%STORM-ID%" and "%WORKER-PORT%" are
- * replaced with appropriate runtime values for this worker.
+ * The jvm opts provided to workers launched by this supervisor. All "%ID%", "%WORKER-ID%", "%TOPOLOGY-ID%"
+ * and "%WORKER-PORT%" substrings are replaced with:
+ * %ID% -> port (for backward compatibility),
+ * %WORKER-ID% -> worker-id,
+ * %TOPOLOGY-ID% -> topology-id,
+ * %WORKER-PORT% -> port.
*/
public static final String WORKER_CHILDOPTS = "worker.childopts";
public static final Object WORKER_CHILDOPTS_SCHEMA = ConfigValidation.StringOrStringListValidator;
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/src/jvm/backtype/storm/utils/Utils.java
----------------------------------------------------------------------
diff --cc storm-core/src/jvm/backtype/storm/utils/Utils.java
index fff91e6,a657d2b..6d31d28
--- a/storm-core/src/jvm/backtype/storm/utils/Utils.java
+++ b/storm-core/src/jvm/backtype/storm/utils/Utils.java
@@@ -478,66 -447,24 +469,87 @@@ public class Utils
return false;
}
+ /**
+ * Is the cluster configured to interact with ZooKeeper in a secure way?
+ * This only works when called from within Nimbus or a Supervisor process.
+ * @param conf the storm configuration, not the topology configuration
+ * @return true if it is configured else false.
+ */
+ public static boolean isZkAuthenticationConfiguredStormServer(Map conf) {
+ return null != System.getProperty("java.security.auth.login.config")
+ || (conf != null
+ && conf.get(Config.STORM_ZOOKEEPER_AUTH_SCHEME) != null
+ && ! ((String)conf.get(Config.STORM_ZOOKEEPER_AUTH_SCHEME)).isEmpty());
+ }
+
+ /**
+ * Is the topology configured to have ZooKeeper authentication.
+ * @param conf the topology configuration
+ * @return true if ZK is configured else false
+ */
+ public static boolean isZkAuthenticationConfiguredTopology(Map conf) {
+ return (conf != null
+ && conf.get(Config.STORM_ZOOKEEPER_TOPOLOGY_AUTH_SCHEME) != null
+ && ! ((String)conf.get(Config.STORM_ZOOKEEPER_TOPOLOGY_AUTH_SCHEME)).isEmpty());
+ }
+
+ public static List<ACL> getWorkerACL(Map conf) {
+ //This is a work around to an issue with ZK where a sasl super user is not super unless there is an open SASL ACL so we are trying to give the correct perms
+ if (!isZkAuthenticationConfiguredTopology(conf)) {
+ return null;
+ }
+ String stormZKUser = (String)conf.get(Config.STORM_ZOOKEEPER_SUPERACL);
+ if (stormZKUser == null) {
+ throw new IllegalArgumentException("Authentication is enabled but "+Config.STORM_ZOOKEEPER_SUPERACL+" is not set");
+ }
+ String[] split = stormZKUser.split(":",2);
+ if (split.length != 2) {
+ throw new IllegalArgumentException(Config.STORM_ZOOKEEPER_SUPERACL+" does not appear to be in the form scheme:acl, i.e. sasl:storm-user");
+ }
+ ArrayList<ACL> ret = new ArrayList<ACL>(ZooDefs.Ids.CREATOR_ALL_ACL);
+ ret.add(new ACL(ZooDefs.Perms.ALL, new Id(split[0], split[1])));
+ return ret;
+ }
+
+ public static String threadDump() {
+ final StringBuilder dump = new StringBuilder();
+ final java.lang.management.ThreadMXBean threadMXBean = java.lang.management.ManagementFactory.getThreadMXBean();
+ final java.lang.management.ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds(), 100);
+ for (java.lang.management.ThreadInfo threadInfo : threadInfos) {
+ dump.append('"');
+ dump.append(threadInfo.getThreadName());
+ dump.append("\" ");
+ final Thread.State state = threadInfo.getThreadState();
+ dump.append("\n java.lang.Thread.State: ");
+ dump.append(state);
+ final StackTraceElement[] stackTraceElements = threadInfo.getStackTrace();
+ for (final StackTraceElement stackTraceElement : stackTraceElements) {
+ dump.append("\n at ");
+ dump.append(stackTraceElement);
+ }
+ dump.append("\n\n");
+ }
+ return dump.toString();
+ }
++
+ // Assumes caller is synchronizing
+ private static SerializationDelegate getSerializationDelegate(Map stormConf) {
+ String delegateClassName = (String)stormConf.get(Config.STORM_META_SERIALIZATION_DELEGATE);
+ SerializationDelegate delegate;
+ try {
+ Class delegateClass = Class.forName(delegateClassName);
+ delegate = (SerializationDelegate) delegateClass.newInstance();
+ } catch (ClassNotFoundException e) {
+ LOG.error("Failed to construct serialization delegate, falling back to default", e);
+ delegate = new DefaultSerializationDelegate();
+ } catch (InstantiationException e) {
+ LOG.error("Failed to construct serialization delegate, falling back to default", e);
+ delegate = new DefaultSerializationDelegate();
+ } catch (IllegalAccessException e) {
+ LOG.error("Failed to construct serialization delegate, falling back to default", e);
+ delegate = new DefaultSerializationDelegate();
+ }
+ delegate.prepare(stormConf);
+ return delegate;
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/84b7deaa/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --cc storm-core/test/clj/backtype/storm/supervisor_test.clj
index 6f02688,5abfa89..c94688f
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@@ -463,94 -354,35 +463,90 @@@
;; TODO just do reassign, and check that cleans up worker states after killing but doesn't get rid of downloaded code
)
+(deftest test-supervisor-data-acls
+ (testing "supervisor-data uses correct ACLs"
+ (let [scheme "digest"
+ digest "storm:thisisapoorpassword"
+ auth-conf {STORM-ZOOKEEPER-AUTH-SCHEME scheme
+ STORM-ZOOKEEPER-AUTH-PAYLOAD digest}
+ expected-acls supervisor/SUPERVISOR-ZK-ACLS
+ fake-isupervisor (reify ISupervisor
+ (getSupervisorId [this] nil)
+ (getAssignmentId [this] nil))]
+ (stubbing [uptime-computer nil
+ cluster/mk-storm-cluster-state nil
+ supervisor-state nil
+ local-hostname nil
+ mk-timer nil]
+ (supervisor/supervisor-data auth-conf nil fake-isupervisor)
+ (verify-call-times-for cluster/mk-storm-cluster-state 1)
+ (verify-first-call-args-for-indices cluster/mk-storm-cluster-state [2]
+ expected-acls)))))
+
+(deftest test-write-log-metadata
+ (testing "supervisor writes correct data to logs metadata file"
+ (let [exp-owner "alice"
+ exp-worker-id "42"
+ exp-storm-id "0123456789"
+ exp-port 4242
+ exp-logs-users ["bob" "charlie" "daryl"]
+ storm-conf {TOPOLOGY-SUBMITTER-USER "alice"
+ TOPOLOGY-USERS ["charlie" "bob"]
+ LOGS-USERS ["daryl"]}
+ exp-data {TOPOLOGY-SUBMITTER-USER exp-owner
+ "worker-id" exp-worker-id
+ LOGS-USERS exp-logs-users}
+ conf {}]
+ (mocking [supervisor/write-log-metadata-to-yaml-file!]
+ (supervisor/write-log-metadata! storm-conf exp-owner exp-worker-id
+ exp-storm-id exp-port conf)
+ (verify-called-once-with-args supervisor/write-log-metadata-to-yaml-file!
+ exp-storm-id exp-port exp-data conf)))))
+
+(deftest test-worker-launcher-requires-user
+ (testing "worker-launcher throws on blank user"
+ (mocking [launch-process]
+ (is (thrown-cause-with-msg? java.lang.IllegalArgumentException
+ #"(?i).*user cannot be blank.*"
+ (supervisor/worker-launcher {} nil ""))))))
+
+(defn found? [sub-str input-str]
+ (if (string? input-str)
+ (contrib-str/substring? sub-str (str input-str))
+ (some? #(contrib-str/substring? sub-str %) input-str)))
+
+(defn not-found? [sub-str input-str]
+ (complement (found? sub-str input-str)))
+
- (deftest test-substitute-childopts-happy-path
+ (deftest test-substitute-childopts-happy-path-string
(testing "worker-launcher replaces ids in childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log"
- ]
- (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
- (is (not-found? "%WORKER-ID%" childopts-with-ids))
- (is (found? "w-01" childopts-with-ids))
- (is (not-found? "%STORM-ID%" childopts-with-ids))
- (is (found? "s-01" childopts-with-ids))
- (is (not-found? "%WORKER-PORT%" childopts-with-ids))
- (is (found? "-9999." childopts-with-ids))
- (is (not-found? "%ID%" childopts-with-ids))
- (is (found? "worker-9999" childopts-with-ids) (str childopts-with-ids))
- )))
+ (let [worker-id "w-01"
+ topology-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%TOPOLOGY-ID%-%WORKER-ID%-%WORKER-PORT%.log -Xms256m"
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
+ (is (= expected-childopts childopts-with-ids)))))
- (deftest test-substitute-childopts-storm-id-alone
+ (deftest test-substitute-childopts-happy-path-list
(testing "worker-launcher replaces ids in childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%STORM-ID%.log"]
- (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
- (is (not-found? "%WORKER-ID%" childopts-with-ids))
- (is (not-found? "w-01" childopts-with-ids))
- (is (not-found? "%STORM-ID%" childopts-with-ids))
- (is (found? "s-01" childopts-with-ids))
- (is (not-found? "%WORKER-PORT%" childopts-with-ids))
- (is (not-found? "-9999." childopts-with-ids))
- (is (not-found? "%ID%" childopts-with-ids))
- (is (not-found? "worker-9999" childopts-with-ids) (str childopts-with-ids)) )))
+ (let [worker-id "w-01"
+ topology-id "s-01"
+ port 9999
+ childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%TOPOLOGY-ID%-%WORKER-ID%-%WORKER-PORT%.log" "-Xms256m")
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
+ (is (= expected-childopts childopts-with-ids)))))
+
+ (deftest test-substitute-childopts-topology-id-alone
+ (testing "worker-launcher replaces ids in childopts"
+ (let [worker-id "w-01"
+ topology-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%TOPOLOGY-ID%.log"
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-s-01.log")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
+ (is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-no-keys
(testing "worker-launcher has no ids to replace in childopts"
[33/50] git commit: Merge branch 'compression' of
https://github.com/revans2/incubator-storm into STORM-376
Posted by bo...@apache.org.
Merge branch 'compression' of https://github.com/revans2/incubator-storm into STORM-376
STORM-376: Add compression to serialization
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/5887eb32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/5887eb32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/5887eb32
Branch: refs/heads/security
Commit: 5887eb329b1f50666d6736136f875da6a6e62f6e
Parents: 2c46e4c a0ca23c
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:57:07 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:57:07 2014 -0500
----------------------------------------------------------------------
conf/defaults.yaml | 1 +
storm-core/src/jvm/backtype/storm/Config.java | 6 ++
.../DefaultSerializationDelegate.java | 61 +++++++++++++++
.../GzipBridgeSerializationDelegate.java | 64 +++++++++++++++
.../GzipSerializationDelegate.java | 68 ++++++++++++++++
.../serialization/SerializationDelegate.java | 35 +++++++++
.../src/jvm/backtype/storm/utils/Utils.java | 61 +++++++++------
.../GzipBridgeSerializationDelegateTest.java | 82 ++++++++++++++++++++
8 files changed, 355 insertions(+), 23 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5887eb32/conf/defaults.yaml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5887eb32/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --cc storm-core/src/jvm/backtype/storm/Config.java
index e09e673,0f1535e..134068f
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@@ -94,8 -95,14 +94,14 @@@ public class Config extends HashMap<Str
* We check with this interval that whether the Netty channel is writable and try to write pending messages
*/
public static final String STORM_NETTY_FLUSH_CHECK_INTERVAL_MS = "storm.messaging.netty.flush.check.interval.ms";
- public static final Object STORM_NETTY_FLUSH_CHECK_INTERVAL_MS_SCHEMA = Number.class;
+ public static final Object STORM_NETTY_FLUSH_CHECK_INTERVAL_MS_SCHEMA = ConfigValidation.IntegerValidator;
+ /**
+ * The delegate for serializing metadata, should be used for serialized objects stored in zookeeper and on disk.
+ * This is NOT used for compressing serialized tuples sent between topologies.
+ */
+ public static final String STORM_META_SERIALIZATION_DELEGATE = "storm.meta.serialization.delegate";
+ public static final Object STORM_META_SERIALIZATION_DELEGATE_SCHEMA = String.class;
/**
* A list of hosts of ZooKeeper servers used to manage the cluster.
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5887eb32/storm-core/src/jvm/backtype/storm/utils/Utils.java
----------------------------------------------------------------------
[48/50] git commit: Added STORM-401 to changelog.
Posted by bo...@apache.org.
Added STORM-401 to changelog.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/d9b68ab1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/d9b68ab1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/d9b68ab1
Branch: refs/heads/security
Commit: d9b68ab1c7bd3cd3fadfb24d1240a5c1c01890ec
Parents: bfc7ce1
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 26 16:38:28 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 26 16:38:28 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/d9b68ab1/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 653dc38..49cb4c5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -60,6 +60,7 @@
* STORM-274: Add support for command remoteconfvalue in storm.cmd
* STORM-132: sort supervisor by free slot in desending order
* STORM-472: Improve error message for non-completeable testing spouts
+ * STORM-401: handle InterruptedIOException properly.
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[22/50] git commit: Remove unwanted spaces
Posted by bo...@apache.org.
Remove unwanted spaces
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/32c671ad
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/32c671ad
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/32c671ad
Branch: refs/heads/security
Commit: 32c671ad173b4e33b1be1ef1a0e279808b06e3b8
Parents: b2aa030
Author: Kishor Patil <kp...@yahoo-inc.com>
Authored: Wed Aug 20 20:00:03 2014 +0000
Committer: Kishor Patil <kp...@yahoo-inc.com>
Committed: Wed Aug 20 20:00:03 2014 +0000
----------------------------------------------------------------------
.../test/clj/backtype/storm/supervisor_test.clj | 72 ++++++++++----------
1 file changed, 36 insertions(+), 36 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/32c671ad/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/supervisor_test.clj b/storm-core/test/clj/backtype/storm/supervisor_test.clj
index 5f65ae7..68ac09c 100644
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@ -355,62 +355,62 @@
(deftest test-substitute-childopts-happy-path-string
(testing "worker-launcher replaces ids in childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log -Xms256m"
- expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ (let [worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log -Xms256m"
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-happy-path-list
(testing "worker-launcher replaces ids in childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log" "-Xms256m")
- expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ (let [worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log" "-Xms256m")
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-storm-id-alone
(testing "worker-launcher replaces ids in childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%STORM-ID%.log"
- expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-s-01.log")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ (let [worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%STORM-ID%.log"
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-s-01.log")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-no-keys
(testing "worker-launcher has no ids to replace in childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log"
- expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ (let [worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log"
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-nil-childopts
(testing "worker-launcher has nil childopts"
- (let [ worker-id "w-01"
- storm-id "s-01"
- port 9999
- childopts nil
- expected-childopts nil
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ (let [worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts nil
+ expected-childopts nil
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-nil-ids
(testing "worker-launcher has nil ids"
- (let [ worker-id nil
- storm-id "s-01"
- port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log"
- expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01--9999.log")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ (let [worker-id nil
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log"
+ expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01--9999.log")
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-retry-read-assignments
[02/50] git commit: reportError when receiving an error-level message
in ShellBolt.
Posted by bo...@apache.org.
reportError when receiving an error-level message in ShellBolt.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/76353a80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/76353a80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/76353a80
Branch: refs/heads/security
Commit: 76353a80ff8d0121cdd5ef53e518f07f0be95a0a
Parents: 9968b8b
Author: Daniel Blanchard <db...@ets.org>
Authored: Thu Aug 14 16:23:53 2014 -0400
Committer: Daniel Blanchard <db...@ets.org>
Committed: Thu Aug 14 16:23:53 2014 -0400
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/task/ShellBolt.java | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/76353a80/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/task/ShellBolt.java b/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
index 599c6da..046d4e6 100644
--- a/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
+++ b/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
@@ -21,6 +21,7 @@ import backtype.storm.Config;
import backtype.storm.generated.ShellComponent;
import backtype.storm.metric.api.IMetric;
import backtype.storm.metric.api.rpc.IShellMetric;
+import backtype.storm.topology.ReportedFailedException;
import backtype.storm.tuple.MessageId;
import backtype.storm.tuple.Tuple;
import backtype.storm.utils.ShellProcess;
@@ -257,6 +258,7 @@ public class ShellBolt implements IBolt {
break;
case ERROR:
LOG.error(msg);
+ _collector.reportError(new ReportedFailedException(msg));
break;
default:
LOG.info(msg);
[23/50] git commit: Use TOPOLOGY-ID instead of STORM-ID
Posted by bo...@apache.org.
Use TOPOLOGY-ID instead of STORM-ID
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/1d841941
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/1d841941
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/1d841941
Branch: refs/heads/security
Commit: 1d841941a848bfd17e9a830413415cca21ff76f8
Parents: 32c671a
Author: Kishor Patil <kp...@yahoo-inc.com>
Authored: Wed Aug 20 21:14:52 2014 +0000
Committer: Kishor Patil <kp...@yahoo-inc.com>
Committed: Wed Aug 20 21:14:52 2014 +0000
----------------------------------------------------------------------
.../clj/backtype/storm/daemon/supervisor.clj | 6 ++--
storm-core/src/jvm/backtype/storm/Config.java | 4 +--
.../test/clj/backtype/storm/supervisor_test.clj | 34 ++++++++++----------
3 files changed, 22 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/1d841941/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 362452e..4ba9811 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -474,11 +474,11 @@
(str arch-resource-root File/pathSeparator resource-root File/pathSeparator (conf JAVA-LIBRARY-PATH))))
(defn substitute-childopts
- "Generates runtime childopts by replacing keys with storm-id, worker-id, port"
- [value worker-id storm-id port]
+ "Generates runtime childopts by replacing keys with topology-id, worker-id, port"
+ [value worker-id topology-id port]
(let [replacement-map {"%ID%" (str port)
"%WORKER-ID%" (str worker-id)
- "%STORM-ID%" (str storm-id)
+ "%TOPOLOGY-ID%" (str topology-id)
"%WORKER-PORT%" (str port)}
sub-fn #(reduce (fn [string entry]
(apply clojure.string/replace string entry))
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/1d841941/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/Config.java b/storm-core/src/jvm/backtype/storm/Config.java
index 57c1a49..5d7f762 100644
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@ -467,11 +467,11 @@ public class Config extends HashMap<String, Object> {
public static final Object SUPERVISOR_MONITOR_FREQUENCY_SECS_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * The jvm opts provided to workers launched by this supervisor. All "%ID%", "%WORKER-ID%", "%STORM-ID%"
+ * The jvm opts provided to workers launched by this supervisor. All "%ID%", "%WORKER-ID%", "%TOPOLOGY-ID%"
* and "%WORKER-PORT%" substrings are replaced with:
* %ID% -> port (for backward compatibility),
* %WORKER-ID% -> worker-id,
- * %STORM-ID% -> topology-id,
+ * %TOPOLOGY-ID% -> topology-id,
* %WORKER-PORT% -> port.
*/
public static final String WORKER_CHILDOPTS = "worker.childopts";
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/1d841941/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/supervisor_test.clj b/storm-core/test/clj/backtype/storm/supervisor_test.clj
index 68ac09c..1c19aed 100644
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@ -356,61 +356,61 @@
(deftest test-substitute-childopts-happy-path-string
(testing "worker-launcher replaces ids in childopts"
(let [worker-id "w-01"
- storm-id "s-01"
+ topology-id "s-01"
port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log -Xms256m"
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%TOPOLOGY-ID%-%WORKER-ID%-%WORKER-PORT%.log -Xms256m"
expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-happy-path-list
(testing "worker-launcher replaces ids in childopts"
(let [worker-id "w-01"
- storm-id "s-01"
+ topology-id "s-01"
port 9999
- childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log" "-Xms256m")
+ childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%TOPOLOGY-ID%-%WORKER-ID%-%WORKER-PORT%.log" "-Xms256m")
expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01-w-01-9999.log" "-Xms256m")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
(is (= expected-childopts childopts-with-ids)))))
-(deftest test-substitute-childopts-storm-id-alone
+(deftest test-substitute-childopts-topology-id-alone
(testing "worker-launcher replaces ids in childopts"
(let [worker-id "w-01"
- storm-id "s-01"
+ topology-id "s-01"
port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%STORM-ID%.log"
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%TOPOLOGY-ID%.log"
expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-s-01.log")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-no-keys
(testing "worker-launcher has no ids to replace in childopts"
(let [worker-id "w-01"
- storm-id "s-01"
+ topology-id "s-01"
port 9999
childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log"
expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-nil-childopts
(testing "worker-launcher has nil childopts"
(let [worker-id "w-01"
- storm-id "s-01"
+ topology-id "s-01"
port 9999
childopts nil
expected-childopts nil
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-substitute-childopts-nil-ids
(testing "worker-launcher has nil ids"
(let [worker-id nil
- storm-id "s-01"
+ topology-id "s-01"
port 9999
- childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log"
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%TOPOLOGY-ID%-%WORKER-ID%-%WORKER-PORT%.log"
expected-childopts '("-Xloggc:/home/y/lib/storm/current/logs/gc.worker-9999-s-01--9999.log")
- childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port)]
+ childopts-with-ids (supervisor/substitute-childopts childopts worker-id topology-id port)]
(is (= expected-childopts childopts-with-ids)))))
(deftest test-retry-read-assignments
[41/50] git commit: Merge branch 'STORM-274' of
https://github.com/sureshms/incubator-storm into STORM-274
Posted by bo...@apache.org.
Merge branch 'STORM-274' of https://github.com/sureshms/incubator-storm into STORM-274
STORM-274: Add support for command remoteconfvalue in storm.cmd
Conflicts:
bin/storm.cmd
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/4042b9de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/4042b9de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/4042b9de
Branch: refs/heads/security
Commit: 4042b9dea8c8bb647c5860ecfc257fb3ad22acf3
Parents: 2b4cbe9 cee992a
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 12:50:32 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 12:50:32 2014 -0500
----------------------------------------------------------------------
bin/storm.cmd | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/4042b9de/bin/storm.cmd
----------------------------------------------------------------------
diff --cc bin/storm.cmd
index 17c01d6,8f16107..359acb6
--- a/bin/storm.cmd
+++ b/bin/storm.cmd
@@@ -55,7 -55,7 +55,7 @@@
goto :eof
)
- set corecommands=activate deactivate dev-zookeeper drpc kill list nimbus logviewer rebalance repl shell supervisor ui
- set corecommands=activate deactivate dev-zookeeper drpc kill list nimbus rebalance remoteconfvalue repl shell supervisor ui
++ set corecommands=activate deactivate dev-zookeeper drpc kill list nimbus logviewer rebalance remoteconfvalue repl shell supervisor ui
for %%i in ( %corecommands% ) do (
if %storm-command% == %%i set corecommand=true
)
[15/50] git commit: Merge branch 'shellbolt_reportfailedexception' of
https://github.com/dan-blanchard/incubator-storm into STORM-455
Posted by bo...@apache.org.
Merge branch 'shellbolt_reportfailedexception' of https://github.com/dan-blanchard/incubator-storm into STORM-455
STORM-455: Report error-level messages from ShellBolt children
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/96574686
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/96574686
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/96574686
Branch: refs/heads/security
Commit: 96574686e5ec3b83f906fe85793d6937a4729ee2
Parents: b607ca9 76353a8
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 11:24:04 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 11:24:04 2014 -0500
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/task/ShellBolt.java | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
[17/50] git commit: Merge branch 'Multilang_JsonSerializer' of
https://github.com/dashengju/incubator-storm into STORM-443
Posted by bo...@apache.org.
Merge branch 'Multilang_JsonSerializer' of https://github.com/dashengju/incubator-storm into STORM-443
STORM-443: multilang log's loglevel protocol can cause hang
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/0d82c2fc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/0d82c2fc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/0d82c2fc
Branch: refs/heads/security
Commit: 0d82c2fc127343f18c3b41ae7951989a6515f371
Parents: aa33ecd 837b9e5
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Aug 19 12:32:55 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Aug 19 12:32:55 2014 -0500
----------------------------------------------------------------------
.../src/jvm/backtype/storm/multilang/JsonSerializer.java | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
[03/50] git commit: remove unnessery test and change port numbers
order
Posted by bo...@apache.org.
remove unnessery test and change port numbers order
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/a2484f3e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/a2484f3e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/a2484f3e
Branch: refs/heads/security
Commit: a2484f3e7659841d986439b73c79116f8ba0d70b
Parents: d6a7114
Author: Kang Xiao <kx...@gmail.com>
Authored: Sat Aug 16 00:12:53 2014 +0800
Committer: Kang Xiao <kx...@gmail.com>
Committed: Sat Aug 16 00:12:53 2014 +0800
----------------------------------------------------------------------
storm-core/test/clj/backtype/storm/scheduler_test.clj | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a2484f3e/storm-core/test/clj/backtype/storm/scheduler_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/scheduler_test.clj b/storm-core/test/clj/backtype/storm/scheduler_test.clj
index 9744ac6..856ddeb 100644
--- a/storm-core/test/clj/backtype/storm/scheduler_test.clj
+++ b/storm-core/test/clj/backtype/storm/scheduler_test.clj
@@ -262,10 +262,6 @@
))
(deftest test-sort-slots
- ;; test supervisor1 and supervisor2 has the same free slot
- (is (= '(["supervisor1" 6700] ["supervisor2" 6700]
- ["supervisor1" 6701] ["supervisor2" 6701])
- (sort-slots [["supervisor1" 6700] ["supervisor1" 6701] ["supervisor2" 6700] ["supervisor2" 6701]])))
;; test supervisor2 has more free slots
(is (= '(["supervisor2" 6700] ["supervisor1" 6700]
["supervisor2" 6701] ["supervisor1" 6701]
@@ -275,12 +271,12 @@
])))
;; test supervisor3 has more free slots
(is (= '(["supervisor3" 6700] ["supervisor2" 6700] ["supervisor1" 6700]
- ["supervisor3" 6701] ["supervisor2" 6701] ["supervisor1" 6701]
+ ["supervisor3" 6703] ["supervisor2" 6701] ["supervisor1" 6701]
["supervisor3" 6702] ["supervisor2" 6702]
- ["supervisor3" 6703])
+ ["supervisor3" 6701])
(sort-slots [["supervisor1" 6700] ["supervisor1" 6701]
["supervisor2" 6700] ["supervisor2" 6701] ["supervisor2" 6702]
- ["supervisor3" 6700] ["supervisor3" 6701] ["supervisor3" 6702] ["supervisor3" 6703]
+ ["supervisor3" 6700] ["supervisor3" 6703] ["supervisor3" 6702] ["supervisor3" 6701]
])))
)
[26/50] git commit: Improve error message for non-completeable
testing spouts
Posted by bo...@apache.org.
Improve error message for non-completeable testing spouts
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/2058f400
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/2058f400
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/2058f400
Branch: refs/heads/security
Commit: 2058f40095f5134834821d768a74ffb0905639dd
Parents: d4ebb6d
Author: Dave Golombek <dg...@mylookout.com>
Authored: Fri Aug 22 15:56:21 2014 -0400
Committer: Dave Golombek <dg...@mylookout.com>
Committed: Fri Aug 22 15:56:21 2014 -0400
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/testing.clj | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2058f400/storm-core/src/clj/backtype/storm/testing.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/testing.clj b/storm-core/src/clj/backtype/storm/testing.clj
index 54f40e0..73136f3 100644
--- a/storm-core/src/clj/backtype/storm/testing.clj
+++ b/storm-core/src/clj/backtype/storm/testing.clj
@@ -475,7 +475,7 @@
(.set_spout_object spout-spec (serialize-component-object spout))))
(doseq [spout (spout-objects spouts)]
(when-not (extends? CompletableSpout (.getClass spout))
- (throw (RuntimeException. "Cannot complete topology unless every spout is a CompletableSpout (or mocked to be)"))))
+ (throw (RuntimeException. (str "Cannot complete topology unless every spout is a CompletableSpout (or mocked to be); failed by " spout)))))
(doseq [spout (spout-objects spouts)]
(startup spout))
[24/50] git commit: STORM-361. Add JSON-P support to Storm UI API.
Posted by bo...@apache.org.
STORM-361. Add JSON-P support to Storm UI API.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/2db902e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/2db902e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/2db902e3
Branch: refs/heads/security
Commit: 2db902e3883519c3c602c046003b8354d6edfbc3
Parents: d4ebb6d
Author: Sriharsha Chintalapani <ma...@harsha.io>
Authored: Fri Jul 25 09:03:36 2014 -0700
Committer: Sriharsha Chintalapani <ma...@harsha.io>
Committed: Wed Aug 20 15:18:10 2014 -0700
----------------------------------------------------------------------
STORM-UI-REST-API.md | 1 +
storm-core/src/clj/backtype/storm/ui/core.clj | 42 +++++++++++++---------
2 files changed, 26 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2db902e3/STORM-UI-REST-API.md
----------------------------------------------------------------------
diff --git a/STORM-UI-REST-API.md b/STORM-UI-REST-API.md
index 6dda435..df446f3 100644
--- a/STORM-UI-REST-API.md
+++ b/STORM-UI-REST-API.md
@@ -1,6 +1,7 @@
# Storm UI REST API
Storm UI server provides a REST Api to access cluster, topology, component overview and metrics.
This api returns json response.
+REST API supports JSONP. User can pass callback query param to wrap json in the callback function.
Please ignore undocumented elements in the json repsonse.
## Using the UI REST Api
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2db902e3/storm-core/src/clj/backtype/storm/ui/core.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/ui/core.clj b/storm-core/src/clj/backtype/storm/ui/core.clj
index 0930807..9937607 100644
--- a/storm-core/src/clj/backtype/storm/ui/core.clj
+++ b/storm-core/src/clj/backtype/storm/ui/core.clj
@@ -294,7 +294,7 @@
(if error
(time-delta (.get_error_time_secs ^ErrorInfo error))))
-(defn get-error-data
+(defn get-error-data
[error]
(if error
(error-subset (.get_error ^ErrorInfo error))
@@ -816,29 +816,37 @@
[sys?]
(if (or (nil? sys?) (= "false" sys?)) false true))
-(defn json-response [data & [status]]
- {:status (or status 200)
- :headers {"Content-Type" "application/json"}
- :body (to-json data)})
+(defn wrap-json-in-callback [callback response]
+ (str callback "(" response ");"))
+
+(defnk json-response
+ [data callback :serialize-fn to-json :status 200]
+ {:status status
+ :headers (if (not-nil? callback) {"Content-Type" "application/javascript"}
+ {"Content-Type" "application/json"})
+ :body (if (not-nil? callback)
+ (wrap-json-in-callback callback (serialize-fn data))
+ (serialize-fn data))})
(defroutes main-routes
- (GET "/api/v1/cluster/configuration" []
- (cluster-configuration))
- (GET "/api/v1/cluster/summary" []
- (json-response (cluster-summary)))
- (GET "/api/v1/supervisor/summary" []
- (json-response (supervisor-summary)))
- (GET "/api/v1/topology/summary" []
- (json-response (all-topologies-summary)))
+ (GET "/api/v1/cluster/configuration" [& m]
+ (json-response (cluster-configuration)
+ (:callback m) :serialize-fn identity))
+ (GET "/api/v1/cluster/summary" [& m]
+ (json-response (cluster-summary) (:callback m)))
+ (GET "/api/v1/supervisor/summary" [& m]
+ (json-response (supervisor-summary) (:callback m)))
+ (GET "/api/v1/topology/summary" [& m]
+ (json-response (all-topologies-summary) (:callback m)))
(GET "/api/v1/topology/:id" [id & m]
(let [id (url-decode id)]
- (json-response (topology-page id (:window m) (check-include-sys? (:sys m))))))
+ (json-response (topology-page id (:window m) (check-include-sys? (:sys m))) (:callback m))))
(GET "/api/v1/topology/:id/visualization" [:as {:keys [cookies servlet-request]} id & m]
- (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys m)))))
+ (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys m))) (:callback m)))
(GET "/api/v1/topology/:id/component/:component" [id component & m]
(let [id (url-decode id)
component (url-decode component)]
- (json-response (component-page id component (:window m) (check-include-sys? (:sys m))))))
+ (json-response (component-page id component (:window m) (check-include-sys? (:sys m))) (:callback m))))
(POST "/api/v1/topology/:id/activate" [id]
(with-nimbus nimbus
(let [id (url-decode id)
@@ -896,7 +904,7 @@
(try
(handler request)
(catch Exception ex
- (json-response (exception->json ex) 500)))))
+ (json-response (exception->json ex) ((:query-params request) "callback") :status 500)))))
(def app
(handler/site (-> main-routes
[34/50] git commit: Added STORM-376 to Changelog
Posted by bo...@apache.org.
Added STORM-376 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/93e712b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/93e712b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/93e712b3
Branch: refs/heads/security
Commit: 93e712b3781916e23f4f17693d96615e603f9c9e
Parents: 5887eb3
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 10:57:49 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 10:57:49 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/93e712b3/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9ed1a08..fc5554a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -53,6 +53,7 @@
* STORM-449: Updated ShellBolt to not exit when shutting down.
* STORM-464: Simulated time advanced after test cluster exits causes intermittent test failures
* STORM-463: added static version of metrics helpers for Config
+ * STORM-376: Add compression to serialization
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[21/50] git commit: STORM-464: Simulated time advanced after test
cluster exits causes intermitent test failures
Posted by bo...@apache.org.
STORM-464: Simulated time advanced after test cluster exits causes intermitent test failures
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/8604e904
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/8604e904
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/8604e904
Branch: refs/heads/security
Commit: 8604e9044ab08dedcb2851685b70abe54c278c2b
Parents: d4ebb6d
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Aug 20 11:04:16 2014 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Wed Aug 20 11:04:16 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/testing.clj | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/8604e904/storm-core/src/clj/backtype/storm/testing.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/testing.clj b/storm-core/src/clj/backtype/storm/testing.clj
index 54f40e0..0e4b23c 100644
--- a/storm-core/src/clj/backtype/storm/testing.clj
+++ b/storm-core/src/clj/backtype/storm/testing.clj
@@ -235,10 +235,11 @@
(log-error t# "Error in cluster")
(throw t#))
(finally
- (let [keep-waiting?# (atom true)]
- (future (while @keep-waiting?# (simulate-wait ~cluster-sym)))
+ (let [keep-waiting?# (atom true)
+ f# (future (while @keep-waiting?# (simulate-wait ~cluster-sym)))]
(kill-local-storm-cluster ~cluster-sym)
- (reset! keep-waiting?# false))))))
+ (reset! keep-waiting?# false)
+ @f#)))))
(defmacro with-simulated-time-local-cluster
[& args]
[46/50] git commit: Added STORM-472 to Changelog
Posted by bo...@apache.org.
Added STORM-472 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/97608671
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/97608671
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/97608671
Branch: refs/heads/security
Commit: 97608671d59c8d8267822c62481e4a40963834c8
Parents: 3fcaf69
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Aug 25 15:51:25 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Aug 25 15:51:25 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/97608671/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2962c47..653dc38 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -59,6 +59,7 @@
* STORM-373: Provide Additional String substitutions for *.worker.childopts
* STORM-274: Add support for command remoteconfvalue in storm.cmd
* STORM-132: sort supervisor by free slot in desending order
+ * STORM-472: Improve error message for non-completeable testing spouts
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/97608671/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 341eb31..465211d 100644
--- a/README.markdown
+++ b/README.markdown
@@ -168,6 +168,7 @@ under the License.
* Marc Beitchman ([@mbeitchman](https://github.com/mbeitchman))
* Dan Blanchard ([@dan-blanchard](https://github.com/dan-blanchard))
* Mykhailo "Michael" Pershyn ([@pershyn](https://github.com/pershyn))
+* Dave Golombek ([@dgolombek](https://github.com/dgolombek))
## Acknowledgements