You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2014/07/29 22:58:00 UTC
[01/50] git commit: Merge branch 'master' of
https://github.com/HeartSaVioR/incubator-storm into STORM-381
Repository: incubator-storm
Updated Branches:
refs/heads/security ff8336b70 -> 559c883d5
Merge branch 'master' of https://github.com/HeartSaVioR/incubator-storm into STORM-381
STORM-381: Replace broken jquery.tablesorter.min.js to latest
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/c18276f8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/c18276f8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/c18276f8
Branch: refs/heads/security
Commit: c18276f84c09eb575771c2a0d8c6d8e85e785e6a
Parents: e5226db 8f3ed52
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Tue Jul 15 16:41:00 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Tue Jul 15 16:41:00 2014 -0500
----------------------------------------------------------------------
storm-core/src/ui/public/js/jquery.tablesorter.min.js | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
[33/50] git commit: update CHANGELOG and contributors list for
STORM-420
Posted by bo...@apache.org.
update CHANGELOG and contributors list for STORM-420
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/3c55b103
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/3c55b103
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/3c55b103
Branch: refs/heads/security
Commit: 3c55b103d3703fbf825a8d03bd426e4a37538b6a
Parents: 498d8b2
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Thu Jul 24 16:33:54 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Thu Jul 24 16:38:10 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/3c55b103/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a3a564e..f78dd62 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,7 @@
* STORM-403: heartbeats-to-nimbus in supervisor-test failed due to uninten...
* STORM-402: FileNotFoundException when using storm with apache tika
* STORM-364: The exception time display as default timezone.
+ * STORM-420: Missing quotes in storm-starter python code
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/3c55b103/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 2a6afbf..f5d7873 100644
--- a/README.markdown
+++ b/README.markdown
@@ -157,6 +157,7 @@ under the License.
* Ryan Liu ([@shell0dh](https://github.com/shell0dh))
* Dave Parfitt ([@metadave](https://github.com/metadave))
* Jo Liss ([@joliss](https://github.com/joliss))
+* averykhoo ([@averykhoo](https://github.com/averykhoo))
## Acknowledgements
[06/50] git commit: made read-assignment retry on reading
inconsistent assignments.
Posted by bo...@apache.org.
made read-assignment retry on reading inconsistent assignments.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/14bcc9b2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/14bcc9b2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/14bcc9b2
Branch: refs/heads/security
Commit: 14bcc9b2155ab5be71ccffa43689ef6f51b153f4
Parents: 96e81c1
Author: iwasakims <iw...@example.com>
Authored: Mon Jul 14 10:12:24 2014 -0700
Committer: iwasakims <iw...@example.com>
Committed: Wed Jul 16 08:16:50 2014 -0700
----------------------------------------------------------------------
.../clj/backtype/storm/daemon/supervisor.clj | 26 ++++++---
.../test/clj/backtype/storm/supervisor_test.clj | 56 ++++++++++++++++++++
2 files changed, 74 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/14bcc9b2/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 7ba1c69..8d1ac46 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -67,12 +67,19 @@
[(Integer. port) (LocalAssignment. storm-id (doall executors))]
))))
-
(defn- read-assignments
"Returns map from port to struct containing :storm-id and :executors"
- [assignments-snapshot assignment-id]
- (->> (dofor [sid (keys assignments-snapshot)] (read-my-executors assignments-snapshot sid assignment-id))
- (apply merge-with (fn [& ignored] (throw-runtime "Should not have multiple topologies assigned to one port")))))
+ ([assignments-snapshot assignment-id]
+ (->> (dofor [sid (keys assignments-snapshot)] (read-my-executors assignments-snapshot sid assignment-id))
+ (apply merge-with (fn [& ignored] (throw-runtime "Should not have multiple topologies assigned to one port")))))
+ ([assignments-snapshot assignment-id existing-assignment retries]
+ (try (let [assignments (read-assignments assignments-snapshot assignment-id)]
+ (reset! retries 0)
+ assignments)
+ (catch RuntimeException e
+ (if (> @retries 2) (throw e) (swap! retries inc))
+ (log-warn (.getMessage e))
+ existing-assignment))))
(defn- read-storm-code-locations
[assignments-snapshot]
@@ -212,6 +219,7 @@
(exit-process! 20 "Error when processing an event")
))
:assignment-versions (atom {})
+ :sync-retry (atom 0)
})
(defn sync-processes [supervisor]
@@ -314,13 +322,15 @@
assignment-versions)
storm-code-map (read-storm-code-locations assignments-snapshot)
downloaded-storm-ids (set (read-downloaded-storm-ids conf))
- all-assignment (read-assignments
- assignments-snapshot
- (:assignment-id supervisor))
+ existing-assignment (.get local-state LS-LOCAL-ASSIGNMENTS)
+ all-assignment (read-assignments assignments-snapshot
+ (:assignment-id supervisor)
+ existing-assignment
+ (:sync-retry supervisor))
new-assignment (->> all-assignment
(filter-key #(.confirmAssigned isupervisor %)))
assigned-storm-ids (assigned-storm-ids-from-port-assignments new-assignment)
- existing-assignment (.get local-state LS-LOCAL-ASSIGNMENTS)]
+ ]
(log-debug "Synchronizing supervisor")
(log-debug "Storm code map: " storm-code-map)
(log-debug "Downloaded storm ids: " downloaded-storm-ids)
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/14bcc9b2/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/supervisor_test.clj b/storm-core/test/clj/backtype/storm/supervisor_test.clj
index eb69f11..15d0982 100644
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@ -352,3 +352,59 @@
;; TODO just do reassign, and check that cleans up worker states after killing but doesn't get rid of downloaded code
)
+(deftest test-retry-read-assignments
+ (with-simulated-time-local-cluster [cluster
+ :supervisors 0
+ :ports-per-supervisor 2
+ :daemon-conf {NIMBUS-REASSIGN false
+ NIMBUS-MONITOR-FREQ-SECS 10
+ TOPOLOGY-MESSAGE-TIMEOUT-SECS 30
+ TOPOLOGY-ACKER-EXECUTORS 0}]
+ (letlocals
+ (bind sup1 (add-supervisor cluster :id "sup1" :ports [1 2 3 4]))
+ (bind topology1 (thrift/mk-topology
+ {"1" (thrift/mk-spout-spec (TestPlannerSpout. true) :parallelism-hint 2)}
+ {}))
+ (bind topology2 (thrift/mk-topology
+ {"1" (thrift/mk-spout-spec (TestPlannerSpout. true) :parallelism-hint 2)}
+ {}))
+ (bind state (:storm-cluster-state cluster))
+ (bind changed (capture-changed-workers
+ (submit-mocked-assignment
+ (:nimbus cluster)
+ "topology1"
+ {TOPOLOGY-WORKERS 2}
+ topology1
+ {1 "1"
+ 2 "1"}
+ {[1] ["sup1" 1]
+ [2] ["sup1" 2]
+ })
+ (submit-mocked-assignment
+ (:nimbus cluster)
+ "topology2"
+ {TOPOLOGY-WORKERS 2}
+ topology2
+ {1 "1"
+ 2 "1"}
+ {[1] ["sup1" 1]
+ [2] ["sup1" 2]
+ })
+ (advance-cluster-time cluster 10)
+ ))
+ (is (empty? (:launched changed)))
+ (bind options (RebalanceOptions.))
+ (.set_wait_secs options 0)
+ (bind changed (capture-changed-workers
+ (.rebalance (:nimbus cluster) "topology2" options)
+ (advance-cluster-time cluster 10)
+ (heartbeat-workers cluster "sup1" [1 2 3 4])
+ (advance-cluster-time cluster 10)
+ ))
+ (validate-launched-once (:launched changed)
+ {"sup1" [1 2]}
+ (get-storm-id (:storm-cluster-state cluster) "topology1"))
+ (validate-launched-once (:launched changed)
+ {"sup1" [3 4]}
+ (get-storm-id (:storm-cluster-state cluster) "topology2"))
+ )))
[30/50] git commit: Updated README for pull 179
Posted by bo...@apache.org.
Updated README for pull 179
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/9b3cb183
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/9b3cb183
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/9b3cb183
Branch: refs/heads/security
Commit: 9b3cb1830489cc4f629d5fa38cf7ab2181475c4f
Parents: 4ac8dc3
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Thu Jul 24 16:11:24 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Thu Jul 24 16:11:24 2014 -0500
----------------------------------------------------------------------
README.markdown | 1 +
storm-core/test/clj/backtype/storm/cluster_test.clj | 3 ++-
2 files changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/9b3cb183/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 341b62c..5e89333 100644
--- a/README.markdown
+++ b/README.markdown
@@ -155,6 +155,7 @@ under the License.
* Masatake Iwasaki ([@iwasakims](https://github.com/iwasakims))
* Milad Fatenejad ([@icksa](https://github.com/icksa))
* Ryan Liu ([@shell0dh](https://github.com/shell0dh))
+* Jo Liss ([@joliss](https://github.com/joliss))
## Acknowledgements
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/9b3cb183/storm-core/test/clj/backtype/storm/cluster_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/cluster_test.clj b/storm-core/test/clj/backtype/storm/cluster_test.clj
index a6bb10e..63efd30 100644
--- a/storm-core/test/clj/backtype/storm/cluster_test.clj
+++ b/storm-core/test/clj/backtype/storm/cluster_test.clj
@@ -208,8 +208,9 @@
(let [state (mk-storm-state zk-port)]
(.report-error state "a" "1"(local-hostname) 6700 (RuntimeException.))
(validate-errors! state "a" "1" ["RuntimeException"])
+ (advance-time-secs! 1)
(.report-error state "a" "1" (local-hostname) 6700 (IllegalArgumentException.))
- (validate-errors! state "a" "1" ["RuntimeException" "IllegalArgumentException"])
+ (validate-errors! state "a" "1" ["IllegalArgumentException" "RuntimeException"])
(doseq [i (range 10)]
(.report-error state "a" "2" (local-hostname) 6700 (RuntimeException.))
(advance-time-secs! 2))
[02/50] git commit: Updating LICENSE and CHANGELOG for STORM-381
Posted by bo...@apache.org.
Updating LICENSE and CHANGELOG for STORM-381
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/96e81c1f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/96e81c1f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/96e81c1f
Branch: refs/heads/security
Commit: 96e81c1fb567006023a37a3f50cc6b8e051f52c7
Parents: c18276f
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Tue Jul 15 16:45:18 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Tue Jul 15 16:45:18 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
LICENSE | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/96e81c1f/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6a77bdf..f198eea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,7 @@
* STORM-351: multilang python process fall into endless loop
* STORM-375: Smarter downloading of assignments by supervisors and workers
* STORM-328: More restrictive Config checks, strict range check within Utils.getInt()
+ * STORM-381: Replace broken jquery.tablesorter.min.js to latest
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/96e81c1f/LICENSE
----------------------------------------------------------------------
diff --git a/LICENSE b/LICENSE
index 6df4130..4efd7ee 100644
--- a/LICENSE
+++ b/LICENSE
@@ -254,7 +254,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-----------------------------------------------------------------------
-For jQuery TableSorter 2.0.5b (storm-core/src/ui/public/js/jquery.tablesorter.min.js)
+For jQuery TableSorter 2.17.3 (storm-core/src/ui/public/js/jquery.tablesorter.min.js)
Copyright (c) 2007 Christian Bach
Examples and docs at: http://tablesorter.com
[11/50] git commit: support loglevel to multilang's log interface and
python & ruby implementation
Posted by bo...@apache.org.
support loglevel to multilang's log interface and python & ruby implementation
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/ea30b703
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/ea30b703
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/ea30b703
Branch: refs/heads/security
Commit: ea30b703865bf043bea2fc4f7c689f5a13d59c0f
Parents: 69b4601
Author: dashengju <da...@qq.com>
Authored: Sun Jul 20 17:34:41 2014 +0800
Committer: dashengju <da...@qq.com>
Committed: Sun Jul 20 17:34:41 2014 +0800
----------------------------------------------------------------------
.../storm/multilang/JsonSerializer.java | 7 ++++-
.../jvm/backtype/storm/multilang/ShellMsg.java | 26 ++++++++++++++++
.../jvm/backtype/storm/spout/ShellSpout.java | 30 ++++++++++++++++--
.../src/jvm/backtype/storm/task/ShellBolt.java | 32 ++++++++++++++++++--
.../jvm/backtype/storm/utils/ShellProcess.java | 2 +-
storm-core/src/multilang/py/storm.py | 19 ++++++++++--
storm-core/src/multilang/rb/storm.rb | 24 +++++++++++++--
7 files changed, 129 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/jvm/backtype/storm/multilang/JsonSerializer.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/multilang/JsonSerializer.java b/storm-core/src/jvm/backtype/storm/multilang/JsonSerializer.java
index 4d3c3f8..fc97073 100644
--- a/storm-core/src/jvm/backtype/storm/multilang/JsonSerializer.java
+++ b/storm-core/src/jvm/backtype/storm/multilang/JsonSerializer.java
@@ -144,7 +144,12 @@ public class JsonSerializer implements ISerializer {
shellMsg.setMetricName(metricName);
Object paramsObj = msg.get("params");
- shellMsg.setMetricParams(paramsObj);
+ shellMsg.setMetricParams(paramsObj);
+
+ if (command.equals("log")) {
+ long logLevel = (Long)msg.get("level");
+ shellMsg.setLogLevel((int)logLevel);
+ }
return shellMsg;
}
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/jvm/backtype/storm/multilang/ShellMsg.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/multilang/ShellMsg.java b/storm-core/src/jvm/backtype/storm/multilang/ShellMsg.java
index ed803c4..e035df0 100644
--- a/storm-core/src/jvm/backtype/storm/multilang/ShellMsg.java
+++ b/storm-core/src/jvm/backtype/storm/multilang/ShellMsg.java
@@ -46,6 +46,24 @@ public class ShellMsg {
private String metricName;
private Object metricParams;
+ //logLevel
+ public enum ShellLogLevel {
+ TRACE, DEBUG, INFO, WARN, ERROR;
+
+ public static ShellLogLevel fromInt(int i) {
+ switch (i) {
+ case 0: return TRACE;
+ case 1: return DEBUG;
+ case 2: return INFO;
+ case 3: return WARN;
+ case 4: return ERROR;
+ default: return INFO;
+ }
+ }
+ }
+
+ private ShellLogLevel logLevel = ShellLogLevel.INFO;
+
public String getCommand() {
return command;
}
@@ -139,4 +157,12 @@ public class ShellMsg {
public Object getMetricParams() {
return metricParams;
}
+
+ public ShellLogLevel getLogLevel() {
+ return logLevel;
+ }
+
+ public void setLogLevel(int logLevel) {
+ this.logLevel = ShellLogLevel.fromInt(logLevel);
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java b/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java
index b271bfd..70bac5d 100644
--- a/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java
+++ b/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java
@@ -131,8 +131,7 @@ public class ShellSpout implements ISpout {
if (command.equals("sync")) {
return;
} else if (command.equals("log")) {
- String msg = shellMsg.getMsg();
- LOG.info("Shell msg: " + msg + _process.getProcessInfoString());
+ handleLog(shellMsg);
} else if (command.equals("emit")) {
String stream = shellMsg.getStream();
Long task = shellMsg.getTask();
@@ -158,6 +157,33 @@ public class ShellSpout implements ISpout {
}
}
+ private void handleLog(ShellMsg shellMsg) {
+ String msg = shellMsg.getMsg();
+ msg = "ShellLog " + _process.getProcessInfoString() + " " + msg;
+ ShellMsg.ShellLogLevel logLevel = shellMsg.getLogLevel();
+
+ switch (logLevel) {
+ case TRACE:
+ LOG.trace(msg);
+ break;
+ case DEBUG:
+ LOG.debug(msg);
+ break;
+ case INFO:
+ LOG.info(msg);
+ break;
+ case WARN:
+ LOG.warn(msg);
+ break;
+ case ERROR:
+ LOG.error(msg);
+ break;
+ default:
+ LOG.info(msg);
+ break;
+ }
+ }
+
@Override
public void activate() {
}
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/task/ShellBolt.java b/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
index 430581d..4f8bca5 100644
--- a/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
+++ b/storm-core/src/jvm/backtype/storm/task/ShellBolt.java
@@ -122,8 +122,7 @@ public class ShellBolt implements IBolt {
} else if (command.equals("error")) {
handleError(shellMsg.getMsg());
} else if (command.equals("log")) {
- String msg = shellMsg.getMsg();
- LOG.info("Shell msg: " + msg + _process.getProcessInfoString());
+ handleLog(shellMsg);
} else if (command.equals("emit")) {
handleEmit(shellMsg);
} else if (command.equals("metrics")) {
@@ -234,7 +233,34 @@ public class ShellBolt implements IBolt {
shellMsg.getStream(), anchors, shellMsg.getTuple());
}
}
-
+
+ private void handleLog(ShellMsg shellMsg) {
+ String msg = shellMsg.getMsg();
+ msg = "ShellLog " + _process.getProcessInfoString() + " " + msg;
+ ShellMsg.ShellLogLevel logLevel = shellMsg.getLogLevel();
+
+ switch (logLevel) {
+ case TRACE:
+ LOG.trace(msg);
+ break;
+ case DEBUG:
+ LOG.debug(msg);
+ break;
+ case INFO:
+ LOG.info(msg);
+ break;
+ case WARN:
+ LOG.warn(msg);
+ break;
+ case ERROR:
+ LOG.error(msg);
+ break;
+ default:
+ LOG.info(msg);
+ break;
+ }
+ }
+
private void handleMetrics(ShellMsg shellMsg) {
//get metric name
String name = shellMsg.getMetricName();
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/jvm/backtype/storm/utils/ShellProcess.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/utils/ShellProcess.java b/storm-core/src/jvm/backtype/storm/utils/ShellProcess.java
index 79e7d50..6bcdf26 100644
--- a/storm-core/src/jvm/backtype/storm/utils/ShellProcess.java
+++ b/storm-core/src/jvm/backtype/storm/utils/ShellProcess.java
@@ -173,7 +173,7 @@ public class ShellProcess implements Serializable {
}
public String getProcessInfoString() {
- return String.format(" pid:%s, name:%s ", pid, componentName);
+ return String.format("pid:%s, name:%s", pid, componentName);
}
public String getProcessTerminationInfoString() {
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/multilang/py/storm.py
----------------------------------------------------------------------
diff --git a/storm-core/src/multilang/py/storm.py b/storm-core/src/multilang/py/storm.py
index adeeb98..d2a3082 100755
--- a/storm-core/src/multilang/py/storm.py
+++ b/storm-core/src/multilang/py/storm.py
@@ -135,8 +135,23 @@ def fail(tup):
def reportError(msg):
sendMsgToParent({"command": "error", "msg": msg})
-def log(msg):
- sendMsgToParent({"command": "log", "msg": msg})
+def log(msg, level=2):
+ sendMsgToParent({"command": "log", "msg": msg, "level":level})
+
+def logTrace(msg):
+ log(msg, 0)
+
+def logDebug(msg):
+ log(msg, 1)
+
+def logInfo(msg):
+ log(msg, 2)
+
+def logWarn(msg):
+ log(msg, 3)
+
+def logError(msg):
+ log(msg, 4)
def rpcMetrics(name, params):
sendMsgToParent({"command": "metrics", "name": name, "params": params})
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/ea30b703/storm-core/src/multilang/rb/storm.rb
----------------------------------------------------------------------
diff --git a/storm-core/src/multilang/rb/storm.rb b/storm-core/src/multilang/rb/storm.rb
index 57d45f1..17232d1 100644
--- a/storm-core/src/multilang/rb/storm.rb
+++ b/storm-core/src/multilang/rb/storm.rb
@@ -124,8 +124,28 @@ module Storm
send_msg_to_parent :command => :error, :msg => msg.to_s
end
- def log(msg)
- send_msg_to_parent :command => :log, :msg => msg.to_s
+ def log(msg, level=2)
+ send_msg_to_parent :command => :log, :msg => msg.to_s, :level => level
+ end
+
+ def logTrace(msg)
+ log(msg, 0)
+ end
+
+ def logDebug(msg)
+ log(msg, 1)
+ end
+
+ def logInfo(msg)
+ log(msg, 2)
+ end
+
+ def logWarn(msg)
+ log(msg, 3)
+ end
+
+ def logError(msg)
+ log(msg, 4)
end
def handshake
[10/50] git commit: Added STROM-354 to changelog and readme.
Posted by bo...@apache.org.
Added STROM-354 to changelog and readme.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/85c9cd5a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/85c9cd5a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/85c9cd5a
Branch: refs/heads/security
Commit: 85c9cd5aee09ac2bf510fbc7fe877ee16c76ee28
Parents: 44aaaa9
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 18 08:28:52 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 18 08:28:52 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/85c9cd5a/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2817e5d..777ed9a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,6 +19,7 @@
* STORM-328: More restrictive Config checks, strict range check within Utils.getInt()
* STORM-381: Replace broken jquery.tablesorter.min.js to latest
* STORM-312: add storm monitor tools to monitor throughtput interactively
+ * STORM-354: Testing: allow users to pass TEST-TIMEOUT-MS as param for complete-topology
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/85c9cd5a/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 03b4ccd..063de70 100644
--- a/README.markdown
+++ b/README.markdown
@@ -151,6 +151,7 @@ under the License.
* DashengJu ([@dashengju](https://github.com/dashengju))
* Jungtaek Lim ([@HeartSaVioR](https://github.com/HeartSaVioR))
* Li Jiahong ([@Gvain](https://github.com/Gvain))
+* Aaron Levin ([@aaronlevin](https://github.com/aaronlevin))
## Acknowledgements
[18/50] git commit: Merge branch 'master' of
https://github.com/icksa/incubator-storm into STORM-402
Posted by bo...@apache.org.
Merge branch 'master' of https://github.com/icksa/incubator-storm into STORM-402
STORM-402: FileNotFoundException when using storm with apache tika
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/5b425e41
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/5b425e41
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/5b425e41
Branch: refs/heads/security
Commit: 5b425e41bae2f455cae2c365ed617d5ac833847b
Parents: c1aed00 f019eff
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Jul 21 11:41:43 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Jul 21 11:41:43 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
storm-core/src/clj/backtype/storm/daemon/supervisor.clj | 10 +++++++---
3 files changed, 9 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5b425e41/CHANGELOG.md
----------------------------------------------------------------------
diff --cc CHANGELOG.md
index 1e9a7e6,7c6a7a4..661283f
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -15,13 -15,6 +15,14 @@@
* STORM-367: Storm UI REST api documentation.
* STORM-200: Proposal for Multilang's Metrics feature
* STORM-351: multilang python process fall into endless loop
+ * STORM-375: Smarter downloading of assignments by supervisors and workers
+ * STORM-328: More restrictive Config checks, strict range check within Utils.getInt()
+ * STORM-381: Replace broken jquery.tablesorter.min.js to latest
+ * STORM-312: add storm monitor tools to monitor throughtput interactively
+ * STORM-354: Testing: allow users to pass TEST-TIMEOUT-MS as param for complete-topology
+ * STORM-254: one Spout/Bolt can register metric twice with same name in different timeBucket
+ * STORM-403: heartbeats-to-nimbus in supervisor-test failed due to uninten...
++ * STORM-402: FileNotFoundException when using storm with apache tika
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5b425e41/README.markdown
----------------------------------------------------------------------
diff --cc README.markdown
index c495c19,f026e6d..8a7afc3
--- a/README.markdown
+++ b/README.markdown
@@@ -149,10 -149,6 +149,11 @@@ under the License
* Parth-Brahmbhatt ([@Parth-Brahmbhatt](https://github.com/Parth-Brahmbhatt))
* Adrian Petrescu ([@apetresc](https://github.com/apetresc))
* DashengJu ([@dashengju](https://github.com/dashengju))
+* Jungtaek Lim ([@HeartSaVioR](https://github.com/HeartSaVioR))
+* Li Jiahong ([@Gvain](https://github.com/Gvain))
+* Aaron Levin ([@aaronlevin](https://github.com/aaronlevin))
+* Masatake Iwasaki ([@iwasakims](https://github.com/iwasakims))
++* Milad Fatenejad ([@icksa](https://github.com/icksa))
## Acknowledgements
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5b425e41/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
[49/50] git commit: Added STORM-406 to Changelog
Posted by bo...@apache.org.
Added STORM-406 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/b2a8a77c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/b2a8a77c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/b2a8a77c
Branch: refs/heads/security
Commit: b2a8a77c3b307137527c706d0cd7635a6afe25bf
Parents: 848a748
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Jul 29 07:17:46 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Jul 29 07:17:46 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b2a8a77c/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ccf0c3b..3b38c34 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -32,6 +32,7 @@
* STORM-415: validate-launched-once in supervisor-test can not handle multiple topologies
* STORM-155: Storm rebalancing code causes multiple topologies assigned to a single port
* STORM-419: Updated test so sort ordering is very explicit.
+ * STORM-406: Fix for reconnect logic in netty client.
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[38/50] git commit: Merge branch 'Multilang_protocol_loglevel' of
https://github.com/dashengju/incubator-storm into STORM-414
Posted by bo...@apache.org.
Merge branch 'Multilang_protocol_loglevel' of https://github.com/dashengju/incubator-storm into STORM-414
STORM-414: support logging level to multilang protocol spout and bolt
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/5f599173
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/5f599173
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/5f599173
Branch: refs/heads/security
Commit: 5f599173f29679b39df924177c7e04452a728d94
Parents: 30efc8a ea30b70
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 15:00:13 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 15:00:13 2014 -0500
----------------------------------------------------------------------
.../storm/multilang/JsonSerializer.java | 7 ++++-
.../jvm/backtype/storm/multilang/ShellMsg.java | 26 ++++++++++++++++
.../jvm/backtype/storm/spout/ShellSpout.java | 30 ++++++++++++++++--
.../src/jvm/backtype/storm/task/ShellBolt.java | 32 ++++++++++++++++++--
.../jvm/backtype/storm/utils/ShellProcess.java | 2 +-
storm-core/src/multilang/py/storm.py | 19 ++++++++++--
storm-core/src/multilang/rb/storm.rb | 24 +++++++++++++--
7 files changed, 129 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
[08/50] git commit: Merge branch 'master' into STORM-364
Posted by bo...@apache.org.
Merge branch 'master' into STORM-364
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/fe95ca4b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/fe95ca4b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/fe95ca4b
Branch: refs/heads/security
Commit: fe95ca4bafa75e120c6b7e8dc828d3f8e8c2461c
Parents: 9c186aa 69b4601
Author: shelld0h <ry...@gmail.com>
Authored: Fri Jul 18 18:49:27 2014 +0800
Committer: shelld0h <ry...@gmail.com>
Committed: Fri Jul 18 18:49:27 2014 +0800
----------------------------------------------------------------------
BYLAWS.md | 96 ++++
CHANGELOG.md | 12 +
LICENSE | 2 +-
README.markdown | 4 +
STORM-UI-REST-API.md | 546 +++++++++++++++++++
bin/storm | 20 +-
conf/storm_env.ini | 2 +-
.../storm-kafka/src/jvm/storm/kafka/Broker.java | 9 +-
.../src/jvm/storm/kafka/Partition.java | 9 +-
storm-core/src/clj/backtype/storm/cluster.clj | 29 +
.../src/clj/backtype/storm/command/monitor.clj | 37 ++
.../src/clj/backtype/storm/daemon/logviewer.clj | 181 ++++--
.../clj/backtype/storm/daemon/supervisor.clj | 29 +-
.../src/clj/backtype/storm/daemon/worker.clj | 16 +-
storm-core/src/clj/backtype/storm/testing.clj | 5 +-
.../src/clj/backtype/storm/ui/helpers.clj | 5 +
storm-core/src/clj/backtype/storm/zookeeper.clj | 25 +
.../src/dev/resources/tester_bolt_metrics.py | 35 ++
.../src/dev/resources/tester_spout_metrics.py | 51 ++
storm-core/src/jvm/backtype/storm/Config.java | 121 ++--
.../jvm/backtype/storm/ConfigValidation.java | 70 +++
.../metric/api/rpc/AssignableShellMetric.java | 30 +
.../metric/api/rpc/CombinedShellMetric.java | 31 ++
.../storm/metric/api/rpc/CountShellMetric.java | 38 ++
.../storm/metric/api/rpc/IShellMetric.java | 31 ++
.../metric/api/rpc/ReducedShellMetric.java | 32 ++
.../storm/multilang/JsonSerializer.java | 10 +
.../jvm/backtype/storm/multilang/ShellMsg.java | 20 +
.../jvm/backtype/storm/spout/ShellSpout.java | 44 +-
.../src/jvm/backtype/storm/task/ShellBolt.java | 47 +-
.../backtype/storm/task/TopologyContext.java | 24 +
.../storm/testing/PythonShellMetricsBolt.java | 32 ++
.../storm/testing/PythonShellMetricsSpout.java | 35 ++
.../storm/ui/InvalidRequestException.java | 20 +
.../src/jvm/backtype/storm/utils/Monitor.java | 249 +++++++++
.../jvm/backtype/storm/utils/ShellProcess.java | 46 +-
.../src/jvm/backtype/storm/utils/Utils.java | 23 +-
storm-core/src/multilang/py/storm.py | 11 +-
storm-core/src/ui/public/component.html | 22 +-
storm-core/src/ui/public/css/style.css | 1 +
.../src/ui/public/js/jquery.tablesorter.min.js | 9 +-
storm-core/src/ui/public/topology.html | 4 +-
.../test/clj/backtype/storm/config_test.clj | 41 +-
.../test/clj/backtype/storm/metrics_test.clj | 206 ++++---
44 files changed, 2079 insertions(+), 231 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/fe95ca4b/storm-core/src/clj/backtype/storm/ui/helpers.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/fe95ca4b/storm-core/src/ui/public/component.html
----------------------------------------------------------------------
[12/50] git commit: add memoized version of local-hostname in util ns
Posted by bo...@apache.org.
add memoized version of local-hostname in util ns
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/a5c97d79
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/a5c97d79
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/a5c97d79
Branch: refs/heads/security
Commit: a5c97d79e0e809a43b962a8a6753f376e8bc020f
Parents: 85c9cd5
Author: Homer Strong <ho...@gmail.com>
Authored: Sun Jul 20 12:39:20 2014 -0700
Committer: Homer Strong <ho...@gmail.com>
Committed: Sun Jul 20 12:39:20 2014 -0700
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/util.clj | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a5c97d79/storm-core/src/clj/backtype/storm/util.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/util.clj b/storm-core/src/clj/backtype/storm/util.clj
index 7c1d19e..3df25b7 100644
--- a/storm-core/src/clj/backtype/storm/util.clj
+++ b/storm-core/src/clj/backtype/storm/util.clj
@@ -204,6 +204,8 @@
[]
(.getCanonicalHostName (InetAddress/getLocalHost)))
+(def memoized-local-hostname (memoize local-hostname))
+
(letfn [(try-port [port]
(with-open [socket (java.net.ServerSocket. port)]
(.getLocalPort socket)))]
[41/50] git commit: Updated Changelog for STORM-321
Posted by bo...@apache.org.
Updated Changelog for STORM-321
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/5219f494
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/5219f494
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/5219f494
Branch: refs/heads/security
Commit: 5219f494e0acc333cd40cb5203458fbd3a34ca7b
Parents: 2f73776
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 15:11:34 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 15:11:34 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/5219f494/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 076aad7..43dd049 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -28,6 +28,7 @@
* STORM-399: Kafka Spout defaulting to latest offset when current offset is older then 100k
* STORM-421: Memoize local hostname lookup in executor
* STORM-414: support logging level to multilang protocol spout and bolt
+ * STORM-321: Added a tool to see the current status of STORM JIRA and github pulls.
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[19/50] git commit: Updated the REST API documentation,
add the component-errors property description.
Posted by bo...@apache.org.
Updated the REST API documentation,add the component-errors property description.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/9098f9d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/9098f9d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/9098f9d7
Branch: refs/heads/security
Commit: 9098f9d765bbae96e1577f000abb416092d1fc34
Parents: fe95ca4
Author: shelld0h <ry...@gmail.com>
Authored: Tue Jul 22 13:35:16 2014 +0800
Committer: shelld0h <ry...@gmail.com>
Committed: Tue Jul 22 13:35:16 2014 +0800
----------------------------------------------------------------------
STORM-UI-REST-API.md | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/9098f9d7/STORM-UI-REST-API.md
----------------------------------------------------------------------
diff --git a/STORM-UI-REST-API.md b/STORM-UI-REST-API.md
index 5222ca7..e8a412d 100644
--- a/STORM-UI-REST-API.md
+++ b/STORM-UI-REST-API.md
@@ -342,6 +342,8 @@ Response Fields:
|windowHint| String | window param value in "hh mm ss" format. Default value is "All Time"|
|executors| Integer |Number of executor tasks in the component|
|componentErrors| Array of Strings | List of component errors|
+|componentErrors.time| Long | Timestamp when the exception occurred |
+|componentErrors.error| String | The stack trace of an exception info |
|topologyId| String | Topology's Id|
|tasks| Integer |Number of instances of component|
|window |String. Default value "All Time" | window duration for metrics in seconds|
@@ -375,7 +377,26 @@ Sample Response:
"componentType": "spout",
"windowHint": "10m 0s",
"executors": 5,
- "componentErrors": [],
+ "componentErrors": [
+ {
+ "time":1406006074000,
+ "error":"java.lang.RuntimeException: java.lang.NullPointerException
+ at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
+ at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)
+ at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)
+ at backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67)
+ at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
+ at clojure.lang.AFn.run(AFn.java:24)
+ at java.lang.Thread.run(Thread.java:662)
+Caused by: java.lang.NullPointerException
+ at backtype.storm.serialization.SerializationFactory$IdDictionary.getStreamName(SerializationFactory.java:137)
+ at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:34)
+ at backtype.storm.daemon.executor$mk_task_receiver$fn__3967.invoke(executor.clj:311)
+ at backtype.storm.disruptor$clojure_handler$reify__1585.onEvent(disruptor.clj:43)
+ at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81)
+ ... 6 more"
+ }
+ ],
"topologyId": "WordCount3-1-1402960825",
"tasks": 5,
"window": "600",
[26/50] git commit: Increase max tries and reset local channel
variable
Posted by bo...@apache.org.
Increase max tries and reset local channel variable
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/f1f5ecd9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/f1f5ecd9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/f1f5ecd9
Branch: refs/heads/security
Commit: f1f5ecd92c7ea55d01163c8dbc2360466f34fd3a
Parents: 9b3a632
Author: Kishor Patil <kp...@yahoo-inc.com>
Authored: Thu Jul 24 20:49:22 2014 +0000
Committer: Kishor Patil <kp...@yahoo-inc.com>
Committed: Thu Jul 24 20:49:22 2014 +0000
----------------------------------------------------------------------
conf/defaults.yaml | 3 ++-
storm-core/src/jvm/backtype/storm/messaging/netty/Client.java | 2 ++
2 files changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/f1f5ecd9/conf/defaults.yaml
----------------------------------------------------------------------
diff --git a/conf/defaults.yaml b/conf/defaults.yaml
index 8c07075..2864adc 100644
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@ -108,7 +108,8 @@ zmq.hwm: 0
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
-storm.messaging.netty.max_retries: 120
+# Since nimbus.task.launch.secs and supervisor.worker.start.timeout.secs are 120, other workers should also wait at least that long before giving up on connecting to the other worker.
+storm.messaging.netty.max_retries: 300
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/f1f5ecd9/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java b/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
index 80b4537..5e4133e 100644
--- a/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
+++ b/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
@@ -207,6 +207,7 @@ public class Client implements IConnection {
while (msgs.hasNext()) {
if (!channel.isConnected()) {
connect();
+ channel = channelRef.get();
}
TaskMessage message = msgs.next();
if (null == messageBatch) {
@@ -351,3 +352,4 @@ public class Client implements IConnection {
});
}
}
+
[29/50] git commit: Merge branch 'storm-starter-readme' of
https://github.com/joliss/incubator-storm into pull-179
Posted by bo...@apache.org.
Merge branch 'storm-starter-readme' of https://github.com/joliss/incubator-storm into pull-179
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/4ac8dc3e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/4ac8dc3e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/4ac8dc3e
Branch: refs/heads/security
Commit: 4ac8dc3ed127134d6e8ff7d379bcabf083fef3b7
Parents: 7ac24b8 1b2ea79
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Thu Jul 24 16:09:37 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Thu Jul 24 16:09:37 2014 -0500
----------------------------------------------------------------------
examples/storm-starter/README.markdown | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
[21/50] git commit: Updated License, Changelog,
and README for STORM-364.
Posted by bo...@apache.org.
Updated License, Changelog, and README for STORM-364.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/7ac24b89
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/7ac24b89
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/7ac24b89
Branch: refs/heads/security
Commit: 7ac24b893c2162c96efb4dfdee16e6109510283b
Parents: aa0139c
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Jul 22 08:53:55 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Jul 22 08:53:55 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
LICENSE | 28 ++++++++++++++++++++++++++++
README.markdown | 1 +
3 files changed, 30 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/7ac24b89/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 661283f..a3a564e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,6 +23,7 @@
* STORM-254: one Spout/Bolt can register metric twice with same name in different timeBucket
* STORM-403: heartbeats-to-nimbus in supervisor-test failed due to uninten...
* STORM-402: FileNotFoundException when using storm with apache tika
+ * STORM-364: The exception time display as default timezone.
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/7ac24b89/LICENSE
----------------------------------------------------------------------
diff --git a/LICENSE b/LICENSE
index 4efd7ee..18ba484 100644
--- a/LICENSE
+++ b/LICENSE
@@ -405,3 +405,31 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+
+-----------------------------------------------------------------------
+
+For moment.js (storm-core/src/ui/public/js/moment.min.js)
+
+Copyright (c) 2011-2014 Tim Wood, Iskren Chernev, Moment.js contributors
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/7ac24b89/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 8a7afc3..341b62c 100644
--- a/README.markdown
+++ b/README.markdown
@@ -154,6 +154,7 @@ under the License.
* Aaron Levin ([@aaronlevin](https://github.com/aaronlevin))
* Masatake Iwasaki ([@iwasakims](https://github.com/iwasakims))
* Milad Fatenejad ([@icksa](https://github.com/icksa))
+* Ryan Liu ([@shell0dh](https://github.com/shell0dh))
## Acknowledgements
[36/50] git commit: Merge branch
'memoize-local-hostname-metrics-tick' of
https://github.com/strongh/incubator-storm into STORM-421
Posted by bo...@apache.org.
Merge branch 'memoize-local-hostname-metrics-tick' of https://github.com/strongh/incubator-storm into STORM-421
STORM-421: Memoize local hostname lookup in executor
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/6c2f08f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/6c2f08f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/6c2f08f0
Branch: refs/heads/security
Commit: 6c2f08f02364f3d117db483e39450ee849d7b5ec
Parents: 3a19ccc 28de7e5
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 13:45:14 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 13:45:14 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/executor.clj | 4 ++--
storm-core/src/clj/backtype/storm/util.clj | 2 ++
2 files changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
[45/50] git commit: Added STORM-155 to Changelog
Posted by bo...@apache.org.
Added STORM-155 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/a6bf4c3e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/a6bf4c3e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/a6bf4c3e
Branch: refs/heads/security
Commit: a6bf4c3ebd5e3211550c3634e7b0f85d9f6fadf0
Parents: 6eea872
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 16:28:34 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 16:28:34 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/a6bf4c3e/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eafa3d1..79c8b14 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -30,6 +30,7 @@
* STORM-414: support logging level to multilang protocol spout and bolt
* STORM-321: Added a tool to see the current status of STORM JIRA and github pulls.
* STORM-415: validate-launched-once in supervisor-test can not handle multiple topologies
+ * STORM-155: Storm rebalancing code causes multiple topologies assigned to a single port
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[39/50] git commit: Added STORM-414 to Changelog
Posted by bo...@apache.org.
Added STORM-414 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/dc8907b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/dc8907b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/dc8907b5
Branch: refs/heads/security
Commit: dc8907b5e4c896772ad0b38bee8d35fa5753e3f9
Parents: 5f59917
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 15:00:59 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 15:00:59 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/dc8907b5/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b722928..076aad7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -27,6 +27,7 @@
* STORM-420: Missing quotes in storm-starter python code
* STORM-399: Kafka Spout defaulting to latest offset when current offset is older then 100k
* STORM-421: Memoize local hostname lookup in executor
+ * STORM-414: support logging level to multilang protocol spout and bolt
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[22/50] git commit: added retrying infos to the messages on
inconsistent assignments.
Posted by bo...@apache.org.
added retrying infos to the messages on inconsistent assignments.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/cfda011f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/cfda011f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/cfda011f
Branch: refs/heads/security
Commit: cfda011f936d9bc23870e9b0687d3d55941f1ab0
Parents: 14bcc9b
Author: iwasakims <iw...@example.com>
Authored: Wed Jul 23 06:53:18 2014 -0700
Committer: iwasakims <iw...@example.com>
Committed: Wed Jul 23 06:53:18 2014 -0700
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/supervisor.clj | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/cfda011f/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 8d1ac46..ec96c10 100644
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@ -78,7 +78,7 @@
assignments)
(catch RuntimeException e
(if (> @retries 2) (throw e) (swap! retries inc))
- (log-warn (.getMessage e))
+ (log-warn (.getMessage e) ": retrying " @retries " of 3")
existing-assignment))))
(defn- read-storm-code-locations
[24/50] git commit: Fix netty client reconnect issue
Posted by bo...@apache.org.
Fix netty client reconnect issue
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/e1e6a602
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/e1e6a602
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/e1e6a602
Branch: refs/heads/security
Commit: e1e6a602e330d71410b1876ca9fb6bfc29761f35
Parents: 7ac24b8
Author: Kishor Patil <kp...@yahoo-inc.com>
Authored: Thu Jul 24 19:34:38 2014 +0000
Committer: Kishor Patil <kp...@yahoo-inc.com>
Committed: Thu Jul 24 19:34:38 2014 +0000
----------------------------------------------------------------------
.../src/jvm/backtype/storm/messaging/netty/Client.java | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/e1e6a602/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java b/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
index 8d2d221..80b4537 100644
--- a/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
+++ b/storm-core/src/jvm/backtype/storm/messaging/netty/Client.java
@@ -132,12 +132,12 @@ public class Client implements IConnection {
*/
private synchronized void connect() {
try {
- if (channelRef.get() != null) {
+
+ Channel channel = channelRef.get();
+ if (channel != null && channel.isConnected()) {
return;
}
- Channel channel = null;
-
int tried = 0;
while (tried <= max_retries) {
@@ -205,6 +205,9 @@ public class Client implements IConnection {
}
while (msgs.hasNext()) {
+ if (!channel.isConnected()) {
+ connect();
+ }
TaskMessage message = msgs.next();
if (null == messageBatch) {
messageBatch = new MessageBatch(messageBatchSize);
@@ -347,4 +350,4 @@ public class Client implements IConnection {
}
});
}
-}
\ No newline at end of file
+}
[23/50] git commit: Updated test so sort ordering is very explicit.
Posted by bo...@apache.org.
Updated test so sort ordering is very explicit.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/3414a79c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/3414a79c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/3414a79c
Branch: refs/heads/security
Commit: 3414a79c8f1f656c735360fb2bc97057a29e46cd
Parents: 7ac24b8
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Wed Jul 23 10:02:23 2014 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Wed Jul 23 10:02:23 2014 -0500
----------------------------------------------------------------------
storm-core/test/clj/backtype/storm/cluster_test.clj | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/3414a79c/storm-core/test/clj/backtype/storm/cluster_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/cluster_test.clj b/storm-core/test/clj/backtype/storm/cluster_test.clj
index a6bb10e..63efd30 100644
--- a/storm-core/test/clj/backtype/storm/cluster_test.clj
+++ b/storm-core/test/clj/backtype/storm/cluster_test.clj
@@ -208,8 +208,9 @@
(let [state (mk-storm-state zk-port)]
(.report-error state "a" "1"(local-hostname) 6700 (RuntimeException.))
(validate-errors! state "a" "1" ["RuntimeException"])
+ (advance-time-secs! 1)
(.report-error state "a" "1" (local-hostname) 6700 (IllegalArgumentException.))
- (validate-errors! state "a" "1" ["RuntimeException" "IllegalArgumentException"])
+ (validate-errors! state "a" "1" ["IllegalArgumentException" "RuntimeException"])
(doseq [i (range 10)]
(.report-error state "a" "2" (local-hostname) 6700 (RuntimeException.))
(advance-time-secs! 2))
[15/50] git commit: Updated Changelog with STORM-254
Posted by bo...@apache.org.
Updated Changelog with STORM-254
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/48cc069a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/48cc069a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/48cc069a
Branch: refs/heads/security
Commit: 48cc069ae6585716c45d5b903a6b428691062b76
Parents: 8fa0f78
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Jul 21 11:18:11 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Jul 21 11:18:11 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/48cc069a/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 777ed9a..76237d6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,7 @@
* STORM-381: Replace broken jquery.tablesorter.min.js to latest
* STORM-312: add storm monitor tools to monitor throughtput interactively
* STORM-354: Testing: allow users to pass TEST-TIMEOUT-MS as param for complete-topology
+ * STORM-254: one Spout/Bolt can register metric twice with same name in different timeBucket
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[44/50] git commit: Merge branch 'STORM-155' of
https://github.com/iwasakims/incubator-storm into STORM-155
Posted by bo...@apache.org.
Merge branch 'STORM-155' of https://github.com/iwasakims/incubator-storm into STORM-155
STORM-155: Storm rebalancing code causes multiple topologies assigned to a single port
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/6eea872b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/6eea872b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/6eea872b
Branch: refs/heads/security
Commit: 6eea872b52d6a926a0a90925abff783318b85626
Parents: 556ac8a cfda011
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 16:27:56 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 16:27:56 2014 -0500
----------------------------------------------------------------------
.../clj/backtype/storm/daemon/supervisor.clj | 26 ++++++---
.../test/clj/backtype/storm/supervisor_test.clj | 56 ++++++++++++++++++++
2 files changed, 74 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/6eea872b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/6eea872b/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
[32/50] git commit: missed out a pair of quotes
Posted by bo...@apache.org.
missed out a pair of quotes
line 90: attempt to read item 'directTask' from kwargs
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/498d8b22
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/498d8b22
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/498d8b22
Branch: refs/heads/security
Commit: 498d8b22262a7706ec25097d745f1f6c63c9cb35
Parents: 07a561a
Author: averykhoo <av...@gmail.com>
Authored: Fri Jul 11 15:25:26 2014 +0800
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Thu Jul 24 16:37:54 2014 -0500
----------------------------------------------------------------------
examples/storm-starter/multilang/resources/storm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/498d8b22/examples/storm-starter/multilang/resources/storm.py
----------------------------------------------------------------------
diff --git a/examples/storm-starter/multilang/resources/storm.py b/examples/storm-starter/multilang/resources/storm.py
index 4bc9e82..fdf7751 100644
--- a/examples/storm-starter/multilang/resources/storm.py
+++ b/examples/storm-starter/multilang/resources/storm.py
@@ -87,7 +87,7 @@ def emit(*args, **kwargs):
return readTaskIds()
def emitDirect(task, *args, **kwargs):
- kwargs[directTask] = task
+ kwargs['directTask'] = task
__emit(*args, **kwargs)
def __emit(*args, **kwargs):
[48/50] git commit: Merge branch 'netty-client-fix' of
https://github.com/kishorvpatil/incubator-storm into STORM-406
Posted by bo...@apache.org.
Merge branch 'netty-client-fix' of https://github.com/kishorvpatil/incubator-storm into STORM-406
STORM-406: Fix for reconnect logic in netty client
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/848a748c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/848a748c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/848a748c
Branch: refs/heads/security
Commit: 848a748c5a00de9fc24174b2f7ccb58e0fa393f7
Parents: c3a9a37 f1f5ecd
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Jul 29 07:16:40 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Jul 29 07:16:40 2014 -0500
----------------------------------------------------------------------
conf/defaults.yaml | 3 ++-
.../src/jvm/backtype/storm/messaging/netty/Client.java | 13 +++++++++----
2 files changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[27/50] git commit: Merge branch 'dp_typo' of
https://github.com/metadave/incubator-storm into fix-typo-pr-178
Posted by bo...@apache.org.
Merge branch 'dp_typo' of https://github.com/metadave/incubator-storm into fix-typo-pr-178
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/097a0720
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/097a0720
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/097a0720
Branch: refs/heads/security
Commit: 097a07204af215e813efebdb60fb079130056f98
Parents: 7ac24b8 92edcef
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Thu Jul 24 16:07:02 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Thu Jul 24 16:07:02 2014 -0500
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/spout/ISpout.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
[50/50] git commit: Merge branch 'master' into security-upmerge
Posted by bo...@apache.org.
Merge branch 'master' into security-upmerge
Conflicts:
bin/storm
conf/defaults.yaml
storm-core/src/clj/backtype/storm/daemon/drpc.clj
storm-core/src/clj/backtype/storm/daemon/logviewer.clj
storm-core/src/clj/backtype/storm/daemon/nimbus.clj
storm-core/src/clj/backtype/storm/daemon/supervisor.clj
storm-core/src/clj/backtype/storm/daemon/worker.clj
storm-core/src/clj/backtype/storm/testing.clj
storm-core/src/clj/backtype/storm/ui/helpers.clj
storm-core/src/clj/backtype/storm/util.clj
storm-core/src/jvm/backtype/storm/Config.java
storm-core/src/jvm/backtype/storm/utils/Utils.java
storm-core/src/ui/public/component.html
storm-core/test/clj/backtype/storm/supervisor_test.clj
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/559c883d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/559c883d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/559c883d
Branch: refs/heads/security
Commit: 559c883d5331362808b7e1ada647cbac76a88ab3
Parents: ff8336b b2a8a77
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Jul 29 15:57:03 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Jul 29 15:57:03 2014 -0500
----------------------------------------------------------------------
BYLAWS.md | 96 ++++
CHANGELOG.md | 31 +
LICENSE | 30 +-
README.markdown | 14 +
STORM-UI-REST-API.md | 567 +++++++++++++++++++
bin/storm | 77 ++-
conf/defaults.yaml | 5 +-
conf/storm_env.ini | 2 +-
dev-tools/github/__init__.py | 109 ++++
dev-tools/jira-github-join.py | 80 +++
dev-tools/jira/__init__.py | 232 ++++++++
examples/storm-starter/README.markdown | 30 +-
.../storm-starter/multilang/resources/storm.py | 2 +-
.../src/jvm/storm/starter/RollingTopWords.java | 62 +-
.../src/jvm/storm/starter/util/StormRunner.java | 9 +
.../storm-kafka/src/jvm/storm/kafka/Broker.java | 9 +-
.../src/jvm/storm/kafka/KafkaConfig.java | 2 +-
.../src/jvm/storm/kafka/Partition.java | 9 +-
pom.xml | 6 +-
.../src/clj/backtype/storm/LocalCluster.clj | 7 +-
storm-core/src/clj/backtype/storm/cluster.clj | 31 +-
.../src/clj/backtype/storm/command/monitor.clj | 37 ++
.../src/clj/backtype/storm/daemon/common.clj | 2 +-
.../src/clj/backtype/storm/daemon/drpc.clj | 6 +-
.../src/clj/backtype/storm/daemon/executor.clj | 4 +-
.../src/clj/backtype/storm/daemon/nimbus.clj | 7 +-
.../clj/backtype/storm/daemon/supervisor.clj | 104 +++-
.../src/clj/backtype/storm/daemon/worker.clj | 72 +--
storm-core/src/clj/backtype/storm/disruptor.clj | 2 +-
storm-core/src/clj/backtype/storm/event.clj | 2 +-
storm-core/src/clj/backtype/storm/testing.clj | 49 +-
storm-core/src/clj/backtype/storm/ui/core.clj | 2 +-
.../src/clj/backtype/storm/ui/helpers.clj | 5 -
storm-core/src/clj/backtype/storm/util.clj | 52 +-
storm-core/src/clj/backtype/storm/zookeeper.clj | 25 +
.../src/dev/resources/tester_bolt_metrics.py | 35 ++
.../src/dev/resources/tester_spout_metrics.py | 51 ++
storm-core/src/jvm/backtype/storm/Config.java | 171 +++---
.../jvm/backtype/storm/ConfigValidation.java | 70 +++
.../backtype/storm/messaging/netty/Client.java | 13 +-
.../metric/api/rpc/AssignableShellMetric.java | 30 +
.../metric/api/rpc/CombinedShellMetric.java | 31 +
.../storm/metric/api/rpc/CountShellMetric.java | 38 ++
.../storm/metric/api/rpc/IShellMetric.java | 31 +
.../metric/api/rpc/ReducedShellMetric.java | 32 ++
.../storm/multilang/JsonSerializer.java | 15 +
.../jvm/backtype/storm/multilang/ShellMsg.java | 46 ++
.../src/jvm/backtype/storm/spout/ISpout.java | 2 +-
.../jvm/backtype/storm/spout/ShellSpout.java | 72 ++-
.../src/jvm/backtype/storm/task/ShellBolt.java | 75 ++-
.../backtype/storm/task/TopologyContext.java | 28 +
.../storm/testing/PythonShellMetricsBolt.java | 32 ++
.../storm/testing/PythonShellMetricsSpout.java | 35 ++
.../src/jvm/backtype/storm/utils/Monitor.java | 249 ++++++++
.../jvm/backtype/storm/utils/ShellProcess.java | 46 +-
.../src/jvm/backtype/storm/utils/Utils.java | 23 +-
storm-core/src/multilang/py/storm.py | 30 +-
storm-core/src/multilang/rb/storm.rb | 24 +-
storm-core/src/ui/public/component.html | 3 +-
.../src/ui/public/js/jquery.tablesorter.min.js | 9 +-
storm-core/src/ui/public/js/moment.min.js | 6 +
storm-core/src/ui/public/js/script.js | 9 +
.../test/clj/backtype/storm/cluster_test.clj | 3 +-
.../test/clj/backtype/storm/config_test.clj | 41 +-
.../test/clj/backtype/storm/metrics_test.clj | 206 ++++---
.../test/clj/backtype/storm/supervisor_test.clj | 135 ++++-
66 files changed, 3003 insertions(+), 367 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/bin/storm
----------------------------------------------------------------------
diff --cc bin/storm
index 550e01d,a4aadb1..1b5be36
--- a/bin/storm
+++ b/bin/storm
@@@ -449,8 -468,7 +480,8 @@@ COMMANDS = {"jar": jar, "kill": kill, "
"drpc": drpc, "supervisor": supervisor, "localconfvalue": print_localconfvalue,
"remoteconfvalue": print_remoteconfvalue, "repl": repl, "classpath": print_classpath,
"activate": activate, "deactivate": deactivate, "rebalance": rebalance, "help": print_usage,
- "list": listtopos, "dev-zookeeper": dev_zookeeper, "version": version,
- "list": listtopos, "dev-zookeeper": dev_zookeeper, "version": version, "monitor": monitor}
++ "list": listtopos, "dev-zookeeper": dev_zookeeper, "version": version, "monitor": monitor,
+ "upload-credentials": upload_credentials}
def parse_config(config_list):
global CONFIG_OPTS
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/conf/defaults.yaml
----------------------------------------------------------------------
diff --cc conf/defaults.yaml
index d4283a4,2864adc..05948e1
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@@ -188,6 -150,7 +189,8 @@@ topology.max.error.report.per.interval
topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
topology.tuple.serializer: "backtype.storm.serialization.types.ListDelegateSerializer"
topology.trident.batch.emit.interval.millis: 500
+topology.testing.always.try.serialize: false
+ topology.classpath: null
+ topology.environment: null
dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/examples/storm-starter/src/jvm/storm/starter/util/StormRunner.java
----------------------------------------------------------------------
diff --cc examples/storm-starter/src/jvm/storm/starter/util/StormRunner.java
index f916ec6,f3017ce..eb25a86
--- a/examples/storm-starter/src/jvm/storm/starter/util/StormRunner.java
+++ b/examples/storm-starter/src/jvm/storm/starter/util/StormRunner.java
@@@ -19,6 -19,9 +19,10 @@@ package storm.starter.util
import backtype.storm.Config;
import backtype.storm.LocalCluster;
+ import backtype.storm.StormSubmitter;
+ import backtype.storm.generated.AlreadyAliveException;
++import backtype.storm.generated.AuthorizationException;
+ import backtype.storm.generated.InvalidTopologyException;
import backtype.storm.generated.StormTopology;
public final class StormRunner {
@@@ -36,4 -39,9 +40,9 @@@
cluster.killTopology(topologyName);
cluster.shutdown();
}
+
+ public static void runTopologyRemotely(StormTopology topology, String topologyName, Config conf)
- throws AlreadyAliveException, InvalidTopologyException {
++ throws AlreadyAliveException, InvalidTopologyException, AuthorizationException {
+ StormSubmitter.submitTopology(topologyName, conf, topology);
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/LocalCluster.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/cluster.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/cluster.clj
index 2c6f6f3,8ff5a2c..8ead710
--- a/storm-core/src/clj/backtype/storm/cluster.clj
+++ b/storm-core/src/clj/backtype/storm/cluster.clj
@@@ -25,15 -23,16 +25,17 @@@
(:require [backtype.storm.daemon [common :as common]]))
(defprotocol ClusterState
- (set-ephemeral-node [this path data])
+ (set-ephemeral-node [this path data acls])
(delete-node [this path])
- (create-sequential [this path data])
+ (create-sequential [this path data acls])
;; if node does not exist, create persistent with this data
- (set-data [this path data])
+ (set-data [this path data acls])
(get-data [this path watch?])
+ (get-version [this path watch?])
+ (get-data-with-version [this path watch?])
(get-children [this path watch?])
- (mkdirs [this path])
+ (mkdirs [this path acls])
+ (exists-node? [this path watch?])
(close [this])
(register [this callback])
(unregister [this id]))
@@@ -240,31 -231,31 +252,33 @@@
(into {}))))
;; Watches should be used for optimization. When ZK is reconnecting, they're not guaranteed to be called.
-(defn mk-storm-cluster-state
- [cluster-state-spec]
+(defnk mk-storm-cluster-state
+ [cluster-state-spec :acls nil]
(let [[solo? cluster-state] (if (satisfies? ClusterState cluster-state-spec)
[false cluster-state-spec]
- [true (mk-distributed-cluster-state cluster-state-spec)])
+ [true (mk-distributed-cluster-state cluster-state-spec :auth-conf cluster-state-spec :acls acls)])
assignment-info-callback (atom {})
+ assignment-info-with-version-callback (atom {})
+ assignment-version-callback (atom {})
supervisors-callback (atom nil)
assignments-callback (atom nil)
storm-base-callback (atom {})
+ credentials-callback (atom {})
state-id (register
- cluster-state
- (fn [type path]
- (let [[subtree & args] (tokenize-path path)]
- (condp = subtree
+ cluster-state
+ (fn [type path]
+ (let [[subtree & args] (tokenize-path path)]
+ (condp = subtree
ASSIGNMENTS-ROOT (if (empty? args)
- (issue-callback! assignments-callback)
- (issue-map-callback! assignment-info-callback (first args)))
+ (issue-callback! assignments-callback)
+ (issue-map-callback! assignment-info-callback (first args)))
SUPERVISORS-ROOT (issue-callback! supervisors-callback)
STORMS-ROOT (issue-map-callback! storm-base-callback (first args))
+ CREDENTIALS-ROOT (issue-map-callback! credentials-callback (first args))
;; this should never happen
- (halt-process! 30 "Unknown callback for subtree " subtree args)))))]
+ (exit-process! 30 "Unknown callback for subtree " subtree args)))))]
(doseq [p [ASSIGNMENTS-SUBTREE STORMS-SUBTREE SUPERVISORS-SUBTREE WORKERBEATS-SUBTREE ERRORS-SUBTREE]]
- (mkdirs cluster-state p))
+ (mkdirs cluster-state p acls))
(reify
StormClusterState
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/daemon/common.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/daemon/drpc.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/daemon/drpc.clj
index 1340e49,3527b7c..68128c3
--- a/storm-core/src/clj/backtype/storm/daemon/drpc.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/drpc.clj
@@@ -199,48 -133,37 +199,48 @@@
(let [conf (read-storm-config)
worker-threads (int (conf DRPC-WORKER-THREADS))
queue-size (int (conf DRPC-QUEUE-SIZE))
- service-handler (service-handler)
-
- ;; Requests and returns need to be on separate thread pools, since
- ;; calls to "execute" don't unblock until other thrift methods are
- ;; called. So if 64 threads are calling execute, the server won't
- ;; accept the result invocations that will unblock those threads.
-
- handler-server
- (THsHaServer. (-> (TNonblockingServerSocket. (int (conf DRPC-PORT)))
- (THsHaServer$Args.)
- (.workerThreads 64)
- (.executorService
- (ThreadPoolExecutor.
- worker-threads worker-threads 60 TimeUnit/SECONDS
- (ArrayBlockingQueue. queue-size)))
- (.protocolFactory (TBinaryProtocol$Factory.))
- (.processor (DistributedRPC$Processor. service-handler))))
-
- invoke-server
- (THsHaServer. (-> (TNonblockingServerSocket. (int (conf DRPC-INVOCATIONS-PORT)))
- (THsHaServer$Args.)
- (.workerThreads 64)
- (.protocolFactory (TBinaryProtocol$Factory.))
- (.processor
- (DistributedRPCInvocations$Processor. service-handler))))]
+ drpc-http-port (int (conf DRPC-HTTP-PORT))
+ drpc-port (int (conf DRPC-PORT))
+ drpc-service-handler (service-handler conf)
+ ;; requests and returns need to be on separate thread pools, since calls to
+ ;; "execute" don't unblock until other thrift methods are called. So if
+ ;; 64 threads are calling execute, the server won't accept the result
+ ;; invocations that will unblock those threads
+ handler-server (when (> drpc-port 0)
+ (ThriftServer. conf
+ (DistributedRPC$Processor. drpc-service-handler)
+ ThriftConnectionType/DRPC))
+ invoke-server (ThriftServer. conf
+ (DistributedRPCInvocations$Processor. drpc-service-handler)
+ ThriftConnectionType/DRPC_INVOCATIONS)
+ http-creds-handler (AuthUtils/GetDrpcHttpCredentialsPlugin conf)]
- (.addShutdownHook (Runtime/getRuntime) (Thread. (fn []
- (if handler-server (.stop handler-server))
- (.stop invoke-server))))
+ (add-shutdown-hook-with-force-kill-in-1-sec (fn []
- (.stop handler-server)
++ (if handler-server (.stop handler-server))
+ (.stop invoke-server)))
(log-message "Starting Distributed RPC servers...")
(future (.serve invoke-server))
- (.serve handler-server))))
+ (when (> drpc-http-port 0)
+ (let [app (webapp drpc-service-handler http-creds-handler)
+ filter-class (conf DRPC-HTTP-FILTER)
+ filter-params (conf DRPC-HTTP-FILTER-PARAMS)
+ filters-confs [{:filter-class filter-class
+ :filter-params filter-params}]
+ https-port (int (conf DRPC-HTTPS-PORT))
+ https-ks-path (conf DRPC-HTTPS-KEYSTORE-PATH)
+ https-ks-password (conf DRPC-HTTPS-KEYSTORE-PASSWORD)
+ https-ks-type (conf DRPC-HTTPS-KEYSTORE-TYPE)]
+
+ (run-jetty app
+ {:port drpc-http-port :join? false
+ :configurator (fn [server]
+ (config-ssl server
+ https-port
+ https-ks-path
+ https-ks-password
+ https-ks-type)
+ (config-filter server app filters-confs))})))
+ (when handler-server
+ (.serve handler-server)))))
(defn -main []
(launch-server!))
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/daemon/executor.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/daemon/nimbus.clj
index e2952d0,6265479..44ccc6c
--- a/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/nimbus.clj
@@@ -81,11 -73,9 +81,11 @@@
:validator (new-instance (conf NIMBUS-TOPOLOGY-VALIDATOR))
:timer (mk-timer :kill-fn (fn [t]
(log-error t "Error when processing event")
- (halt-process! 20 "Error when processing an event")
+ (exit-process! 20 "Error when processing an event")
))
:scheduler (mk-scheduler conf inimbus)
+ :id->sched-status (atom {})
+ :cred-renewers (AuthUtils/GetCredentialRenewers conf)
}))
(defn inbox [nimbus]
@@@ -1304,13 -1148,18 +1304,14 @@@
(defn launch-server! [conf nimbus]
(validate-distributed-mode! conf)
(let [service-handler (service-handler conf nimbus)
- ;;TODO need to honor NIMBUS-THRIFT-MAX-BUFFER-SIZE for different transports
- options (-> (TNonblockingServerSocket. (int (conf NIMBUS-THRIFT-PORT)))
- (THsHaServer$Args.)
- (.workerThreads 64)
- (.protocolFactory (TBinaryProtocol$Factory. false true (conf NIMBUS-THRIFT-MAX-BUFFER-SIZE)))
- (.processor (Nimbus$Processor. service-handler))
- )
- server (THsHaServer. (do (set! (. options maxReadBufferBytes)(conf NIMBUS-THRIFT-MAX-BUFFER-SIZE)) options))]
+ server (ThriftServer. conf (Nimbus$Processor. service-handler)
+ ThriftConnectionType/NIMBUS)]
- (.addShutdownHook (Runtime/getRuntime) (Thread. (fn [] (.shutdown service-handler) (.stop server))))
+ (add-shutdown-hook-with-force-kill-in-1-sec (fn []
+ (.shutdown service-handler)
+ (.stop server)))
(log-message "Starting Nimbus server...")
- (.serve server)))
+ (.serve server)
+ service-handler))
;; distributed implementation
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/daemon/supervisor.clj
index 2fe7fb8,cfa8f85..d8ff6b5
--- a/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/supervisor.clj
@@@ -14,8 -14,8 +14,9 @@@
;; See the License for the specific language governing permissions and
;; limitations under the License.
(ns backtype.storm.daemon.supervisor
+ (:import [java.io OutputStreamWriter BufferedWriter IOException])
- (:import [backtype.storm.scheduler ISupervisor])
+ (:import [backtype.storm.scheduler ISupervisor]
+ [java.net JarURLConnection])
(:use [backtype.storm bootstrap])
(:use [backtype.storm.daemon common])
(:require [backtype.storm.daemon [worker :as worker]])
@@@ -235,22 -190,16 +253,26 @@@
(when thread-pid
(psim/kill-process thread-pid))
(doseq [pid pids]
- (kill-process-with-sig-term pid))
+ (if as-user
++ (worker-launcher-and-wait conf user ["signal" pid "9"] :log-prefix (str "kill -15 " pid))
++ (kill-process-with-sig-term pid)))
+ (if-not (empty? pids) (sleep-secs 1)) ;; allow 1 second for execution of cleanup threads on worker.
+ (doseq [pid pids]
- (force-kill-process pid)
- (try
- (rmpath (worker-pid-path conf id pid))
- (catch Exception e))) ;; on windows, the supervisor may still holds the lock on the worker directory
- (try-cleanup-worker conf id))
++ (if as-user
+ (worker-launcher-and-wait conf user ["signal" pid "9"] :log-prefix (str "kill -9 " pid))
- (ensure-process-killed! pid))
++ (force-kill-process pid))
+ (if as-user
- (rmr-as-user conf id user (worker-pid-path conf id pid))
++ (rmr-as-user conf id user (worker-pid-path conf id pid))
+ (try
+ (rmpath (worker-pid-path conf id pid))
- (catch Exception e)) ;; on windows, the supervisor may still holds the lock on the worker directory
- ))
++ (catch Exception e)))) ;; on windows, the supervisor may still holds the lock on the worker directory
+ (try-cleanup-worker conf id user))
(log-message "Shut down " (:supervisor-id supervisor) ":" id))
+(def SUPERVISOR-ZK-ACLS
+ [(first ZooDefs$Ids/CREATOR_ALL_ACL)
+ (ACL. (bit-or ZooDefs$Perms/READ ZooDefs$Perms/CREATE) ZooDefs$Ids/ANYONE_ID_UNSAFE)])
+
(defn supervisor-data [conf shared-context ^ISupervisor isupervisor]
{:conf conf
:shared-context shared-context
@@@ -582,14 -495,20 +612,23 @@@
jlp (jlp stormroot conf)
stormjar (supervisor-stormjar-path stormroot)
storm-conf (read-supervisor-storm-conf conf storm-id)
- classpath (add-to-classpath (current-classpath) [stormjar])
+ topo-classpath (if-let [cp (storm-conf TOPOLOGY-CLASSPATH)]
+ [cp]
+ [])
+ classpath (-> (current-classpath)
+ (add-to-classpath [stormjar])
+ (add-to-classpath topo-classpath))
+ top-gc-opts (storm-conf TOPOLOGY-WORKER-GC-CHILDOPTS)
+ gc-opts (substitute-childopts (if top-gc-opts top-gc-opts (conf WORKER-GC-CHILDOPTS)) worker-id storm-id port)
+ user (storm-conf TOPOLOGY-SUBMITTER-USER)
+ logfilename (logs-filename storm-id port)
-
- worker-childopts (substitute-childopts (conf WORKER-CHILDOPTS) worker-id storm-id port)
- topo-worker-childopts (substitute-childopts (storm-conf TOPOLOGY-WORKER-CHILDOPTS) worker-id storm-id port)
+ worker-childopts (when-let [s (conf WORKER-CHILDOPTS)]
- (substitute-worker-childopts s port))
++ (substitute-childopts s worker-id storm-id port))
+ topo-worker-childopts (when-let [s (storm-conf TOPOLOGY-WORKER-CHILDOPTS)]
- (substitute-worker-childopts s port))
++ (substitute-childopts s worker-id storm-id port))
+ topology-worker-environment (if-let [env (storm-conf TOPOLOGY-ENVIRONMENT)]
+ (merge env {"LD_LIBRARY_PATH" jlp})
+ {"LD_LIBRARY_PATH" jlp})
- logfilename (str "worker-" port ".log")
command (concat
[(java-cmd) "-server"]
worker-childopts
@@@ -608,21 -526,13 +647,20 @@@
(:assignment-id supervisor)
port
worker-id])
- command (->> command (map str) (filter (complement empty?)))
- shell-cmd (->> command
- (map #(str \' (clojure.string/escape % {\' "\\'"}) \'))
- (clojure.string/join " "))]
- (log-message "Launching worker with command: " shell-cmd)
- (launch-process command :environment topology-worker-environment)
- ))
+ command (->> command (map str) (filter (complement empty?)))]
-
+ (log-message "Launching worker with command: " (shell-cmd command))
+ (write-log-metadata! storm-conf user worker-id storm-id port conf)
+ (set-worker-user! conf worker-id user)
+ (let [log-prefix (str "Worker Process " worker-id)
+ callback (fn [exit-code]
+ (log-message log-prefix " exited with code: " exit-code)
+ (add-dead-worker worker-id))]
+ (remove-dead-worker worker-id)
+ (if run-worker-as-user
+ (let [worker-dir (worker-root conf worker-id)]
- (worker-launcher conf user ["worker" worker-dir (write-script worker-dir command :environment {"LD_LIBRARY_PATH" jlp})] :log-prefix log-prefix :exit-code-callback callback))
- (launch-process command :environment {"LD_LIBRARY_PATH" jlp} :log-prefix log-prefix :exit-code-callback callback)
++ (worker-launcher conf user ["worker" worker-dir (write-script worker-dir command :environment topology-worker-environment)] :log-prefix log-prefix :exit-code-callback callback))
++ (launch-process command :environment topology-worker-environment :log-prefix log-prefix :exit-code-callback callback)
+ ))))
;; local implementation
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/daemon/worker.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/daemon/worker.clj
index d09ea35,aeabdf6..f1acdec
--- a/storm-core/src/clj/backtype/storm/daemon/worker.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/worker.clj
@@@ -190,17 -175,22 +191,19 @@@
)
:timer-name timer-name))
- (defn recursive-map-worker-data [conf mq-context storm-id assignment-id port
- storm-conf
- worker-id
- cluster-state
- storm-cluster-state
- executors
- transfer-queue
- executor-receive-queue-map
- receive-queue-map
- topology]
- (recursive-map
-(defn worker-data [conf mq-context storm-id assignment-id port worker-id]
++(defn worker-data [conf mq-context storm-id assignment-id port worker-id storm-conf cluster-state storm-cluster-state]
+ (let [assignment-versions (atom {})
- cluster-state (cluster/mk-distributed-cluster-state conf)
- storm-cluster-state (cluster/mk-storm-cluster-state cluster-state)
- storm-conf (read-supervisor-storm-conf conf storm-id)
+ executors (set (read-worker-executors storm-conf storm-cluster-state storm-id assignment-id port assignment-versions))
+ transfer-queue (disruptor/disruptor-queue "worker-transfer-queue" (storm-conf TOPOLOGY-TRANSFER-BUFFER-SIZE)
+ :wait-strategy (storm-conf TOPOLOGY-DISRUPTOR-WAIT-STRATEGY))
+ executor-receive-queue-map (mk-receive-queue-map storm-conf executors)
+
+ receive-queue-map (->> executor-receive-queue-map
+ (mapcat (fn [[e queue]] (for [t (executor-id->tasks e)] [t queue])))
+ (into {}))
+
+ topology (read-supervisor-topology conf storm-id)]
+ (recursive-map
:conf conf
:mq-context (if mq-context
mq-context
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/testing.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/testing.clj
index 39f3759,54f40e0..f603086
--- a/storm-core/src/clj/backtype/storm/testing.clj
+++ b/storm-core/src/clj/backtype/storm/testing.clj
@@@ -208,8 -208,8 +213,8 @@@
supervisors
; because a worker may already be dead
workers)]
- (while-timeout TEST-TIMEOUT-MS (not (every? (memfn waiting?) daemons))
+ (while-timeout timeout-ms (not (every? (memfn waiting?) daemons))
- (Thread/sleep 10)
+ (Thread/sleep (rand-int 20))
;; (doseq [d daemons]
;; (if-not ((memfn waiting?) d)
;; (println d)))
@@@ -481,10 -483,7 +490,10 @@@
(submit-local-topology (:nimbus cluster-map) storm-name storm-conf topology)
(let [storm-id (common/get-storm-id state storm-name)]
+ ;;Give the topology time to come up without using it to wait for the spouts to complete
+ (simulate-wait cluster-map)
+
- (while-timeout TEST-TIMEOUT-MS (not (every? exhausted? (spout-objects spouts)))
+ (while-timeout timeout-ms (not (every? exhausted? (spout-objects spouts)))
(simulate-wait cluster-map))
(.killTopologyWithOpts (:nimbus cluster-map) storm-name (doto (KillOptions.) (.set_wait_secs 0)))
@@@ -583,22 -582,25 +592,24 @@@
(defn tracked-wait
"Waits until topology is idle and 'amt' more tuples have been emitted by spouts."
([tracked-topology]
- (tracked-wait tracked-topology 1))
+ (tracked-wait tracked-topology 1 TEST-TIMEOUT-MS))
([tracked-topology amt]
+ (tracked-wait tracked-topology amt TEST-TIMEOUT-MS))
+ ([tracked-topology amt timeout-ms]
- (let [target (+ amt @(:last-spout-emit tracked-topology))
- track-id (-> tracked-topology :cluster ::track-id)
- waiting? (fn []
- (or (not= target (global-amt track-id "spout-emitted"))
- (not= (global-amt track-id "transferred")
- (global-amt track-id "processed"))
- ))]
- (while-timeout TEST-TIMEOUT-MS (waiting?)
- ;; (println "Spout emitted: " (global-amt track-id "spout-emitted"))
- ;; (println "Processed: " (global-amt track-id "processed"))
- ;; (println "Transferred: " (global-amt track-id "transferred"))
- (Thread/sleep 500))
- (reset! (:last-spout-emit tracked-topology) target))))
-
-(defnk test-tuple
+ (let [target (+ amt @(:last-spout-emit tracked-topology))
+ track-id (-> tracked-topology :cluster ::track-id)
+ waiting? (fn []
+ (or (not= target (global-amt track-id "spout-emitted"))
+ (not= (global-amt track-id "transferred")
+ (global-amt track-id "processed"))))]
- (while-timeout TEST-TIMEOUT-MS (waiting?)
++ (while-timeout timeout-ms (waiting?)
+ ;; (println "Spout emitted: " (global-amt track-id "spout-emitted"))
+ ;; (println "Processed: " (global-amt track-id "processed"))
+ ;; (println "Transferred: " (global-amt track-id "transferred"))
+ (Thread/sleep (rand-int 200)))
+ (reset! (:last-spout-emit tracked-topology) target))))
+
+(defnk test-tuple
[values
:stream Utils/DEFAULT_STREAM_ID
:component "component"
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/ui/core.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/ui/core.clj
index 3c1be09,13e4d41..ec9759c
--- a/storm-core/src/clj/backtype/storm/ui/core.clj
+++ b/storm-core/src/clj/backtype/storm/ui/core.clj
@@@ -728,10 -688,10 +728,10 @@@
reverse)]
{"componentErrors"
(for [^ErrorInfo e errors]
- {"time" (date-str (.get_error_time_secs e))
+ {"time" (* 1000 (long (.get_error_time_secs e)))
"errorHost" (.get_host e)
"errorPort" (.get_port e)
- "errorWorkerLogLink" (worker-log-link (.get_host e) (.get_port e))
+ "errorWorkerLogLink" (worker-log-link (.get_host e) (.get_port e) topology-id)
"error" (.get_error e)})}))
(defn spout-stats
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/ui/helpers.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/util.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/util.clj
index ed2e30b,3df25b7..0173081
--- a/storm-core/src/clj/backtype/storm/util.clj
+++ b/storm-core/src/clj/backtype/storm/util.clj
@@@ -398,25 -411,30 +414,35 @@@
(catch ExecuteException e
(log-message "Error when trying to kill " pid ". Process is probably already dead."))))
+(defn read-and-log-stream
+ [prefix stream]
+ (try
+ (let [reader (BufferedReader. (InputStreamReader. stream))]
+ (loop []
+ (if-let [line (.readLine reader)]
+ (do
+ (log-warn (str prefix ":" line))
+ (recur)))))
+ (catch IOException e
+ (log-warn "Error while trying to log stream" e))))
+
- (defn sleep-secs [secs]
- (when (pos? secs)
- (Time/sleep (* (long secs) 1000))))
+ (defn force-kill-process
+ [pid]
+ (send-signal-to-process pid sig-kill))
- (defn sleep-until-secs [target-secs]
- (Time/sleepUntil (* (long target-secs) 1000)))
+ (defn kill-process-with-sig-term
+ [pid]
+ (send-signal-to-process pid sig-term))
+
+ (defn add-shutdown-hook-with-force-kill-in-1-sec
+ "adds the user supplied function as a shutdown hook for cleanup.
+ Also adds a function that sleeps for a second and then sends kill -9 to process to avoid any zombie process in case
+ cleanup function hangs."
+ [func]
+ (.addShutdownHook (Runtime/getRuntime) (Thread. #(func)))
+ (.addShutdownHook (Runtime/getRuntime) (Thread. #((sleep-secs 1)
+ (.halt (Runtime/getRuntime) 20)))))
-(defnk launch-process [command :environment {}]
- (let [builder (ProcessBuilder. command)
- process-env (.environment builder)]
- (doseq [[k v] environment]
- (.put process-env k v))
- (.start builder)))
-
(defprotocol SmartThread
(start [this])
(join [this])
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/clj/backtype/storm/zookeeper.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/jvm/backtype/storm/Config.java
----------------------------------------------------------------------
diff --cc storm-core/src/jvm/backtype/storm/Config.java
index ea54313,ac8b6b6..d6b45ea
--- a/storm-core/src/jvm/backtype/storm/Config.java
+++ b/storm-core/src/jvm/backtype/storm/Config.java
@@@ -17,10 -17,8 +17,9 @@@
*/
package backtype.storm;
- import backtype.storm.ConfigValidation;
import backtype.storm.serialization.IKryoDecorator;
import backtype.storm.serialization.IKryoFactory;
+import backtype.storm.utils.Utils;
import com.esotericsoftware.kryo.Serializer;
import java.util.ArrayList;
import java.util.HashMap;
@@@ -66,20 -57,20 +65,20 @@@ public class Config extends HashMap<Str
/**
* Netty based messaging: The max # of retries that a peer will perform when a remote is not accessible
*/
- public static final String STORM_MESSAGING_NETTY_MAX_RETRIES = "storm.messaging.netty.max_retries";
- public static final Object STORM_MESSAGING_NETTY_MAX_RETRIES_SCHEMA = Number.class;
+ public static final String STORM_MESSAGING_NETTY_MAX_RETRIES = "storm.messaging.netty.max_retries";
+ public static final Object STORM_MESSAGING_NETTY_MAX_RETRIES_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * Netty based messaging: The min # of milliseconds that a peer will wait.
+ * Netty based messaging: The min # of milliseconds that a peer will wait.
*/
- public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = "storm.messaging.netty.min_wait_ms";
- public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = Number.class;
+ public static final String STORM_MESSAGING_NETTY_MIN_SLEEP_MS = "storm.messaging.netty.min_wait_ms";
+ public static final Object STORM_MESSAGING_NETTY_MIN_SLEEP_MS_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * Netty based messaging: The max # of milliseconds that a peer will wait.
+ * Netty based messaging: The max # of milliseconds that a peer will wait.
*/
- public static final String STORM_MESSAGING_NETTY_MAX_SLEEP_MS = "storm.messaging.netty.max_wait_ms";
- public static final Object STORM_MESSAGING_NETTY_MAX_SLEEP_MS_SCHEMA = Number.class;
+ public static final String STORM_MESSAGING_NETTY_MAX_SLEEP_MS = "storm.messaging.netty.max_wait_ms";
+ public static final Object STORM_MESSAGING_NETTY_MAX_SLEEP_MS_SCHEMA = ConfigValidation.IntegerValidator;
/**
* Netty based messaging: The # of worker threads for the server.
@@@ -103,9 -94,9 +102,8 @@@
* We check with this interval that whether the Netty channel is writable and try to write pending messages
*/
public static final String STORM_NETTY_FLUSH_CHECK_INTERVAL_MS = "storm.messaging.netty.flush.check.interval.ms";
- public static final Object STORM_NETTY_FLUSH_CHECK_INTERVAL_MS_SCHEMA = Number.class;
-
-
+ public static final Object STORM_NETTY_FLUSH_CHECK_INTERVAL_MS_SCHEMA = ConfigValidation.IntegerValidator;
+
-
/**
* A list of hosts of ZooKeeper servers used to manage the cluster.
*/
@@@ -234,10 -200,10 +232,10 @@@
* The ceiling of the interval between retries of a Zookeeper operation.
*/
public static final String STORM_ZOOKEEPER_RETRY_INTERVAL_CEILING="storm.zookeeper.retry.intervalceiling.millis";
- public static final Object STORM_ZOOKEEPER_RETRY_INTERVAL_CEILING_SCHEMA = Number.class;
+ public static final Object STORM_ZOOKEEPER_RETRY_INTERVAL_CEILING_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * The Zookeeper authentication scheme to use, e.g. "digest". Defaults to no authentication.
+ * The cluster Zookeeper authentication scheme to use, e.g. "digest". Defaults to no authentication.
*/
public static final String STORM_ZOOKEEPER_AUTH_SCHEME="storm.zookeeper.auth.scheme";
public static final Object STORM_ZOOKEEPER_AUTH_SCHEME_SCHEMA = String.class;
@@@ -307,35 -231,15 +305,35 @@@
* connect to this port to upload jars and submit topologies.
*/
public static final String NIMBUS_THRIFT_PORT = "nimbus.thrift.port";
- public static final Object NIMBUS_THRIFT_PORT_SCHEMA = Number.class;
+ public static final Object NIMBUS_THRIFT_PORT_SCHEMA = ConfigValidation.IntegerValidator;
/**
+ * The number of threads that should be used by the nimbus thrift server.
+ */
+ public static final String NIMBUS_THRIFT_THREADS = "nimbus.thrift.threads";
+ public static final Object NIMBUS_THRIFT_THREADS_SCHEMA = Number.class;
+
+ /**
+ * A list of users that are cluster admins and can run any command. To use this set
+ * nimbus.authorizer to backtype.storm.security.auth.authorizer.SimpleACLAuthorizer
+ */
+ public static final String NIMBUS_ADMINS = "nimbus.admins";
+ public static final Object NIMBUS_ADMINS_SCHEMA = ConfigValidation.StringsValidator;
+
+ /**
+ * A list of users that run the supervisors and should be authorized to interact with
+ * nimbus as a supervisor would. To use this set
+ * nimbus.authorizer to backtype.storm.security.auth.authorizer.SimpleACLAuthorizer
+ */
+ public static final String NIMBUS_SUPERVISOR_USERS = "nimbus.supervisor.users";
+ public static final Object NIMBUS_SUPERVISOR_USERS_SCHEMA = ConfigValidation.StringsValidator;
+
+ /**
* The maximum buffer size thrift should use when reading messages.
*/
public static final String NIMBUS_THRIFT_MAX_BUFFER_SIZE = "nimbus.thrift.max_buffer_size";
- public static final Object NIMBUS_THRIFT_MAX_BUFFER_SIZE_SCHEMA = Number.class;
+ public static final Object NIMBUS_THRIFT_MAX_BUFFER_SIZE_SCHEMA = ConfigValidation.IntegerValidator;
-
/**
* This parameter is used by the storm-deploy project to configure the
* jvm options for the nimbus daemon.
@@@ -395,10 -299,10 +393,10 @@@
* to launching new JVM's and configuring them.</p>
*/
public static final String NIMBUS_TASK_LAUNCH_SECS = "nimbus.task.launch.secs";
- public static final Object NIMBUS_TASK_LAUNCH_SECS_SCHEMA = Number.class;
+ public static final Object NIMBUS_TASK_LAUNCH_SECS_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * Whether or not nimbus should reassign tasks if it detects that a task goes down.
+ * Whether or not nimbus should reassign tasks if it detects that a task goes down.
* Defaults to true, and it's not recommended to change this value.
*/
public static final String NIMBUS_REASSIGN = "nimbus.reassign";
@@@ -565,77 -369,27 +563,77 @@@
* This port is used by Storm DRPC for receiving DPRC requests from clients.
*/
public static final String DRPC_PORT = "drpc.port";
- public static final Object DRPC_PORT_SCHEMA = Number.class;
+ public static final Object DRPC_PORT_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * DRPC thrift server worker threads
+ * Class name for authorization plugin for DRPC client
+ */
+ public static final String DRPC_AUTHORIZER = "drpc.authorizer";
+ public static final Object DRPC_AUTHORIZER_SCHEMA = String.class;
+
+ /**
+ * The Access Control List for the DRPC Authorizer.
+ * @see DRPCSimpleAclAuthorizer
+ */
+ public static final String DRPC_AUTHORIZER_ACL = "drpc.authorizer.acl";
+ public static final Object DRPC_AUTHORIZER_ACL_SCHEMA = Map.class;
+
+ /**
+ * File name of the DRPC Authorizer ACL.
+ * @see DRPCSimpleAclAuthorizer
+ */
+ public static final String DRPC_AUTHORIZER_ACL_FILENAME = "drpc.authorizer.acl.filename";
+ public static final Object DRPC_AUTHORIZER_ACL_FILENAME_SCHEMA = String.class;
+
+ /**
+ * Whether the DRPCSimpleAclAuthorizer should deny requests for operations
+ * involving functions that have no explicit ACL entry. When set to false
+ * (the default) DRPC functions that have no entry in the ACL will be
+ * permitted, which is appropriate for a development environment. When set
+ * to true, explicit ACL entries are required for every DRPC function, and
+ * any request for functions will be denied.
+ * @see DRPCSimpleAclAuthorizer
+ */
+ public static final String DRPC_AUTHORIZER_ACL_STRICT = "drpc.authorizer.acl.strict";
+ public static final Object DRPC_AUTHORIZER_ACL_STRICT_SCHEMA = Boolean.class;
+
+ /**
+ * DRPC thrift server worker threads
*/
public static final String DRPC_WORKER_THREADS = "drpc.worker.threads";
- public static final Object DRPC_WORKER_THREADS_SCHEMA = Number.class;
+ public static final Object DRPC_WORKER_THREADS_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * DRPC thrift server queue size
+ * The maximum buffer size thrift should use when reading messages for DRPC.
+ */
+ public static final String DRPC_MAX_BUFFER_SIZE = "drpc.max_buffer_size";
+ public static final Object DRPC_MAX_BUFFER_SIZE_SCHEMA = Number.class;
+
+ /**
+ * DRPC thrift server queue size
*/
public static final String DRPC_QUEUE_SIZE = "drpc.queue.size";
- public static final Object DRPC_QUEUE_SIZE_SCHEMA = Number.class;
+ public static final Object DRPC_QUEUE_SIZE_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * This port on Storm DRPC is used by DRPC topologies to receive function invocations and send results back.
+ * The DRPC invocations transport plug-in for Thrift client/server communication
+ */
+ public static final String DRPC_INVOCATIONS_THRIFT_TRANSPORT_PLUGIN = "drpc.invocations.thrift.transport";
+ public static final Object DRPC_INVOCATIONS_THRIFT_TRANSPORT_PLUGIN_SCHEMA = String.class;
+
+ /**
+ * This port on Storm DRPC is used by DRPC topologies to receive function invocations and send results back.
*/
public static final String DRPC_INVOCATIONS_PORT = "drpc.invocations.port";
- public static final Object DRPC_INVOCATIONS_PORT_SCHEMA = Number.class;
+ public static final Object DRPC_INVOCATIONS_PORT_SCHEMA = ConfigValidation.IntegerValidator;
/**
+ * DRPC invocations thrift server worker threads
+ */
+ public static final String DRPC_INVOCATIONS_THREADS = "drpc.invocations.threads";
+ public static final Object DRPC_INVOCATIONS_THREADS_SCHEMA = Number.class;
+
+ /**
* The timeout on DRPC requests within the DRPC server. Defaults to 10 minutes. Note that requests can also
* timeout based on the socket timeout on the DRPC client, and separately based on the topology message
* timeout for the topology implementing the DRPC function.
@@@ -672,32 -414,8 +670,32 @@@
* how many workers run on each machine.
*/
public static final String SUPERVISOR_SLOTS_PORTS = "supervisor.slots.ports";
- public static final Object SUPERVISOR_SLOTS_PORTS_SCHEMA = ConfigValidation.NumbersValidator;
+ public static final Object SUPERVISOR_SLOTS_PORTS_SCHEMA = ConfigValidation.IntegersValidator;
+ /**
+ * A number representing the maximum number of workers any single topology can acquire.
+ */
+ public static final String NIMBUS_SLOTS_PER_TOPOLOGY = "nimbus.slots.perTopology";
+ public static final Object NIMBUS_SLOTS_PER_TOPOLOGY_SCHEMA = Number.class;
+
+ /**
+ * A class implementing javax.servlet.Filter for DRPC HTTP requests
+ */
+ public static final String DRPC_HTTP_FILTER = "drpc.http.filter";
+ public static final Object DRPC_HTTP_FILTER_SCHEMA = String.class;
+
+ /**
+ * Initialization parameters for the javax.servlet.Filter of the DRPC HTTP
+ * service
+ */
+ public static final String DRPC_HTTP_FILTER_PARAMS = "drpc.http.filter.params";
+ public static final Object DRPC_HTTP_FILTER_PARAMS_SCHEMA = Map.class;
+
+ /**
+ * A number representing the maximum number of executors any single topology can acquire.
+ */
+ public static final String NIMBUS_EXECUTORS_PER_TOPOLOGY = "nimbus.executors.perTopology";
+ public static final Object NIMBUS_EXECUTORS_PER_TOPOLOGY_SCHEMA = Number.class;
/**
* This parameter is used by the storm-deploy project to configure the
@@@ -711,8 -430,9 +709,8 @@@
* restart the worker process.
*/
public static final String SUPERVISOR_WORKER_TIMEOUT_SECS = "supervisor.worker.timeout.secs";
- public static final Object SUPERVISOR_WORKER_TIMEOUT_SECS_SCHEMA = Number.class;
+ public static final Object SUPERVISOR_WORKER_TIMEOUT_SECS_SCHEMA = ConfigValidation.IntegerValidator;
-
/**
* How long a worker can go without heartbeating during the initial launch before
* the supervisor tries to restart the worker process. This value override
@@@ -720,8 -440,9 +718,8 @@@
* overhead to starting and configuring the JVM on launch.
*/
public static final String SUPERVISOR_WORKER_START_TIMEOUT_SECS = "supervisor.worker.start.timeout.secs";
- public static final Object SUPERVISOR_WORKER_START_TIMEOUT_SECS_SCHEMA = Number.class;
+ public static final Object SUPERVISOR_WORKER_START_TIMEOUT_SECS_SCHEMA = ConfigValidation.IntegerValidator;
-
/**
* Whether or not the supervisor should launch workers assigned to it. Defaults
* to true -- and you should probably never change this value. This configuration
@@@ -742,25 -464,11 +740,25 @@@
* need to be restarted.
*/
public static final String SUPERVISOR_MONITOR_FREQUENCY_SECS = "supervisor.monitor.frequency.secs";
- public static final Object SUPERVISOR_MONITOR_FREQUENCY_SECS_SCHEMA = Number.class;
+ public static final Object SUPERVISOR_MONITOR_FREQUENCY_SECS_SCHEMA = ConfigValidation.IntegerValidator;
/**
+ * Should the supervior try to run the worker as the lauching user or not. Defaults to false.
+ */
+ public static final String SUPERVISOR_RUN_WORKER_AS_USER = "supervisor.run.worker.as.user";
+ public static final Object SUPERVISOR_RUN_WORKER_AS_USER_SCHEMA = Boolean.class;
+
+ /**
+ * Full path to the worker-laucher executable that will be used to lauch workers when
+ * SUPERVISOR_RUN_WORKER_AS_USER is set to true.
+ */
+ public static final String SUPERVISOR_WORKER_LAUNCHER = "supervisor.worker.launcher";
+ public static final Object SUPERVISOR_WORKER_LAUNCHER_SCHEMA = String.class;
+
+ /**
* The jvm opts provided to workers launched by this supervisor. All "%ID%" substrings are replaced
- * with an identifier for this worker.
+ * with an identifier for this worker. Also, "%WORKER-ID%", "%STORM-ID%" and "%WORKER-PORT%" are
+ * replaced with appropriate runtime values for this worker.
*/
public static final String WORKER_CHILDOPTS = "worker.childopts";
public static final Object WORKER_CHILDOPTS_SCHEMA = ConfigValidation.StringOrStringListValidator;
@@@ -800,22 -500,9 +798,22 @@@
* come through.
*/
public static final String TASK_REFRESH_POLL_SECS = "task.refresh.poll.secs";
- public static final Object TASK_REFRESH_POLL_SECS_SCHEMA = Number.class;
+ public static final Object TASK_REFRESH_POLL_SECS_SCHEMA = ConfigValidation.IntegerValidator;
+ /**
+ * How often a task should sync credentials, worst case.
+ */
+ public static final String TASK_CREDENTIALS_POLL_SECS = "task.credentials.poll.secs";
+ public static final Object TASK_CREDENTIALS_POLL_SECS_SCHEMA = Number.class;
+
+
+ /**
+ * A list of users that are allowed to interact with the topology. To use this set
+ * nimbus.authorizer to backtype.storm.security.auth.authorizer.SimpleACLAuthorizer
+ */
+ public static final String TOPOLOGY_USERS = "topology.users";
+ public static final Object TOPOLOGY_USERS_SCHEMA = ConfigValidation.StringsValidator;
/**
* True if Storm should timeout messages or not. Defaults to true. This is meant to be used
@@@ -936,15 -623,15 +934,15 @@@
/**
- * The maximum number of tuples that can be pending on a spout task at any given time.
- * This config applies to individual tasks, not to spouts or topologies as a whole.
- *
+ * The maximum number of tuples that can be pending on a spout task at any given time.
+ * This config applies to individual tasks, not to spouts or topologies as a whole.
+ *
* A pending tuple is one that has been emitted from a spout but has not been acked or failed yet.
- * Note that this config parameter has no effect for unreliable spouts that don't tag
+ * Note that this config parameter has no effect for unreliable spouts that don't tag
* their tuples with a message id.
*/
- public static final String TOPOLOGY_MAX_SPOUT_PENDING="topology.max.spout.pending";
- public static final Object TOPOLOGY_MAX_SPOUT_PENDING_SCHEMA = Number.class;
+ public static final String TOPOLOGY_MAX_SPOUT_PENDING="topology.max.spout.pending";
+ public static final Object TOPOLOGY_MAX_SPOUT_PENDING_SCHEMA = ConfigValidation.IntegerValidator;
/**
* A class that implements a strategy for what to do when a spout needs to wait. Waiting is
@@@ -973,13 -660,13 +971,13 @@@
* The percentage of tuples to sample to produce stats for a task.
*/
public static final String TOPOLOGY_STATS_SAMPLE_RATE="topology.stats.sample.rate";
- public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = Number.class;
+ public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator;
/**
- * The time period that builtin metrics data in bucketed into.
+ * The time period that builtin metrics data in bucketed into.
*/
public static final String TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS="topology.builtin.metrics.bucket.size.secs";
- public static final Object TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS_SCHEMA = Number.class;
+ public static final Object TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS_SCHEMA = ConfigValidation.IntegerValidator;
/**
* Whether or not to use Java serialization in a topology.
@@@ -994,12 -681,19 +992,25 @@@
public static final Object TOPOLOGY_WORKER_CHILDOPTS_SCHEMA = ConfigValidation.StringOrStringListValidator;
/**
+ * Topology-specific options GC for the worker child process. This overrides WORKER_GC_CHILDOPTS.
+ */
+ public static final String TOPOLOGY_WORKER_GC_CHILDOPTS="topology.worker.gc.childopts";
+ public static final Object TOPOLOGY_WORKER_GC_CHILDOPTS_SCHEMA = ConfigValidation.StringOrStringListValidator;
+
+ /**
+ * Topology-specific classpath for the worker child process. This is combined to the usual classpath.
+ */
+ public static final String TOPOLOGY_CLASSPATH="topology.classpath";
+ public static final Object TOPOLOGY_CLASSPATH_SCHEMA = ConfigValidation.StringOrStringListValidator;
+
+ /**
+ * Topology-specific environment variables for the worker child process.
+ * This is added to the existing environment (that of the supervisor)
+ */
+ public static final String TOPOLOGY_ENVIRONMENT="topology.environment";
+ public static final Object TOPOLOGY_ENVIRONMENT_SCHEMA = Map.class;
+
+ /**
* This config is available for TransactionalSpouts, and contains the id ( a String) for
* the transactional topology. This id is used to store the state of the transactional
* topology in Zookeeper.
@@@ -1061,10 -755,10 +1072,10 @@@
* via the TopologyContext.
*/
public static final String TOPOLOGY_WORKER_SHARED_THREAD_POOL_SIZE="topology.worker.shared.thread.pool.size";
- public static final Object TOPOLOGY_WORKER_SHARED_THREAD_POOL_SIZE_SCHEMA = Number.class;
+ public static final Object TOPOLOGY_WORKER_SHARED_THREAD_POOL_SIZE_SCHEMA = ConfigValidation.IntegerValidator;
/**
- * The interval in seconds to use for determining whether to throttle error reported to Zookeeper. For example,
+ * The interval in seconds to use for determining whether to throttle error reported to Zookeeper. For example,
* an interval of 10 seconds with topology.max.error.report.per.interval set to 5 will only allow 5 errors to be
* reported to Zookeeper per task for every 10 second interval of time.
*/
@@@ -1183,25 -853,27 +1194,41 @@@
* to backtype.storm.scheduler.IsolationScheduler to make use of the isolation scheduler.
*/
public static final String ISOLATION_SCHEDULER_MACHINES = "isolation.scheduler.machines";
- public static final Object ISOLATION_SCHEDULER_MACHINES_SCHEMA = Map.class;
+ public static final Object ISOLATION_SCHEDULER_MACHINES_SCHEMA = ConfigValidation.MapOfStringToNumberValidator;
+
+ /**
+ * A map from the user name to the number of machines that should that user is allowed to use. Set storm.scheduler
+ * to backtype.storm.scheduler.multitenant.MultitenantScheduler
+ */
+ public static final String MULTITENANT_SCHEDULER_USER_POOLS = "multitenant.scheduler.user.pools";
+ public static final Object MULTITENANT_SCHEDULER_USER_POOLS_SCHEMA = ConfigValidation.MapOfStringToNumberValidator;
+
+ /**
+ * The number of machines that should be used by this topology to isolate it from all others. Set storm.scheduler
+ * to backtype.storm.scheduler.multitenant.MultitenantScheduler
+ */
+ public static final String TOPOLOGY_ISOLATED_MACHINES = "topology.isolate.machines";
+ public static final Object TOPOLOGY_ISOLATED_MACHINES_SCHEMA = Number.class;
+ public static void setClasspath(Map conf, String cp) {
+ conf.put(Config.TOPOLOGY_CLASSPATH, cp);
+ }
+
+ public void setClasspath(String cp) {
+ setClasspath(this, cp);
+ }
+
+ public static void setEnvironment(Map conf, Map env) {
+ conf.put(Config.TOPOLOGY_ENVIRONMENT, env);
+ }
+
+ public void setEnvironment(Map env) {
+ setEnvironment(this, env);
+ }
+
public static void setDebug(Map conf, boolean isOn) {
conf.put(Config.TOPOLOGY_DEBUG, isOn);
- }
+ }
public void setDebug(boolean isOn) {
setDebug(this, isOn);
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/jvm/backtype/storm/ConfigValidation.java
----------------------------------------------------------------------
diff --cc storm-core/src/jvm/backtype/storm/ConfigValidation.java
index e990921,156ccf8..14394a0
--- a/storm-core/src/jvm/backtype/storm/ConfigValidation.java
+++ b/storm-core/src/jvm/backtype/storm/ConfigValidation.java
@@@ -177,9 -79,79 +177,79 @@@ public class ConfigValidation
/**
* Validates is a list of Maps.
*/
- public static Object MapsValidator = FieldListValidatorFactory(Map.class);
+ public static Object MapsValidator = listFv(Map.class, true);
/**
+ * Validates a Integer.
+ */
+ public static Object IntegerValidator = new FieldValidator() {
+ @Override
+ public void validateField(String name, Object o) throws IllegalArgumentException {
+ if (o == null) {
+ // A null value is acceptable.
+ return;
+ }
+ final long i;
+ if (o instanceof Number &&
+ (i = ((Number)o).longValue()) == ((Number)o).doubleValue()) {
+ if (i <= Integer.MAX_VALUE && i >= Integer.MIN_VALUE) {
+ return;
+ }
+ }
+
+ throw new IllegalArgumentException("Field " + name + " must be an Integer within type range.");
+ }
+ };
+
+ /**
+ * Validates is a list of Integers.
+ */
+ public static Object IntegersValidator = new FieldValidator() {
+ @Override
+ public void validateField(String name, Object field)
+ throws IllegalArgumentException {
+ if (field == null) {
+ // A null value is acceptable.
+ return;
+ }
+ if (field instanceof Iterable) {
+ for (Object o : (Iterable)field) {
+ final long i;
+ if (o instanceof Number &&
+ ((i = ((Number)o).longValue()) == ((Number)o).doubleValue()) &&
+ (i <= Integer.MAX_VALUE && i >= Integer.MIN_VALUE)) {
+ // pass the test
+ } else {
+ throw new IllegalArgumentException(
+ "Each element of the list " + name + " must be an Integer within type range.");
+ }
+ }
+ return;
+ }
+ }
+ };
+
+ /**
+ * Validates a Double.
+ */
+ public static Object DoubleValidator = new FieldValidator() {
+ @Override
+ public void validateField(String name, Object o) throws IllegalArgumentException {
+ if (o == null) {
+ // A null value is acceptable.
+ return;
+ }
+
+ // we can provide a lenient way to convert int/long to double with losing some precision
+ if (o instanceof Number) {
+ return;
+ }
+
+ throw new IllegalArgumentException("Field " + name + " must be an Double.");
+ }
+ };
+
+ /**
* Validates a power of 2.
*/
public static Object PowerOf2Validator = new FieldValidator() {
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/jvm/backtype/storm/utils/Utils.java
----------------------------------------------------------------------
diff --cc storm-core/src/jvm/backtype/storm/utils/Utils.java
index 87f1654,364b53f..fff91e6
--- a/storm-core/src/jvm/backtype/storm/utils/Utils.java
+++ b/storm-core/src/jvm/backtype/storm/utils/Utils.java
@@@ -323,18 -314,19 +323,21 @@@ public class Utils
if (null == o) {
return defaultValue;
}
-
- if(o instanceof Long) {
- return ((Long) o ).intValue();
- } else if (o instanceof Integer) {
- return (Integer) o;
- } else if (o instanceof Short) {
- return ((Short) o).intValue();
+
+ if (o instanceof Integer ||
+ o instanceof Short ||
+ o instanceof Byte) {
+ return ((Number) o).intValue();
+ } else if (o instanceof Long) {
+ final long l = (Long) o;
+ if (l <= Integer.MAX_VALUE && l >= Integer.MIN_VALUE) {
+ return (int) l;
+ }
+ } else if (o instanceof String) {
+ return Integer.parseInt((String) o);
- } else {
- throw new IllegalArgumentException("Don't know how to convert " + o + " + to int");
}
+
+ throw new IllegalArgumentException("Don't know how to convert " + o + " to int");
}
public static boolean getBoolean(Object o, boolean defaultValue) {
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/ui/public/component.html
----------------------------------------------------------------------
diff --cc storm-core/src/ui/public/component.html
index 092eb40,6353142..803fffb
--- a/storm-core/src/ui/public/component.html
+++ b/storm-core/src/ui/public/component.html
@@@ -26,10 -26,9 +26,11 @@@
<script src="/js/purl.js" type="text/javascript"></script>
<script src="/js/bootstrap-twipsy.js" type="text/javascript"></script>
<script src="/js/script.js" type="text/javascript"></script>
+ <script src="/js/moment.min.js" type="text/javascript"></script>
</head>
<body>
+<div id="ui-user">
+</div>
<h1><a href="/">Storm UI</a></h1>
<div id="component-summary">
</div>
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/src/ui/public/js/script.js
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/test/clj/backtype/storm/cluster_test.clj
----------------------------------------------------------------------
diff --cc storm-core/test/clj/backtype/storm/cluster_test.clj
index f30c6a8,63efd30..7ed1028
--- a/storm-core/test/clj/backtype/storm/cluster_test.clj
+++ b/storm-core/test/clj/backtype/storm/cluster_test.clj
@@@ -220,10 -206,11 +220,11 @@@
(with-inprocess-zookeeper zk-port
(with-simulated-time
(let [state (mk-storm-state zk-port)]
- (.report-error state "a" "1"(local-hostname) 6700 (RuntimeException.))
+ (.report-error state "a" "1" (local-hostname) 6700 (RuntimeException.))
(validate-errors! state "a" "1" ["RuntimeException"])
+ (advance-time-secs! 1)
(.report-error state "a" "1" (local-hostname) 6700 (IllegalArgumentException.))
- (validate-errors! state "a" "1" ["RuntimeException" "IllegalArgumentException"])
+ (validate-errors! state "a" "1" ["IllegalArgumentException" "RuntimeException"])
(doseq [i (range 10)]
(.report-error state "a" "2" (local-hostname) 6700 (RuntimeException.))
(advance-time-secs! 2))
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/test/clj/backtype/storm/config_test.clj
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/559c883d/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --cc storm-core/test/clj/backtype/storm/supervisor_test.clj
index 3dcd275,584f0d9..6b40060
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@@ -311,100 -303,39 +313,137 @@@
mock-worker-id)
(verify-first-call-args-for-indices launch-process
[0]
- exp-args)))))))
+ exp-args))))
+ (testing "testing topology.classpath is added to classpath"
+ (let [topo-cp "/any/path"
+ exp-args (exp-args-fn [] [] (add-to-classpath mock-cp [topo-cp]))
+ mock-supervisor {:conf {STORM-CLUSTER-MODE :distributed}}]
+ (stubbing [read-supervisor-storm-conf {TOPOLOGY-CLASSPATH topo-cp}
+ supervisor-stormdist-root nil
+ supervisor/jlp nil
++ set-worker-user! nil
++ supervisor/write-log-metadata! nil
+ launch-process nil
+ current-classpath "/base"]
+ (supervisor/launch-worker mock-supervisor
+ mock-storm-id
+ mock-port
+ mock-worker-id)
+ (verify-first-call-args-for-indices launch-process
+ [0]
+ exp-args))))
+ (testing "testing topology.environment is added to environment for worker launch"
+ (let [topo-env {"THISVAR" "somevalue" "THATVAR" "someothervalue"}
++ full-env (merge topo-env {"LD_LIBRARY_PATH" nil})
+ exp-args (exp-args-fn [] [] mock-cp)
+ mock-supervisor {:conf {STORM-CLUSTER-MODE :distributed}}]
+ (stubbing [read-supervisor-storm-conf {TOPOLOGY-ENVIRONMENT topo-env}
+ supervisor-stormdist-root nil
+ supervisor/jlp nil
+ launch-process nil
++ set-worker-user! nil
++ supervisor/write-log-metadata! nil
+ current-classpath "/base"]
+ (supervisor/launch-worker mock-supervisor
+ mock-storm-id
+ mock-port
+ mock-worker-id)
+ (verify-first-call-args-for-indices launch-process
+ [2]
- (merge topo-env {"LD_LIBRARY_PATH" nil}))))))))
++ full-env)))))))
+
+(defn rm-r [f]
+ (if (.isDirectory f)
+ (for [sub (.listFiles f)] (rm-r sub))
+ (.delete f)
+ ))
+
+(deftest test-worker-launch-command-run-as-user
+ (testing "*.worker.childopts configuration"
+ (let [mock-port "42"
+ mock-storm-id "fake-storm-id"
+ mock-worker-id "fake-worker-id"
+ mock-cp "mock-classpath'quote-on-purpose"
+ storm-local (str "/tmp/" (UUID/randomUUID))
+ worker-script (str storm-local "/workers/" mock-worker-id "/storm-worker-script.sh")
+ exp-launch ["/bin/worker-launcher"
+ "me"
+ "worker"
+ (str storm-local "/workers/" mock-worker-id)
+ worker-script]
+ exp-script-fn (fn [opts topo-opts]
+ (str "#!/bin/bash\n'export' 'LD_LIBRARY_PATH=';\n\nexec 'java' '-server'"
+ " " (shell-cmd opts)
+ " " (shell-cmd topo-opts)
+ " '-Djava.library.path='"
+ " '-Dlogfile.name=" mock-storm-id "-worker-" mock-port ".log'"
+ " '-Dstorm.home='"
+ " '-Dlogback.configurationFile=/logback/cluster.xml'"
+ " '-Dstorm.id=" mock-storm-id "'"
+ " '-Dworker.id=" mock-worker-id "'"
+ " '-Dworker.port=" mock-port "'"
+ " '-cp' 'mock-classpath'\"'\"'quote-on-purpose'"
+ " 'backtype.storm.daemon.worker'"
+ " '" mock-storm-id "'"
+ " '" mock-port "'"
+ " '" mock-worker-id "';"))]
+ (.mkdirs (io/file storm-local "workers" mock-worker-id))
+ (try
+ (testing "testing *.worker.childopts as strings with extra spaces"
+ (let [string-opts "-Dfoo=bar -Xmx1024m"
+ topo-string-opts "-Dkau=aux -Xmx2048m"
+ exp-script (exp-script-fn ["-Dfoo=bar" "-Xmx1024m"]
+ ["-Dkau=aux" "-Xmx2048m"])
+ mock-supervisor {:conf {STORM-CLUSTER-MODE :distributed
+ STORM-LOCAL-DIR storm-local
+ SUPERVISOR-RUN-WORKER-AS-USER true
+ WORKER-CHILDOPTS string-opts}}]
+ (stubbing [read-supervisor-storm-conf {TOPOLOGY-WORKER-CHILDOPTS
+ topo-string-opts
+ TOPOLOGY-SUBMITTER-USER "me"}
+ add-to-classpath mock-cp
+ supervisor-stormdist-root nil
+ launch-process nil
+ set-worker-user! nil
+ supervisor/java-cmd "java"
+ supervisor/jlp nil
+ supervisor/write-log-metadata! nil]
+ (supervisor/launch-worker mock-supervisor
+ mock-storm-id
+ mock-port
+ mock-worker-id)
+ (verify-first-call-args-for-indices launch-process
+ [0]
+ exp-launch))
+ (is (= (slurp worker-script) exp-script))))
+ (testing "testing *.worker.childopts as list of strings, with spaces in values"
+ (let [list-opts '("-Dopt1='this has a space in it'" "-Xmx1024m")
+ topo-list-opts '("-Dopt2='val with spaces'" "-Xmx2048m")
+ exp-script (exp-script-fn list-opts topo-list-opts)
+ mock-supervisor {:conf {STORM-CLUSTER-MODE :distributed
+ STORM-LOCAL-DIR storm-local
+ SUPERVISOR-RUN-WORKER-AS-USER true
+ WORKER-CHILDOPTS list-opts}}]
+ (stubbing [read-supervisor-storm-conf {TOPOLOGY-WORKER-CHILDOPTS
+ topo-list-opts
+ TOPOLOGY-SUBMITTER-USER "me"}
+ add-to-classpath mock-cp
+ supervisor-stormdist-root nil
+ launch-process nil
+ set-worker-user! nil
+ supervisor/java-cmd "java"
+ supervisor/jlp nil
+ supervisor/write-log-metadata! nil]
+ (supervisor/launch-worker mock-supervisor
+ mock-storm-id
+ mock-port
+ mock-worker-id)
+ (verify-first-call-args-for-indices launch-process
+ [0]
+ exp-launch))
+ (is (= (slurp worker-script) exp-script))))
+(finally (rm-r (io/file storm-local)))
+))))
(deftest test-workers-go-bananas
;; test that multiple workers are started for a port, and test that
@@@ -422,142 -353,59 +461,198 @@@
;; TODO just do reassign, and check that cleans up worker states after killing but doesn't get rid of downloaded code
)
+(deftest test-supervisor-data-acls
+ (testing "supervisor-data uses correct ACLs"
+ (let [scheme "digest"
+ digest "storm:thisisapoorpassword"
+ auth-conf {STORM-ZOOKEEPER-AUTH-SCHEME scheme
+ STORM-ZOOKEEPER-AUTH-PAYLOAD digest}
+ expected-acls supervisor/SUPERVISOR-ZK-ACLS
+ fake-isupervisor (reify ISupervisor
+ (getSupervisorId [this] nil)
+ (getAssignmentId [this] nil))]
+ (stubbing [uptime-computer nil
+ cluster/mk-storm-cluster-state nil
+ supervisor-state nil
+ local-hostname nil
+ mk-timer nil]
+ (supervisor/supervisor-data auth-conf nil fake-isupervisor)
+ (verify-call-times-for cluster/mk-storm-cluster-state 1)
+ (verify-first-call-args-for-indices cluster/mk-storm-cluster-state [2]
+ expected-acls)))))
+
+(deftest test-write-log-metadata
+ (testing "supervisor writes correct data to logs metadata file"
+ (let [exp-owner "alice"
+ exp-worker-id "42"
+ exp-storm-id "0123456789"
+ exp-port 4242
+ exp-logs-users ["bob" "charlie" "daryl"]
+ storm-conf {TOPOLOGY-SUBMITTER-USER "alice"
+ TOPOLOGY-USERS ["charlie" "bob"]
+ LOGS-USERS ["daryl"]}
+ exp-data {TOPOLOGY-SUBMITTER-USER exp-owner
+ "worker-id" exp-worker-id
+ LOGS-USERS exp-logs-users}
+ conf {}]
+ (mocking [supervisor/write-log-metadata-to-yaml-file!]
+ (supervisor/write-log-metadata! storm-conf exp-owner exp-worker-id
+ exp-storm-id exp-port conf)
+ (verify-called-once-with-args supervisor/write-log-metadata-to-yaml-file!
+ exp-storm-id exp-port exp-data conf)))))
+
+(deftest test-worker-launcher-requires-user
+ (testing "worker-launcher throws on blank user"
+ (mocking [launch-process]
+ (is (thrown-cause-with-msg? java.lang.IllegalArgumentException
+ #"(?i).*user cannot be blank.*"
+ (supervisor/worker-launcher {} nil ""))))))
+
+(defn found? [sub-str input-str]
+ (if (string? input-str)
+ (contrib-str/substring? sub-str (str input-str))
+ (some? #(contrib-str/substring? sub-str %) input-str)))
+
+(defn not-found? [sub-str input-str]
+ (complement (found? sub-str input-str)))
+
+(deftest test-substitute-childopts-happy-path
+ (testing "worker-launcher replaces ids in childopts"
+ (let [ worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log"
+ ]
+ (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
+ (is (not-found? "%WORKER-ID%" childopts-with-ids))
+ (is (found? "w-01" childopts-with-ids))
+ (is (not-found? "%STORM-ID%" childopts-with-ids))
+ (is (found? "s-01" childopts-with-ids))
+ (is (not-found? "%WORKER-PORT%" childopts-with-ids))
+ (is (found? "-9999." childopts-with-ids))
+ (is (not-found? "%ID%" childopts-with-ids))
+ (is (found? "worker-9999" childopts-with-ids) (str childopts-with-ids))
+ )))
+
+(deftest test-substitute-childopts-storm-id-alone
+ (testing "worker-launcher replaces ids in childopts"
+ (let [ worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%STORM-ID%.log"]
+ (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
+ (is (not-found? "%WORKER-ID%" childopts-with-ids))
+ (is (not-found? "w-01" childopts-with-ids))
+ (is (not-found? "%STORM-ID%" childopts-with-ids))
+ (is (found? "s-01" childopts-with-ids))
+ (is (not-found? "%WORKER-PORT%" childopts-with-ids))
+ (is (not-found? "-9999." childopts-with-ids))
+ (is (not-found? "%ID%" childopts-with-ids))
+ (is (not-found? "worker-9999" childopts-with-ids) (str childopts-with-ids)) )))
+
+(deftest test-substitute-childopts-no-keys
+ (testing "worker-launcher has no ids to replace in childopts"
+ (let [ worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker.log"]
+ (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
+ (is (not-found? "%WORKER-ID%" childopts-with-ids))
+ (is (not-found? "w-01" childopts-with-ids))
+ (is (not-found? "%STORM-ID%" childopts-with-ids))
+ (is (not-found? "s-01" childopts-with-ids))
+ (is (not-found? "%WORKER-PORT%" childopts-with-ids))
+ (is (not-found? "-9999." childopts-with-ids))
+ (is (not-found? "%ID%" childopts-with-ids))
+ (is (not-found? "worker-9999" childopts-with-ids) (str childopts-with-ids)) )))
+
+(deftest test-substitute-childopts-nil-childopts
+ (testing "worker-launcher has nil childopts"
+ (let [ worker-id "w-01"
+ storm-id "s-01"
+ port 9999
+ childopts nil]
+ (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
+ (is (not-found? "%WORKER-ID%" childopts-with-ids))
+ (is (not-found? "w-01" childopts-with-ids))
+ (is (not-found? "%STORM-ID%" childopts-with-ids))
+ (is (not-found? "s-01" childopts-with-ids))
+ (is (not-found? "%WORKER-PORT%" childopts-with-ids))
+ (is (not-found? "-9999." childopts-with-ids))
+ (is (not-found? "%ID%" childopts-with-ids))
+ (is (not-found? "worker-9999" childopts-with-ids) (str childopts-with-ids))
+ )))
+
+(deftest test-substitute-childopts-nil-ids
+ (testing "worker-launcher has nil ids"
+ (let [ worker-id nil
+ storm-id "s-01"
+ port 9999
+ childopts "-Xloggc:/home/y/lib/storm/current/logs/gc.worker-%ID%-%STORM-ID%-%WORKER-ID%-%WORKER-PORT%.log"]
+ (def childopts-with-ids (supervisor/substitute-childopts childopts worker-id storm-id port))
+ (is (not-found? "%WORKER-ID%" childopts-with-ids))
+ (is (not-found? "w-01" childopts-with-ids))
+ (is (not-found? "%STORM-ID%" childopts-with-ids))
+ (is (found? "s-01" childopts-with-ids))
+ (is (not-found? "%WORKER-PORT%" childopts-with-ids))
+ (is (found? "-9999." childopts-with-ids))
+ (is (not-found? "%ID%" childopts-with-ids))
+ (is (found? "worker-9999" childopts-with-ids) (str childopts-with-ids))
+ )))
+
+ (deftest test-retry-read-assignments
+ (with-simulated-time-local-cluster [cluster
+ :supervisors 0
+ :ports-per-supervisor 2
+ :daemon-conf {NIMBUS-REASSIGN false
+ NIMBUS-MONITOR-FREQ-SECS 10
+ TOPOLOGY-MESSAGE-TIMEOUT-SECS 30
+ TOPOLOGY-ACKER-EXECUTORS 0}]
+ (letlocals
+ (bind sup1 (add-supervisor cluster :id "sup1" :ports [1 2 3 4]))
+ (bind topology1 (thrift/mk-topology
+ {"1" (thrift/mk-spout-spec (TestPlannerSpout. true) :parallelism-hint 2)}
+ {}))
+ (bind topology2 (thrift/mk-topology
+ {"1" (thrift/mk-spout-spec (TestPlannerSpout. true) :parallelism-hint 2)}
+ {}))
+ (bind state (:storm-cluster-state cluster))
+ (bind changed (capture-changed-workers
+ (submit-mocked-assignment
+ (:nimbus cluster)
+ "topology1"
+ {TOPOLOGY-WORKERS 2}
+ topology1
+ {1 "1"
+ 2 "1"}
+ {[1] ["sup1" 1]
+ [2] ["sup1" 2]
+ })
+ (submit-mocked-assignment
+ (:nimbus cluster)
+ "topology2"
+ {TOPOLOGY-WORKERS 2}
+ topology2
+ {1 "1"
+ 2 "1"}
+ {[1] ["sup1" 1]
+ [2] ["sup1" 2]
+ })
+ (advance-cluster-time cluster 10)
+ ))
+ (is (empty? (:launched changed)))
+ (bind options (RebalanceOptions.))
+ (.set_wait_secs options 0)
+ (bind changed (capture-changed-workers
+ (.rebalance (:nimbus cluster) "topology2" options)
+ (advance-cluster-time cluster 10)
+ (heartbeat-workers cluster "sup1" [1 2 3 4])
+ (advance-cluster-time cluster 10)
+ ))
+ (validate-launched-once (:launched changed)
+ {"sup1" [1 2]}
+ (get-storm-id (:storm-cluster-state cluster) "topology1"))
+ (validate-launched-once (:launched changed)
+ {"sup1" [3 4]}
+ (get-storm-id (:storm-cluster-state cluster) "topology2"))
+ )))
[28/50] git commit: Adding metadave to contributors list
Posted by bo...@apache.org.
Adding metadave to contributors list
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/2e7b1d45
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/2e7b1d45
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/2e7b1d45
Branch: refs/heads/security
Commit: 2e7b1d454db004c51b846d4668ed7826843a4251
Parents: 097a072
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Thu Jul 24 16:08:27 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Thu Jul 24 16:08:27 2014 -0500
----------------------------------------------------------------------
README.markdown | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/2e7b1d45/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 341b62c..64b8825 100644
--- a/README.markdown
+++ b/README.markdown
@@ -155,6 +155,7 @@ under the License.
* Masatake Iwasaki ([@iwasakims](https://github.com/iwasakims))
* Milad Fatenejad ([@icksa](https://github.com/icksa))
* Ryan Liu ([@shell0dh](https://github.com/shell0dh))
+* Dave Parfitt ([@metadave](https://github.com/metadave))
## Acknowledgements
[09/50] git commit: Merge branch 'master' of
https://github.com/aaronlevin/incubator-storm into STORM-354
Posted by bo...@apache.org.
Merge branch 'master' of https://github.com/aaronlevin/incubator-storm into STORM-354
Conflicts:
storm-core/src/clj/backtype/storm/testing.clj
STORM-354: Testing: allow users to pass TEST-TIMEOUT-MS as param for complete-topology
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/44aaaa91
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/44aaaa91
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/44aaaa91
Branch: refs/heads/security
Commit: 44aaaa91be7cfa5fef8424f08a25bd1e8b3bf458
Parents: 69b4601 c86dbf9
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 18 08:25:44 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 18 08:25:44 2014 -0500
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/testing.clj | 48 ++++++++++++----------
1 file changed, 26 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/44aaaa91/storm-core/src/clj/backtype/storm/testing.clj
----------------------------------------------------------------------
diff --cc storm-core/src/clj/backtype/storm/testing.clj
index 70c783a,fd97255..54f40e0
--- a/storm-core/src/clj/backtype/storm/testing.clj
+++ b/storm-core/src/clj/backtype/storm/testing.clj
@@@ -205,14 -203,14 +206,14 @@@
daemons (concat
[(:nimbus cluster-map)]
supervisors
- workers) ; because a worker may already be dead
- ]
+ ; because a worker may already be dead
+ workers)]
- (while-timeout TEST-TIMEOUT-MS (not (every? (memfn waiting?) daemons))
+ (while-timeout timeout-ms (not (every? (memfn waiting?) daemons))
- (Thread/sleep 10)
-;; (doseq [d daemons]
-;; (if-not ((memfn waiting?) d)
-;; (println d)))
- ))))
+ (Thread/sleep 10)
+ ;; (doseq [d daemons]
+ ;; (if-not ((memfn waiting?) d)
+ ;; (println d)))
- )))
++ ))))
(defn advance-cluster-time
([cluster-map secs increment-secs]
@@@ -437,24 -425,21 +438,25 @@@
(.set_bolts topology
(assoc (clojurify-structure bolts)
(uuid)
- (Bolt.
- (serialize-component-object capturer)
- (mk-plain-component-common (into {} (for [[id direct?] all-streams]
- [id (if direct?
- (mk-direct-grouping)
- (mk-global-grouping))]))
- {}
- nil))
- ))
+ (Bolt.
+ (serialize-component-object capturer)
+ (mk-plain-component-common (into {} (for [[id direct?] all-streams]
+ [id (if direct?
+ (mk-direct-grouping)
+ (mk-global-grouping))]))
+ {}
+ nil))))
{:topology topology
- :capturer capturer}
- ))
+ :capturer capturer}))
;; TODO: mock-sources needs to be able to mock out state spouts as well
-(defnk complete-topology [cluster-map topology :mock-sources {} :storm-conf {} :cleanup-state true :topology-name nil :timeout-ms TEST-TIMEOUT-MS]
+(defnk complete-topology
+ [cluster-map topology
+ :mock-sources {}
+ :storm-conf {}
+ :cleanup-state true
- :topology-name nil]
++ :topology-name nil
++ :timeout-ms TEST-TIMEOUT-MS]
;; TODO: the idea of mocking for transactional topologies should be done an
;; abstraction level above... should have a complete-transactional-topology for this
(let [{topology :topology capturer :capturer} (capture-topology topology)
@@@ -477,16 -467,17 +479,16 @@@
(doseq [spout (spout-objects spouts)]
(startup spout))
-
+
(submit-local-topology (:nimbus cluster-map) storm-name storm-conf topology)
-
-
+
(let [storm-id (common/get-storm-id state storm-name)]
- (while-timeout TEST-TIMEOUT-MS (not (every? exhausted? (spout-objects spouts)))
+ (while-timeout timeout-ms (not (every? exhausted? (spout-objects spouts)))
- (simulate-wait cluster-map))
+ (simulate-wait cluster-map))
(.killTopologyWithOpts (:nimbus cluster-map) storm-name (doto (KillOptions.) (.set_wait_secs 0)))
- (while-timeout TEST-TIMEOUT-MS (.assignment-info state storm-id nil)
+ (while-timeout timeout-ms (.assignment-info state storm-id nil)
- (simulate-wait cluster-map))
+ (simulate-wait cluster-map))
(when cleanup-state
(doseq [spout (spout-objects spouts)]
(cleanup spout))))
@@@ -580,27 -574,29 +582,29 @@@
(defn tracked-wait
"Waits until topology is idle and 'amt' more tuples have been emitted by spouts."
([tracked-topology]
- (tracked-wait tracked-topology 1))
+ (tracked-wait tracked-topology 1 TEST-TIMEOUT-MS))
([tracked-topology amt]
- (let [target (+ amt @(:last-spout-emit tracked-topology))
- track-id (-> tracked-topology :cluster ::track-id)
- waiting? (fn []
- (or (not= target (global-amt track-id "spout-emitted"))
- (not= (global-amt track-id "transferred")
- (global-amt track-id "processed"))
- ))]
- (while-timeout TEST-TIMEOUT-MS (waiting?)
- ;; (println "Spout emitted: " (global-amt track-id "spout-emitted"))
- ;; (println "Processed: " (global-amt track-id "processed"))
- ;; (println "Transferred: " (global-amt track-id "transferred"))
- (Thread/sleep 500))
- (reset! (:last-spout-emit tracked-topology) target))))
-
- (defnk test-tuple
+ (tracked-wait tracked-topology amt TEST-TIMEOUT-MS))
+ ([tracked-topology amt timeout-ms]
+ (let [target (+ amt @(:last-spout-emit tracked-topology))
+ track-id (-> tracked-topology :cluster ::track-id)
+ waiting? (fn []
+ (or (not= target (global-amt track-id "spout-emitted"))
+ (not= (global-amt track-id "transferred")
+ (global-amt track-id "processed"))
+ ))]
+ (while-timeout TEST-TIMEOUT-MS (waiting?)
- ;; (println "Spout emitted: " (global-amt track-id "spout-emitted"))
- ;; (println "Processed: " (global-amt track-id "processed"))
- ;; (println "Transferred: " (global-amt track-id "transferred"))
- (Thread/sleep 500))
- (reset! (:last-spout-emit tracked-topology) target)
- )))
-
-(defnk test-tuple [values
- :stream Utils/DEFAULT_STREAM_ID
- :component "component"
- :fields nil]
++ ;; (println "Spout emitted: " (global-amt track-id "spout-emitted"))
++ ;; (println "Processed: " (global-amt track-id "processed"))
++ ;; (println "Transferred: " (global-amt track-id "transferred"))
++ (Thread/sleep 500))
++ (reset! (:last-spout-emit tracked-topology) target))))
++
++(defnk test-tuple
+ [values
+ :stream Utils/DEFAULT_STREAM_ID
+ :component "component"
+ :fields nil]
(let [fields (or fields
(->> (iterate inc 1)
(take (count values))
[46/50] git commit: Merge branch 'STORM-419' of
https://github.com/revans2/incubator-storm into STORM-419
Posted by bo...@apache.org.
Merge branch 'STORM-419' of https://github.com/revans2/incubator-storm into STORM-419
STORM-419: Updated test so sort ordering is very explicit.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/9db723d8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/9db723d8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/9db723d8
Branch: refs/heads/security
Commit: 9db723d852f1fb0a053efd3eaf46430bbe40a985
Parents: a6bf4c3 3414a79
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 16:44:07 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 16:44:07 2014 -0500
----------------------------------------------------------------------
----------------------------------------------------------------------
[14/50] git commit: Merge branch 'upstream_master_storm254' of
https://github.com/dashengju/incubator-storm into STORM-254
Posted by bo...@apache.org.
Merge branch 'upstream_master_storm254' of https://github.com/dashengju/incubator-storm into STORM-254
STORM-254: one Spout/Bolt can register metric twice with same name in different timeBucket
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/8fa0f788
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/8fa0f788
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/8fa0f788
Branch: refs/heads/security
Commit: 8fa0f7881d0c63db70315cd08da75421b1a1d972
Parents: 85c9cd5 f8c9995
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Jul 21 11:17:06 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Jul 21 11:17:06 2014 -0500
----------------------------------------------------------------------
storm-core/src/jvm/backtype/storm/task/TopologyContext.java | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
[34/50] git commit: Merge branch
'STORM-399-kafka-spout-increase-default' of
https://github.com/curtisallen/incubator-storm into STORM-399
Posted by bo...@apache.org.
Merge branch 'STORM-399-kafka-spout-increase-default' of https://github.com/curtisallen/incubator-storm into STORM-399
STORM-399: Kafka Spout defaulting to latest offset when current offset
is older then 100k
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/e9238bcd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/e9238bcd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/e9238bcd
Branch: refs/heads/security
Commit: e9238bcd15fc7b7eb19e2bd34a8b2968eaa7edd0
Parents: 3c55b10 2b1d6cf
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Thu Jul 24 16:56:14 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Thu Jul 24 16:56:14 2014 -0500
----------------------------------------------------------------------
external/storm-kafka/src/jvm/storm/kafka/KafkaConfig.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
[37/50] git commit: Updated Changelog for STORM-421
Posted by bo...@apache.org.
Updated Changelog for STORM-421
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/30efc8a5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/30efc8a5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/30efc8a5
Branch: refs/heads/security
Commit: 30efc8a53b974dbfb657ec0e13f1d82556ffbe26
Parents: 6c2f08f
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 13:53:59 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 13:53:59 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/30efc8a5/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 366a401..b722928 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -26,6 +26,7 @@
* STORM-364: The exception time display as default timezone.
* STORM-420: Missing quotes in storm-starter python code
* STORM-399: Kafka Spout defaulting to latest offset when current offset is older then 100k
+ * STORM-421: Memoize local hostname lookup in executor
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[16/50] git commit: Merge branch 'STORM-403' of
https://github.com/iwasakims/incubator-storm into STORM-403
Posted by bo...@apache.org.
Merge branch 'STORM-403' of https://github.com/iwasakims/incubator-storm into STORM-403
STORM-403: heartbeats-to-nimbus in supervisor-test failed due to uninten...
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/3dd0f40e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/3dd0f40e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/3dd0f40e
Branch: refs/heads/security
Commit: 3dd0f40e2e87de98195bed792d03adf1cec4848c
Parents: 48cc069 e6d138e
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Jul 21 11:28:44 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Jul 21 11:28:44 2014 -0500
----------------------------------------------------------------------
.../test/clj/backtype/storm/supervisor_test.clj | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
[13/50] git commit: use util/memoized-local-hostname in executor
Posted by bo...@apache.org.
use util/memoized-local-hostname in executor
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/28de7e51
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/28de7e51
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/28de7e51
Branch: refs/heads/security
Commit: 28de7e511340d6c134f2eac039dc054a0107cba4
Parents: a5c97d7
Author: Homer Strong <ho...@gmail.com>
Authored: Sun Jul 20 12:39:40 2014 -0700
Committer: Homer Strong <ho...@gmail.com>
Committed: Sun Jul 20 12:39:40 2014 -0700
----------------------------------------------------------------------
storm-core/src/clj/backtype/storm/daemon/executor.clj | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/28de7e51/storm-core/src/clj/backtype/storm/daemon/executor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/daemon/executor.clj b/storm-core/src/clj/backtype/storm/daemon/executor.clj
index 890a01e..f60aa80 100644
--- a/storm-core/src/clj/backtype/storm/daemon/executor.clj
+++ b/storm-core/src/clj/backtype/storm/daemon/executor.clj
@@ -176,7 +176,7 @@
(when (<= @interval-errors max-per-interval)
(cluster/report-error (:storm-cluster-state executor) (:storm-id executor) (:component-id executor)
- (local-hostname) (.getThisWorkerPort (:worker-context executor)) error)
+ (memoized-local-hostname) (.getThisWorkerPort (:worker-context executor)) error)
))))
;; in its own function so that it can be mocked out by tracked topologies
@@ -278,7 +278,7 @@
task-id (:task-id task-data)
name->imetric (-> interval->task->metric-registry (get interval) (get task-id))
task-info (IMetricsConsumer$TaskInfo.
- (. (java.net.InetAddress/getLocalHost) getCanonicalHostName)
+ (memoized-local-hostname)
(.getThisWorkerPort worker-context)
(:component-id executor-data)
task-id
[20/50] git commit: Merge branch 'STORM-364' of
https://github.com/shell0dh/incubator-storm into STORM-364
Posted by bo...@apache.org.
Merge branch 'STORM-364' of https://github.com/shell0dh/incubator-storm into STORM-364
STORM-364: The exception time display as default timezone.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/aa0139c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/aa0139c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/aa0139c0
Branch: refs/heads/security
Commit: aa0139c08db53f4071a82200d8afb2fae397aa35
Parents: 5b425e4 9098f9d
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Tue Jul 22 08:41:10 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Tue Jul 22 08:41:10 2014 -0500
----------------------------------------------------------------------
STORM-UI-REST-API.md | 23 +++++++++++++++++++-
storm-core/src/clj/backtype/storm/ui/core.clj | 2 +-
.../src/clj/backtype/storm/ui/helpers.clj | 5 -----
storm-core/src/ui/public/component.html | 3 ++-
storm-core/src/ui/public/js/moment.min.js | 6 +++++
storm-core/src/ui/public/js/script.js | 9 ++++++++
6 files changed, 40 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
[43/50] git commit: Added STORM-415 to Changelog
Posted by bo...@apache.org.
Added STORM-415 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/556ac8af
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/556ac8af
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/556ac8af
Branch: refs/heads/security
Commit: 556ac8af1d0a0ee81473b126887e0f1cdc50080f
Parents: f155e6f
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 16:18:46 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 16:18:46 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/556ac8af/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 43dd049..eafa3d1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -29,6 +29,7 @@
* STORM-421: Memoize local hostname lookup in executor
* STORM-414: support logging level to multilang protocol spout and bolt
* STORM-321: Added a tool to see the current status of STORM JIRA and github pulls.
+ * STORM-415: validate-launched-once in supervisor-test can not handle multiple topologies
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[31/50] git commit: Merge branch 'pull-179'
Posted by bo...@apache.org.
Merge branch 'pull-179'
Conflicts:
README.markdown
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/07a561ae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/07a561ae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/07a561ae
Branch: refs/heads/security
Commit: 07a561aeec294bd92e15d155c5e1249c1bb0d5d5
Parents: 2e7b1d4 9b3cb18
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Thu Jul 24 16:12:44 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Thu Jul 24 16:12:44 2014 -0500
----------------------------------------------------------------------
README.markdown | 1 +
examples/storm-starter/README.markdown | 3 +++
storm-core/test/clj/backtype/storm/cluster_test.clj | 3 ++-
3 files changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/07a561ae/README.markdown
----------------------------------------------------------------------
diff --cc README.markdown
index 64b8825,5e89333..2a6afbf
--- a/README.markdown
+++ b/README.markdown
@@@ -155,7 -155,7 +155,8 @@@ under the License
* Masatake Iwasaki ([@iwasakims](https://github.com/iwasakims))
* Milad Fatenejad ([@icksa](https://github.com/icksa))
* Ryan Liu ([@shell0dh](https://github.com/shell0dh))
+* Dave Parfitt ([@metadave](https://github.com/metadave))
+ * Jo Liss ([@joliss](https://github.com/joliss))
## Acknowledgements
[04/50] git commit: fixed the validation logic in
validate-launched-once.
Posted by bo...@apache.org.
fixed the validation logic in validate-launched-once.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/c78e08fe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/c78e08fe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/c78e08fe
Branch: refs/heads/security
Commit: c78e08feef7044d262c0ea30d202faee94c28862
Parents: 5b425e4
Author: iwasakims <iw...@example.com>
Authored: Wed Jul 16 06:30:35 2014 -0700
Committer: iwasakims <iw...@example.com>
Committed: Wed Jul 16 06:30:35 2014 -0700
----------------------------------------------------------------------
storm-core/test/clj/backtype/storm/supervisor_test.clj | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/c78e08fe/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/supervisor_test.clj b/storm-core/test/clj/backtype/storm/supervisor_test.clj
index 1a27c1a..1bae991 100644
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@ -60,10 +60,11 @@
(defn validate-launched-once [launched supervisor->ports storm-id]
(let [counts (map count (vals launched))
- launched-supervisor->ports (apply merge-with concat
- (for [[s p] (keys launched)]
- {s [p]}
- ))]
+ launched-supervisor->ports (apply merge-with set/union
+ (for [[[s p] sids] launched
+ :when (some #(= % storm-id) sids)]
+ {s #{p}}))
+ supervisor->ports (map-val set supervisor->ports)]
(is (every? (partial = 1) counts))
(is (= launched-supervisor->ports supervisor->ports))
))
[40/50] git commit: Merge branch 'jira-github-join' of
https://github.com/revans2/incubator-storm into STORM-321
Posted by bo...@apache.org.
Merge branch 'jira-github-join' of https://github.com/revans2/incubator-storm into STORM-321
STORM-321: Added a tool to see the current status of STORM JIRA and github pulls.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/2f737761
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/2f737761
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/2f737761
Branch: refs/heads/security
Commit: 2f737761600cd8cbc8d321dd5cfcad4d28e29d05
Parents: dc8907b b9017ea
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 15:11:01 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 15:11:01 2014 -0500
----------------------------------------------------------------------
dev-tools/github/__init__.py | 109 +++++++++++++++++
dev-tools/jira-github-join.py | 80 +++++++++++++
dev-tools/jira/__init__.py | 232 +++++++++++++++++++++++++++++++++++++
3 files changed, 421 insertions(+)
----------------------------------------------------------------------
[03/50] git commit: STORM-403: heartbeats-to-nimbus in
supervisor-test failed due to unintentional worker launch
Posted by bo...@apache.org.
STORM-403: heartbeats-to-nimbus in supervisor-test failed due to unintentional worker launch
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/e6d138e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/e6d138e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/e6d138e1
Branch: refs/heads/security
Commit: e6d138e1935ad77099ea8513a1ea3d6141beec82
Parents: 96e81c1
Author: iwasakims <iw...@example.com>
Authored: Wed Jul 16 06:26:15 2014 -0700
Committer: iwasakims <iw...@example.com>
Committed: Wed Jul 16 06:26:15 2014 -0700
----------------------------------------------------------------------
.../test/clj/backtype/storm/supervisor_test.clj | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/e6d138e1/storm-core/test/clj/backtype/storm/supervisor_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/supervisor_test.clj b/storm-core/test/clj/backtype/storm/supervisor_test.clj
index eb69f11..1a27c1a 100644
--- a/storm-core/test/clj/backtype/storm/supervisor_test.clj
+++ b/storm-core/test/clj/backtype/storm/supervisor_test.clj
@@ -229,16 +229,16 @@
{"1" (thrift/mk-spout-spec (TestPlannerSpout. true) :parallelism-hint 4)}
{}))
;; prevent them from launching by capturing them
- (capture-changed-workers (submit-local-topology (:nimbus cluster) "test" {TOPOLOGY-WORKERS 2} topology))
- (advance-cluster-time cluster 3)
- (check-heartbeat cluster "sup" 3)
- (advance-cluster-time cluster 3)
- (check-heartbeat cluster "sup" 3)
- (advance-cluster-time cluster 3)
- (check-heartbeat cluster "sup" 3)
- (advance-cluster-time cluster 20)
- (check-heartbeat cluster "sup" 3)
-
+ (capture-changed-workers
+ (submit-local-topology (:nimbus cluster) "test" {TOPOLOGY-WORKERS 2} topology)
+ (advance-cluster-time cluster 3)
+ (check-heartbeat cluster "sup" 3)
+ (advance-cluster-time cluster 3)
+ (check-heartbeat cluster "sup" 3)
+ (advance-cluster-time cluster 3)
+ (check-heartbeat cluster "sup" 3)
+ (advance-cluster-time cluster 20)
+ (check-heartbeat cluster "sup" 3))
)))
(deftest test-worker-launch-command
[25/50] git commit: Increase netty max retries defaults allowing more
time for other workers to come up
Posted by bo...@apache.org.
Increase netty max retries defaults allowing more time for other workers to come up
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/9b3a6323
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/9b3a6323
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/9b3a6323
Branch: refs/heads/security
Commit: 9b3a632340b66f1fe493220c0e6c22c2912e8025
Parents: e1e6a60
Author: Kishor Patil <kp...@yahoo-inc.com>
Authored: Thu Jul 24 19:36:41 2014 +0000
Committer: Kishor Patil <kp...@yahoo-inc.com>
Committed: Thu Jul 24 19:36:41 2014 +0000
----------------------------------------------------------------------
conf/defaults.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/9b3a6323/conf/defaults.yaml
----------------------------------------------------------------------
diff --git a/conf/defaults.yaml b/conf/defaults.yaml
index c9c813c..8c07075 100644
--- a/conf/defaults.yaml
+++ b/conf/defaults.yaml
@@ -108,7 +108,7 @@ zmq.hwm: 0
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
-storm.messaging.netty.max_retries: 30
+storm.messaging.netty.max_retries: 120
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100
[07/50] git commit: Noticed that a user name was not being parsed
properly.
Posted by bo...@apache.org.
Noticed that a user name was not being parsed properly.
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/b9017eab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/b9017eab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/b9017eab
Branch: refs/heads/security
Commit: b9017eab4beb4d2997688b0c732e937cedb5af1f
Parents: 1ce42cc
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Thu Jul 17 16:23:21 2014 -0500
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Thu Jul 17 16:23:21 2014 -0500
----------------------------------------------------------------------
dev-tools/jira/__init__.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/b9017eab/dev-tools/jira/__init__.py
----------------------------------------------------------------------
diff --git a/dev-tools/jira/__init__.py b/dev-tools/jira/__init__.py
index 5a00ffe..15380aa 100755
--- a/dev-tools/jira/__init__.py
+++ b/dev-tools/jira/__init__.py
@@ -30,7 +30,7 @@ def jiratime(obj):
return None
return datetime.strptime(obj[0:19], "%Y-%m-%dT%H:%M:%S")
-githubUser = re.compile("Git[Hh]ub user (\w+)")
+githubUser = re.compile("Git[Hh]ub user ([\w-]+)")
githubPull = re.compile("https://github.com/[^/\s]+/[^/\s]+/pull/[0-9]+")
hasVote = re.compile("\s+([-+][01])\s*")
isDiff = re.compile("--- End diff --")
[47/50] git commit: Added STORM-419 to Changelog
Posted by bo...@apache.org.
Added STORM-419 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/c3a9a37a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/c3a9a37a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/c3a9a37a
Branch: refs/heads/security
Commit: c3a9a37a09ffa561505bad1c3adc68726dd7ad12
Parents: 9db723d
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 16:44:31 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 16:44:31 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/c3a9a37a/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 79c8b14..ccf0c3b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -31,6 +31,7 @@
* STORM-321: Added a tool to see the current status of STORM JIRA and github pulls.
* STORM-415: validate-launched-once in supervisor-test can not handle multiple topologies
* STORM-155: Storm rebalancing code causes multiple topologies assigned to a single port
+ * STORM-419: Updated test so sort ordering is very explicit.
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[05/50] git commit: Merge branch 'STORM-312'
Posted by bo...@apache.org.
Merge branch 'STORM-312'
Conflicts:
CHANGELOG.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/69b4601f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/69b4601f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/69b4601f
Branch: refs/heads/security
Commit: 69b4601f71bad51605813fca10a18ccb82859dd2
Parents: 96e81c1 ae383e3
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Wed Jul 16 09:20:20 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Wed Jul 16 09:20:20 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
bin/storm | 20 +-
.../src/clj/backtype/storm/command/monitor.clj | 37 +++
.../src/jvm/backtype/storm/utils/Monitor.java | 249 +++++++++++++++++++
5 files changed, 307 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/69b4601f/CHANGELOG.md
----------------------------------------------------------------------
diff --cc CHANGELOG.md
index f198eea,efd21c3..2817e5d
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -17,7 -17,7 +17,8 @@@
* STORM-351: multilang python process fall into endless loop
* STORM-375: Smarter downloading of assignments by supervisors and workers
* STORM-328: More restrictive Config checks, strict range check within Utils.getInt()
+ * STORM-381: Replace broken jquery.tablesorter.min.js to latest
+ * STORM-312: add storm monitor tools to monitor throughtput interactively
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
[42/50] git commit: Merge branch 'STORM-415' of
https://github.com/iwasakims/incubator-storm into STORM-415
Posted by bo...@apache.org.
Merge branch 'STORM-415' of https://github.com/iwasakims/incubator-storm into STORM-415
STORM-415: validate-launched-once in supervisor-test can not handle multiple topologies
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/f155e6f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/f155e6f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/f155e6f4
Branch: refs/heads/security
Commit: f155e6f445705e56b183e70f1ac64260ff74c0be
Parents: 5219f49 c78e08f
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Fri Jul 25 16:18:06 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Fri Jul 25 16:18:06 2014 -0500
----------------------------------------------------------------------
storm-core/test/clj/backtype/storm/supervisor_test.clj | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
[35/50] git commit: Updating contributors and CHANGELOG for STORM-399
Posted by bo...@apache.org.
Updating contributors and CHANGELOG for STORM-399
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/3a19ccc8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/3a19ccc8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/3a19ccc8
Branch: refs/heads/security
Commit: 3a19ccc8450638b1028c81d792d97fcbf226fbf5
Parents: e9238bc
Author: Derek Dagit <de...@yahoo-inc.com>
Authored: Thu Jul 24 16:58:01 2014 -0500
Committer: Derek Dagit <de...@yahoo-inc.com>
Committed: Thu Jul 24 16:58:01 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/3a19ccc8/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f78dd62..366a401 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,7 @@
* STORM-402: FileNotFoundException when using storm with apache tika
* STORM-364: The exception time display as default timezone.
* STORM-420: Missing quotes in storm-starter python code
+ * STORM-399: Kafka Spout defaulting to latest offset when current offset is older then 100k
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/3a19ccc8/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index f5d7873..cf67f64 100644
--- a/README.markdown
+++ b/README.markdown
@@ -158,6 +158,7 @@ under the License.
* Dave Parfitt ([@metadave](https://github.com/metadave))
* Jo Liss ([@joliss](https://github.com/joliss))
* averykhoo ([@averykhoo](https://github.com/averykhoo))
+* Curtis Allen ([@curtisallen](https://github.com/curtisallen))
## Acknowledgements
[17/50] git commit: Updated Changelog and Readme for STORM-403
Posted by bo...@apache.org.
Updated Changelog and Readme for STORM-403
Project: http://git-wip-us.apache.org/repos/asf/incubator-storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-storm/commit/c1aed00e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-storm/tree/c1aed00e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-storm/diff/c1aed00e
Branch: refs/heads/security
Commit: c1aed00e0090aba01a422c512bd8a9e447e80031
Parents: 3dd0f40
Author: Robert (Bobby) Evans <bo...@apache.org>
Authored: Mon Jul 21 11:30:04 2014 -0500
Committer: Robert (Bobby) Evans <bo...@apache.org>
Committed: Mon Jul 21 11:30:04 2014 -0500
----------------------------------------------------------------------
CHANGELOG.md | 1 +
README.markdown | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/c1aed00e/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 76237d6..1e9a7e6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,6 +21,7 @@
* STORM-312: add storm monitor tools to monitor throughtput interactively
* STORM-354: Testing: allow users to pass TEST-TIMEOUT-MS as param for complete-topology
* STORM-254: one Spout/Bolt can register metric twice with same name in different timeBucket
+ * STORM-403: heartbeats-to-nimbus in supervisor-test failed due to uninten...
## 0.9.2-incubating
* STORM-66: send taskid on initial handshake
http://git-wip-us.apache.org/repos/asf/incubator-storm/blob/c1aed00e/README.markdown
----------------------------------------------------------------------
diff --git a/README.markdown b/README.markdown
index 063de70..c495c19 100644
--- a/README.markdown
+++ b/README.markdown
@@ -152,6 +152,7 @@ under the License.
* Jungtaek Lim ([@HeartSaVioR](https://github.com/HeartSaVioR))
* Li Jiahong ([@Gvain](https://github.com/Gvain))
* Aaron Levin ([@aaronlevin](https://github.com/aaronlevin))
+* Masatake Iwasaki ([@iwasakims](https://github.com/iwasakims))
## Acknowledgements