You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by rfarivar <gi...@git.apache.org> on 2016/01/25 18:47:14 UTC
[GitHub] storm pull request: Storm 1226
GitHub user rfarivar opened a pull request:
https://github.com/apache/storm/pull/1043
Storm 1226
This is translating util.clj into java functions mostly in utils/Utils.java.
Translation is half way done, passing all unit tests. Putting the pull request up to get reviews.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rfarivar/storm STORM-1226
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1043.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1043
----
commit cc2daf3b41df371b85c505f3f25cf6de683e8eb8
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-13T20:05:24Z
first batch of util.clj functions translated to java
commit 41d229be2c26b4dc28cb0242624e7194bec4bd9d
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-13T20:59:06Z
upmerged with org.apache.storm reorg
commit 912d14c169a84f25f1814651167cb603ad544fc6
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-14T19:51:07Z
fixing the failing test in supervisor-test due to find-first translation
commit 65522f09e9bbf116d4bd6b5b3d3018ddbf6f3141
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-14T20:17:47Z
translation finished up to find-first, passing all tests
commit d4a7af67596557eccde232f8246617504f5460b6
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-20T16:37:06Z
passing test, done up to uuid
commit b6edf16937a92f97b2263e07a17cbcf1975a8da8
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2016-01-20T22:11:59Z
translate batch 3
commit 7de457eff0e514b9578c7c23ff89aff16d1e87b1
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-20T22:23:27Z
passing tests. In the middle of replacing map-val
commit c0aff49cacd95de10ac31afa770d5d511595739e
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2016-01-20T23:07:27Z
Merge branch 'STORM-1226' of https://github.com/rfarivar/storm into ystorm-2572-util-translate-batch-3
commit 4067809bfcf5921fc1a97915ac331e1a1c4a1d05
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2016-01-21T15:36:49Z
fix failing tests
commit 56d48faac918123af46f1e021e38549fa6a4cf6c
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2016-01-21T20:44:44Z
creates package to help with mocking static methods
commit f478858842ee075492e7392a72a3a1605c48b66e
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-21T20:46:28Z
Batch 1 complete, all tests passing
commit 43da3c607b38f7b45a1f54b6be3149f994407c4e
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2016-01-21T21:32:21Z
moves staticmocking under test/
commit b370d9e9b76fe146728403be69ea151baf933a46
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2016-01-21T21:57:18Z
Merge remote-tracking branch 'reza/STORM-1226' into ystorm-2572-util-translate-batch-3
commit 77e10d4c151dbd9ec3b418686b5f04ddc06db9e4
Author: Reza Farivar <rf...@yahoo-inc.com>
Date: 2016-01-22T21:07:19Z
Merge pull request #1 from d2r/ystorm-2572-util-translate-batch-3
util translate batch 3
commit eff86b2493a09536d79dbda95bb2099dfa1abf67
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-25T17:05:44Z
Translation up to the middle of batch 2, batch 1 and 3 complete
commit 69c3ab748684db5a35f8abf6ca61fe7cfc45894c
Author: rfarivar <rf...@yahoo-inc.com>
Date: 2016-01-25T17:45:07Z
upmerged to master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50895476
--- Diff: storm-core/src/jvm/org/apache/storm/utils/IFn.java ---
@@ -0,0 +1,8 @@
+package org.apache.storm.utils;
+
+/**
+ * Created by rfarivar on 1/12/16.
+ */
+public interface IFn {
+ Object eval (Object obj);
+}
--- End diff --
Please delete this file and the mapVal function in Utils. No IFn conflicts with clojure please it is too confusing and the mapVal function is not actually used so lets just remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/1043#issuecomment-176296318
Also, will you kindly edit the title of this PR so it follows protocol?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50895014
--- Diff: storm-core/src/clj/org/apache/storm/testing.clj ---
@@ -519,10 +556,12 @@
;; TODO: the idea of mocking for transactional topologies should be done an
;; abstraction level above... should have a complete-transactional-topology for this
(let [{topology :topology capturer :capturer} (capture-topology topology)
- storm-name (or topology-name (str "topologytest-" (uuid)))
+ storm-name (or topology-name (str "topologytest-" (Utils/uuid)))
state (:storm-cluster-state cluster-map)
spouts (.get_spouts topology)
replacements (map-val (fn [v]
+; replacements (Utils/mapVal
+; (reify IFn (eval [this v]
--- End diff --
Please remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50897241
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
--- End diff --
Can you make this a javadoc instead of just a comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898272
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
--- End diff --
StringBuilder here too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894857
--- Diff: storm-core/src/clj/org/apache/storm/stats.clj ---
@@ -702,11 +713,16 @@
:acked (sum-or-0 (:acked acc-spout-stats) (:acked spout-stats))
:failed (sum-or-0 (:failed acc-spout-stats) (:failed spout-stats))})
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn aggregate-count-streams
[stats]
(->> stats
- (map-val #(reduce + (vals %)))))
+ (map-val #(reduce + (vals %)))))
+; (Utils/mapVal
+; (reify IFn (eval [this x] (reduce + (vals x))))
+; )))
--- End diff --
Please remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50749227
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
--- End diff --
Is this also supposed to be commented out?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50743788
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
--- End diff --
This if, else, and do-while can be replaced by a while:
```
while (iter != null && iter.hasNext()) {
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50896029
--- Diff: storm-core/src/jvm/org/apache/storm/utils/IPredicate.java ---
@@ -0,0 +1,8 @@
+package org.apache.storm.utils;
+
+/**
+ * Created by rfarivar on 1/12/16.
+ */
+public interface IPredicate {
--- End diff --
Have the javadocs explain the interface, not who created it and when, git is for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893250
--- Diff: storm-core/src/clj/org/apache/storm/cluster_state/zookeeper_state_factory.clj ---
@@ -90,7 +91,7 @@
(if (zk/exists zk-writer path false)
(zk/set-data zk-writer path data)
(do
- (zk/mkdirs zk-writer (parent-path path) acls)
+ (zk/mkdirs zk-writer (Utils/parentPath path) acls)
--- End diff --
Like I said in #1045 there are some path manipulation functions that look like they belong in a Zookeeper.java file not in Utils.java. Please work with them to figure out how to move them, even if it is in a follow on JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50901067
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
+ if (files != null) {
+ for (File f: files) {
+ ret.add(f.getName());
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the value of java.class.path System property. Kept separate for
+ * testing.
+ * @return the classpath
+ */
+ public static String currentClasspath() {
+ return _instance.currentClasspathImpl();
+ }
+
+ public String currentClasspathImpl() {
+ return System.getProperty("java.class.path");
+ }
+
+ /**
+ * Returns a collection of jar file names found under the given directory.
+ * @param dir the directory to search
+ * @return the jar file names
+ */
+ private static Collection<String> getFullJars(String dir) {
+ File[] files = new File(dir).listFiles(new FilenameFilter() {
+ @Override
+ public boolean accept(File dir, String name) {
+ return name.endsWith(".jar");
+ }
+ });
+ Collection<String> ret = new HashSet<String>(files.length);
+ for (File f : files) {
+ ret.add(Paths.get(dir, f.getName()).toString());
+ }
+ return ret;
+ }
+
+ public static String workerClasspath() {
+ String stormDir = System.getProperty("storm.home");
+ String stormLibDir = Paths.get(stormDir, "lib").toString();
+ String stormConfDir =
+ System.getenv("STORM_CONF_DIR") != null ?
+ System.getenv("STORM_CONF_DIR") :
+ Paths.get(stormDir, "conf").toString();
+ String stormExtlibDir = Paths.get(stormDir, "extlib").toString();
+ String extcp = System.getenv("STORM_EXT_CLASSPATH");
+ if (stormDir == null) {
+ return Utils.currentClasspath();
+ }
+ List<String> pathElements = new LinkedList<>();
+ pathElements.addAll(Utils.getFullJars(stormLibDir));
+ pathElements.addAll(Utils.getFullJars(stormExtlibDir));
+ pathElements.add(extcp);
+ pathElements.add(stormConfDir);
+
+ return StringUtils.join(pathElements,
+ System.getProperty("path.separator"));
+ }
+
+ public static String addToClasspath(String classpath,
+ Collection<String> paths) {
+ return _instance.addToClasspathImpl(classpath, paths);
+ }
+
+ public String addToClasspathImpl(String classpath,
+ Collection<String> paths) {
+ if (paths == null || paths.isEmpty()) {
+ return classpath;
+ }
+ List<String> l = new LinkedList<>();
+ l.add(classpath);
--- End diff --
In the previous code it would return "nil:a:b:c" now it will return "null:a:b" In this case I am not sure it makes a difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898350
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
--- End diff --
Blank line above this one please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50896659
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -530,6 +559,10 @@ private static boolean downloadResourcesAsSupervisorAttempt(ClientBlobStore cb,
return isSuccess;
}
+ public static boolean checkFileExists(String path) {
+ return Files.exists(new File(path).toPath());
+ }
--- End diff --
Can we file a JIRA to split the file system utils out of Utils. There are a lot of these and they should be in their own java file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893939
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -178,18 +201,25 @@
(.set_enable (get options :enable false))
(.set_samplingpct (get options :samplingpct 10))))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn thriftify-storm-base [storm-base]
(doto (StormBase.)
(.set_name (:storm-name storm-base))
(.set_launch_time_secs (int (:launch-time-secs storm-base)))
(.set_status (convert-to-status-from-symbol (:status storm-base)))
(.set_num_workers (int (:num-workers storm-base)))
+ ;(.set_component_executors (Utils/mapVal (reify IFn (eval [this x] (int x))) (:component->executors storm-base)))
(.set_component_executors (map-val int (:component->executors storm-base)))
(.set_owner (:owner storm-base))
(.set_topology_action_options (thriftify-topology-action-options storm-base))
(.set_prev_status (convert-to-status-from-symbol (:prev-status storm-base)))
+; (.set_component_debug
+; (Utils/mapVal
+; (reify IFn (eval [this x] (thriftify-debugoptions x)))
+; (:component->debug storm-base)))))
--- End diff --
And these too. Please delete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893972
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -201,17 +231,28 @@
(.get_owner storm-base)
(clojurify-topology-action-options (.get_topology_action_options storm-base))
(convert-to-symbol-from-status (.get_prev_status storm-base))
+; (Utils/mapVal
+; (reify IFn (eval [this x] (clojurify-debugoptions x)))
+; (.get_component_debug storm-base)))))
(map-val clojurify-debugoptions (.get_component_debug storm-base)))))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
--- End diff --
only one needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50897502
--- Diff: storm-core/src/clj/org/apache/storm/util.clj ---
@@ -48,21 +48,21 @@
(:require [ring.util.codec :as codec])
(:use [org.apache.storm log]))
-(defn wrap-in-runtime
- "Wraps an exception in a RuntimeException if needed"
- [^Exception e]
- (if (instance? RuntimeException e)
- e
- (RuntimeException. e)))
+;(defn wrap-in-runtime
+; "Wraps an exception in a RuntimeException if needed"
+; [^Exception e]
+; (if (instance? RuntimeException e)
+; e
+; (RuntimeException. e)))
(def on-windows?
(= "Windows_NT" (System/getenv "OS")))
-(def file-path-separator
- (System/getProperty "file.separator"))
+;(def file-path-separator
+; (System/getProperty "file.separator"))
-(def class-path-separator
- (System/getProperty "path.separator"))
+;(def class-path-separator
+; (System/getProperty "path.separator"))
(defn is-absolute-path? [path]
(.isAbsolute (Paths/get path (into-array String []))))
--- End diff --
Can we delete this now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50899194
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
--- End diff --
Didn't have them before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50752006
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
--- End diff --
Don't want null-pointer checks here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898419
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
--- End diff --
Lets just delete this function, unless we actually end up using it somewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50748875
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
--- End diff --
```
if (toks == null || toks.size() == 0) { ...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894663
--- Diff: storm-core/src/clj/org/apache/storm/local_state.clj ---
@@ -104,17 +105,24 @@
(->executor-list (.get_executors thrift-local-assignment))
(.get_resources thrift-local-assignment)))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn ls-local-assignments!
[^LocalState local-state assignments]
- (let [local-assignment-map (map-val ->LocalAssignment assignments)]
- (.put local-state LS-LOCAL-ASSIGNMENTS
+ (let [local-assignment-map (map-val ->LocalAssignment assignments)]
+; (let [local-assignment-map (Utils/mapVal
+; (reify IFn (eval [this x] (->LocalAssignment x)))
+; assignments)]
--- End diff --
Please remove commented out code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/1043#issuecomment-178148350
It would be good to move `MockedUtils.java` into `org/apache/storm/testing/staticmocking` where the other mocking stuff is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50750687
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
--- End diff --
Don't want null-pointer checks here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50902361
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1050,6 +1083,12 @@ public String getBackupConnectionString() throws Exception {
}
}
+ public static void testSetupBuilder(CuratorFrameworkFactory.Builder
--- End diff --
This was originally in a separate class to separate it out only for testing purposes. I am OK with having this here, but lest add some javadocs to say this is only for testing. The name should give a lot of a warning, but a full explanation would be good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50753665
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
+ if (files != null) {
+ for (File f: files) {
+ ret.add(f.getName());
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the value of java.class.path System property. Kept separate for
+ * testing.
+ * @return the classpath
+ */
+ public static String currentClasspath() {
+ return _instance.currentClasspathImpl();
+ }
+
+ public String currentClasspathImpl() {
+ return System.getProperty("java.class.path");
+ }
+
+ /**
+ * Returns a collection of jar file names found under the given directory.
+ * @param dir the directory to search
+ * @return the jar file names
+ */
+ private static Collection<String> getFullJars(String dir) {
+ File[] files = new File(dir).listFiles(new FilenameFilter() {
+ @Override
+ public boolean accept(File dir, String name) {
+ return name.endsWith(".jar");
+ }
+ });
+ Collection<String> ret = new HashSet<String>(files.length);
+ for (File f : files) {
+ ret.add(Paths.get(dir, f.getName()).toString());
+ }
+ return ret;
+ }
+
+ public static String workerClasspath() {
+ String stormDir = System.getProperty("storm.home");
+ String stormLibDir = Paths.get(stormDir, "lib").toString();
+ String stormConfDir =
+ System.getenv("STORM_CONF_DIR") != null ?
+ System.getenv("STORM_CONF_DIR") :
+ Paths.get(stormDir, "conf").toString();
+ String stormExtlibDir = Paths.get(stormDir, "extlib").toString();
+ String extcp = System.getenv("STORM_EXT_CLASSPATH");
+ if (stormDir == null) {
+ return Utils.currentClasspath();
+ }
+ List<String> pathElements = new LinkedList<>();
+ pathElements.addAll(Utils.getFullJars(stormLibDir));
+ pathElements.addAll(Utils.getFullJars(stormExtlibDir));
+ pathElements.add(extcp);
+ pathElements.add(stormConfDir);
+
+ return StringUtils.join(pathElements,
+ System.getProperty("path.separator"));
+ }
+
+ public static String addToClasspath(String classpath,
+ Collection<String> paths) {
+ return _instance.addToClasspathImpl(classpath, paths);
+ }
+
+ public String addToClasspathImpl(String classpath,
+ Collection<String> paths) {
+ if (paths == null || paths.isEmpty()) {
+ return classpath;
+ }
+ List<String> l = new LinkedList<>();
+ l.add(classpath);
--- End diff --
What if classpath == null
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50900303
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
+ if (files != null) {
+ for (File f: files) {
+ ret.add(f.getName());
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the value of java.class.path System property. Kept separate for
+ * testing.
+ * @return the classpath
+ */
+ public static String currentClasspath() {
+ return _instance.currentClasspathImpl();
+ }
+
+ public String currentClasspathImpl() {
+ return System.getProperty("java.class.path");
+ }
+
+ /**
+ * Returns a collection of jar file names found under the given directory.
+ * @param dir the directory to search
+ * @return the jar file names
+ */
+ private static Collection<String> getFullJars(String dir) {
+ File[] files = new File(dir).listFiles(new FilenameFilter() {
+ @Override
+ public boolean accept(File dir, String name) {
+ return name.endsWith(".jar");
+ }
+ });
+ Collection<String> ret = new HashSet<String>(files.length);
--- End diff --
We need to check for files if it is null or not. Previously if a dir did not exist getFullJars would return an empty set. Now it will NPE here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894778
--- Diff: storm-core/src/clj/org/apache/storm/scheduler/IsolationScheduler.clj ---
@@ -41,6 +51,7 @@
(map vector (repeat-seq (range (.getNumWorkers details))))
(group-by first)
(map-val #(map second %))
+ ;(Utils/mapVal (reify IFn (eval [this x] (map second x))))
--- End diff --
Please remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50896065
--- Diff: storm-core/src/jvm/org/apache/storm/utils/StaticMockable.java ---
@@ -0,0 +1,6 @@
+package org.apache.storm.utils;
--- End diff --
Apache header needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50754345
--- Diff: storm-core/test/clj/org/apache/storm/security/auth/auth_test.clj ---
@@ -14,27 +14,28 @@
;; See the License for the specific language governing permissions and
;; limitations under the License.
(ns org.apache.storm.security.auth.auth-test
- (:use [clojure test])
- (:require [org.apache.storm.daemon [nimbus :as nimbus]])
- (:import [org.apache.thrift TException]
+ (:use [clojure test])
--- End diff --
May not need to change the indentions here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894025
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -201,17 +231,28 @@
(.get_owner storm-base)
(clojurify-topology-action-options (.get_topology_action_options storm-base))
(convert-to-symbol-from-status (.get_prev_status storm-base))
+; (Utils/mapVal
+; (reify IFn (eval [this x] (clojurify-debugoptions x)))
+; (.get_component_debug storm-base)))))
(map-val clojurify-debugoptions (.get_component_debug storm-base)))))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn thriftify-stats [stats]
(if stats
+; (Utils/mapVal
+; (reify IFn (eval [this x] (thriftify-executor-stats x)))
(map-val thriftify-executor-stats
(map-key #(ExecutorInfo. (int (first %1)) (int (last %1)))
stats))
{}))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
--- End diff --
only one needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50899023
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
--- End diff --
FileNotFoundException not IOException. Other IOExceptions are thrown, like I don't have permission to delete this. FileNotFound is I lost a race and don't need to delete it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50900498
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
+ if (files != null) {
+ for (File f: files) {
+ ret.add(f.getName());
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the value of java.class.path System property. Kept separate for
+ * testing.
+ * @return the classpath
+ */
+ public static String currentClasspath() {
+ return _instance.currentClasspathImpl();
+ }
+
+ public String currentClasspathImpl() {
+ return System.getProperty("java.class.path");
+ }
+
+ /**
+ * Returns a collection of jar file names found under the given directory.
+ * @param dir the directory to search
+ * @return the jar file names
+ */
+ private static Collection<String> getFullJars(String dir) {
+ File[] files = new File(dir).listFiles(new FilenameFilter() {
+ @Override
+ public boolean accept(File dir, String name) {
+ return name.endsWith(".jar");
+ }
+ });
+ Collection<String> ret = new HashSet<String>(files.length);
--- End diff --
I am also not sure if we want a HashSet. In the past the order of the jars (which actually is significant for a classpath) was ordered by the operating system. Not perfect but whatever, now it is ordered by HashSet. I would rather see a List returned, and if we have duplicates oh well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50748974
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
--- End diff --
```
if ( amap == null || amap.keySet() == null) { ...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50752250
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
--- End diff --
null-check?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50750102
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
--- End diff --
Can we make a comment so people aren't confused about why we need _instance and *Impl methods? Just like a little one-line thingy saying "for mocking purposes" or something like that over the *Impl(…) methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50900760
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
+ if (files != null) {
+ for (File f: files) {
+ ret.add(f.getName());
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the value of java.class.path System property. Kept separate for
+ * testing.
+ * @return the classpath
+ */
+ public static String currentClasspath() {
+ return _instance.currentClasspathImpl();
+ }
+
+ public String currentClasspathImpl() {
+ return System.getProperty("java.class.path");
+ }
+
+ /**
+ * Returns a collection of jar file names found under the given directory.
+ * @param dir the directory to search
+ * @return the jar file names
+ */
+ private static Collection<String> getFullJars(String dir) {
+ File[] files = new File(dir).listFiles(new FilenameFilter() {
+ @Override
+ public boolean accept(File dir, String name) {
+ return name.endsWith(".jar");
+ }
+ });
+ Collection<String> ret = new HashSet<String>(files.length);
+ for (File f : files) {
+ ret.add(Paths.get(dir, f.getName()).toString());
+ }
+ return ret;
+ }
+
+ public static String workerClasspath() {
+ String stormDir = System.getProperty("storm.home");
+ String stormLibDir = Paths.get(stormDir, "lib").toString();
+ String stormConfDir =
+ System.getenv("STORM_CONF_DIR") != null ?
+ System.getenv("STORM_CONF_DIR") :
+ Paths.get(stormDir, "conf").toString();
+ String stormExtlibDir = Paths.get(stormDir, "extlib").toString();
+ String extcp = System.getenv("STORM_EXT_CLASSPATH");
+ if (stormDir == null) {
+ return Utils.currentClasspath();
+ }
+ List<String> pathElements = new LinkedList<>();
+ pathElements.addAll(Utils.getFullJars(stormLibDir));
+ pathElements.addAll(Utils.getFullJars(stormExtlibDir));
+ pathElements.add(extcp);
+ pathElements.add(stormConfDir);
+
+ return StringUtils.join(pathElements,
+ System.getProperty("path.separator"));
--- End diff --
Why not use the classPathSeparator we defined above?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r51616229
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
--- End diff --
It's used in a lot of places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893638
--- Diff: storm-core/src/clj/org/apache/storm/config.clj ---
@@ -91,25 +91,25 @@
(let [storm-home (System/getProperty "storm.home")
path (conf STORM-LOCAL-DIR)]
(if path
- (if (is-absolute-path? path) path (str storm-home file-path-separator path))
- (str storm-home file-path-separator "storm-local"))))
+ (if (is-absolute-path? path) path (str storm-home Utils/filePathSeparator path))
+ (str storm-home Utils/filePathSeparator "storm-local"))))
--- End diff --
The pull request for Config.clj #1030 and the pull request for zookeeper #1045 each ported the path separator differently. Can we at least file a JIRA to make it common in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50903820
--- Diff: storm-core/test/clj/org/apache/storm/supervisor_test.clj ---
@@ -545,13 +569,14 @@
(stubbing [uptime-computer nil
cluster/mk-storm-cluster-state nil
supervisor-state nil
- local-hostname nil
+ ;local-hostname nil
--- End diff --
delete
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50895500
--- Diff: storm-core/src/jvm/org/apache/storm/utils/IPredicate.java ---
@@ -0,0 +1,8 @@
+package org.apache.storm.utils;
--- End diff --
Apache header
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50754051
--- Diff: storm-core/test/clj/org/apache/storm/nimbus_test.clj ---
@@ -1198,21 +1220,23 @@
(let [expected-name topology-name
expected-conf {TOPOLOGY-NAME expected-name
- :foo :bar}]
+ "foo" "bar"}]
--- End diff --
Should we apply clojurify-structure to the map and avoid changing the key here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50897100
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
--- End diff --
That is fine, please delete this comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894670
--- Diff: storm-core/src/clj/org/apache/storm/local_state.clj ---
@@ -104,17 +105,24 @@
(->executor-list (.get_executors thrift-local-assignment))
(.get_resources thrift-local-assignment)))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn ls-local-assignments!
[^LocalState local-state assignments]
- (let [local-assignment-map (map-val ->LocalAssignment assignments)]
- (.put local-state LS-LOCAL-ASSIGNMENTS
+ (let [local-assignment-map (map-val ->LocalAssignment assignments)]
+; (let [local-assignment-map (Utils/mapVal
+; (reify IFn (eval [this x] (->LocalAssignment x)))
+; assignments)]
+ (.put local-state LS-LOCAL-ASSIGNMENTS
(LSSupervisorAssignments. local-assignment-map))))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn ls-local-assignments
[^LocalState local-state]
(if-let [thrift-local-assignments (.get local-state LS-LOCAL-ASSIGNMENTS)]
(map-val
->local-assignment
+ ; (Utils/mapVal
+ ; (reify IFn (eval [this x] (->LocalAssignment x)))
--- End diff --
here too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1043: Storm 1226
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1043
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50752527
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
+ if (files != null) {
+ for (File f: files) {
+ ret.add(f.getName());
+ }
+ }
+ return ret;
+ }
+
+ /**
+ * Returns the value of java.class.path System property. Kept separate for
+ * testing.
+ * @return the classpath
+ */
+ public static String currentClasspath() {
+ return _instance.currentClasspathImpl();
+ }
+
+ public String currentClasspathImpl() {
+ return System.getProperty("java.class.path");
+ }
+
+ /**
+ * Returns a collection of jar file names found under the given directory.
+ * @param dir the directory to search
+ * @return the jar file names
+ */
+ private static Collection<String> getFullJars(String dir) {
+ File[] files = new File(dir).listFiles(new FilenameFilter() {
--- End diff --
null
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50896559
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Time.java ---
@@ -102,6 +102,14 @@ public static long currentTimeMillis() {
public static int currentTimeSecs() {
return (int) (currentTimeMillis() / 1000);
}
+
+ public static int delta(int timeInSeconds) {
+ return Time.currentTimeSecs() - timeInSeconds;
+ }
+
+ public static long deltaMs(long timeInMilliseconds) {
--- End diff --
I am conflicted about this. The function feels like it should be a part of time, but Time provides APIs that can simulate time, but this function does not. At a minimum can we add some javadocs explaining this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50895133
--- Diff: storm-core/src/clj/org/apache/storm/util.clj ---
@@ -48,21 +48,21 @@
(:require [ring.util.codec :as codec])
(:use [org.apache.storm log]))
-(defn wrap-in-runtime
- "Wraps an exception in a RuntimeException if needed"
- [^Exception e]
- (if (instance? RuntimeException e)
- e
- (RuntimeException. e)))
+;(defn wrap-in-runtime
--- End diff --
+1 yes delete them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894044
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -201,17 +231,28 @@
(.get_owner storm-base)
(clojurify-topology-action-options (.get_topology_action_options storm-base))
(convert-to-symbol-from-status (.get_prev_status storm-base))
+; (Utils/mapVal
+; (reify IFn (eval [this x] (clojurify-debugoptions x)))
+; (.get_component_debug storm-base)))))
(map-val clojurify-debugoptions (.get_component_debug storm-base)))))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn thriftify-stats [stats]
(if stats
+; (Utils/mapVal
+; (reify IFn (eval [this x] (thriftify-executor-stats x)))
(map-val thriftify-executor-stats
(map-key #(ExecutorInfo. (int (first %1)) (int (last %1)))
stats))
{}))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn clojurify-stats [stats]
(if stats
+; (Utils/mapVal
+; (reify IFn (eval [this x] (clojurify-executor-stats x)))
--- End diff --
and delete these too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893744
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -71,6 +72,8 @@
(:worker->resources assignment)))))
thrift-assignment))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-key with a proper for loop HERE
--- End diff --
Only need one copy of this TODO. I assume this will be cleaned up with a follow on JIRA
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50899281
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
--- End diff --
Again didn't have them before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898579
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
--- End diff --
I don't really want commented out code, or dead code. Can we please delete it unless we are actually using it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50896874
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
--- End diff --
On these please follow the java naming convetions for static final values `IS_ON_WINDOWS`, `FILE_PATH_SEPARATOR`, and `CLASS_PATH_SEPARATOR`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/1043#issuecomment-179485969
Should close this PR because of #1074
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50745533
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
--- End diff --
Same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/storm/pull/1043#issuecomment-178178677
Let's rename the title of this pull request so it describes the work being done. Something about translating the utils namespace to Java.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893784
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -80,6 +83,17 @@
(fn [list-of-executors]
(into [] list-of-executors)) ; list of executors must be coverted to clojure vector to ensure it is sortable.
executor->node_port))))
+;(defn clojurify-executor->node_port [executor->node_port]
+; (into {}
+; (Utils/mapVal
+; (reify IFn (eval [this nodeInfo]
+; (concat [(.get_node nodeInfo)] (.get_port nodeInfo)))) ;nodeInfo should be converted to [node,port1,port2..]
+; (map-key
+; (fn [list-of-executors]
+; (into [] list-of-executors)) ; list of executors must be coverted to clojure vector to ensure it is sortable.
+; executor->node_port))))
+
+
--- End diff --
Please delete this if it is not needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50754732
--- Diff: storm-core/src/clj/org/apache/storm/util.clj ---
@@ -48,21 +48,21 @@
(:require [ring.util.codec :as codec])
(:use [org.apache.storm log]))
-(defn wrap-in-runtime
- "Wraps an exception in a RuntimeException if needed"
- [^Exception e]
- (if (instance? RuntimeException e)
- e
- (RuntimeException. e)))
+;(defn wrap-in-runtime
--- End diff --
May delete the clojure functions rather than comment them out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50749149
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
--- End diff --
So..... Is the recommendation not to use this function, or what does this mean?
Add a prescriptive comment describing what we want the user of this code to do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50899355
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
+ }
+ LOG.error("halting process: " + combinedErrorMessage, new RuntimeException(combinedErrorMessage));
+ Runtime.getRuntime().exit(val);
+ }
+
+ public static Double sum(Collection<Number> vals) {
+ double sum = 0.0;
+ double dVal;
+ if (vals == null) {
+ return 0.0;
+ }
+ for (Number val: vals) {
+ dVal = val.doubleValue();
+ sum = sum + dVal;
+ }
+ return new Double(sum);
+ }
+
+
+ public static Object defaulted(Object val, Object defaultObj) {
+ if (val != null) {
+ return val;
+ } else {
+ return defaultObj;
+ }
+ }
+
+ /**
+ * Deletes a file or directory and its contents if it exists. Does not
+ * complain if the input is null or does not exist.
+ * @param path the path to the file or directory
+ */
+ public static void forceDelete(String path) {
+ _instance.forceDeleteImpl(path);
+ }
+
+ protected void forceDeleteImpl(String path) {
+ LOG.debug("Deleting path {}", path);
+ if (checkFileExists(path)) {
+ try {
+ FileUtils.forceDelete(new File(path));
+ } catch (IOException ignored) {}
+ }
+ }
+
+ /**
+ * Creates a symbolic link to the target
+ * @param dir the parent directory of the link
+ * @param targetDir the parent directory of the link's target
+ * @param targetFilename the file name of the links target
+ * @param filename the file name of the link
+ * @return the path of the link if it did not exist, otherwise null
+ * @throws IOException
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename, String filename) throws IOException {
+ Path path = Paths.get(dir, filename).toAbsolutePath();
+ Path target = Paths.get(targetDir, targetFilename).toAbsolutePath();
+ LOG.debug("Creating symlink [{}] to [{}]", path, target);
+ if (!path.toFile().exists()) {
+ return Files.createSymbolicLink(path, target);
+ }
+ return null;
+ }
+
+ /**
+ * Convenience method for the case when the link's file name should be the
+ * same as the file name of the target
+ */
+ public static Path createSymlink(String dir, String targetDir,
+ String targetFilename) throws IOException {
+ return Utils.createSymlink(dir, targetDir, targetFilename,
+ targetFilename);
+ }
+
+ /**
+ * Returns a Collection of file names found under the given directory.
+ * @param dir a directory
+ * @return the Collection of file names
+ */
+ public static Collection<String> readDirContents(String dir) {
+ Collection<String> ret = new HashSet<>();
+ File[] files = new File(dir).listFiles();
--- End diff --
Didn't have it before. Will still get an NPE if it happens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50897000
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
--- End diff --
The interface for pred returns a `Boolean` not a `boolean` as such an NPE is possible here. Can we switch it to little b boolean to avoid this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893895
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -131,6 +151,9 @@
(if (:num-workers rebalance-options)
(.set_num_workers thrift-rebalance-options (int (:num-workers rebalance-options))))
(if (:component->executors rebalance-options)
+; (.set_num_executors thrift-rebalance-options (Utils/mapVal
+; (reify IFn (eval [this x] (int x)))
+; (:component->executors rebalance-options))))
--- End diff --
Please delete this if it is not needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898231
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
--- End diff --
Can we please use a StringBuilder instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50902614
--- Diff: storm-core/test/clj/org/apache/storm/logviewer_test.clj ---
@@ -360,23 +367,24 @@
;; match.
exp-offset-fn #(- (/ logviewer/default-bytes-per-page 2) %)]
- (stubbing [local-hostname expected-host
- logviewer/logviewer-port expected-port]
-
- (testing "Logviewer link centers the match in the page"
- (let [expected-fname "foobar.log"]
- (is (= (str "http://"
- expected-host
- ":"
- expected-port
- "/log?file="
- expected-fname
- "&start=1947&length="
- logviewer/default-bytes-per-page)
- (logviewer/url-to-match-centered-in-log-page (byte-array 42)
- expected-fname
- 27526
- 8888)))))
+ ;(stubbing [local-hostname expected-host
--- End diff --
Please delete
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898732
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
+ for (Object oneMessage: msg) {
+ combinedErrorMessage = combinedErrorMessage + oneMessage.toString();
--- End diff --
Please don't use toString, use String Builder.append. I know that if it is null it will output "null" but if this is null we will get an NPE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898645
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
+
+ public static long secsToMillisLong(double secs) {
+ return (long) (1000 * secs);
+ }
+
+ public static Vector<String> tokenizePath (String path) {
+ String[] tokens = path.split("/");
+ Vector<String> outputs = new Vector<String>();
+ if (tokens == null || tokens.length == 0) {
+ return null;
+ }
+ for (String tok: tokens) {
+ if (!tok.isEmpty()) {
+ outputs.add(tok);
+ }
+ }
+ return outputs;
+ }
+
+ public static String parentPath(String path) {
+ if (path == null) {
+ return "/";
+ }
+ Vector<String> tokens = tokenizePath(path);
+ int length = tokens.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length - 1; i++) { //length - 1 to mimic "butlast" from the old clojure code
+ output = output + "/" + tokens.get(i);
+ }
+ return output;
+ }
+
+ public static String toksToPath (Vector<String> toks) {
+ if (toks == null) {
+ return "/";
+ }
+ int length = toks.size();
+ if (length == 0) {
+ return "/";
+ }
+ String output = "";
+ for (int i = 0; i < length; i++) {
+ output = output + "/" + toks.get(i);
+ }
+ return output;
+ }
+ public static String normalizePath (String path) {
+ return toksToPath(tokenizePath(path));
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+ public static Map mapVal (IFn aFn, Map amap) {
+ Map newMap = new HashMap();
+ if (amap == null) {
+ return newMap;
+ }
+ if (amap.keySet()==null) {
+ return newMap;
+ }
+ for (Object key: amap.keySet()) {
+ Object value = amap.get(key);
+ if (value == null) {
+ newMap.put(key, null);
+ } else {
+ Object newValue = aFn.eval(value);
+ newMap.put(key, newValue);
+ }
+ }
+ return newMap;
+ }
+
+ /* TODO: This function was originally written to replace map-val in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map filterVal(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(value)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace filter-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop + if conditionals. This is mentioned in TODOs
+ across the clojure files.
+ */
+// public static Map filterKey(IPredicate aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// if(aFn.test(key)) {
+// newMap.put(key, value);
+// }
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace map-key in util.clj. But we decided to change the coding
+ style in the caller functions to a more Java style for loop. This is mentioned in TODOs across the clojure
+ files.
+ */
+// public static Map mapKey (IFn aFn, Map amap) {
+// Map newMap = new HashMap();
+// if (amap == null) {
+// return newMap;
+// }
+// for (Object key: amap.keySet()) {
+// Object value = amap.get(key);
+// Object newKey = aFn.eval(key);
+// newMap.put(newKey, value);
+// }
+// return newMap;
+// }
+
+ /* TODO: This function was originally written to replace separate in util.clj. But since it was only used in
+ transactional_test.clj, the separate function was moved there for now.
+ */
+ public static Vector<Collection> separate (IPredicate pred, Collection aseq) {
+ Vector<Collection> outputVector = new Vector<Collection>();
+ Collection pass = new HashSet();
+ Collection notPass = new HashSet();
+ for (Object obj: aseq) {
+ if (pred.test(obj)) {
+ pass.add(obj);
+ } else {
+ notPass.add(obj);
+ }
+ }
+ outputVector.add(pass);
+ outputVector.add(notPass);
+ return outputVector;
+ }
+
+ public static void exitProcess (int val, Object... msg) {
+ String combinedErrorMessage = "";
--- End diff --
Can we please us a StringBuilder here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50898011
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
+ }
+
+ public static String uuid() {
+ return UUID.randomUUID().toString();
+ }
+
+ public static int currentTimeSecs() {
+ return Time.currentTimeSecs();
+ }
+
+ public static long currentTimeMillis() {
+ return Time.currentTimeMillis();
+ }
--- End diff --
Can we please inline `currentTimeSecs` and `currentTimeMillis`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50753833
--- Diff: storm-core/test/clj/org/apache/storm/nimbus_test.clj ---
@@ -352,6 +372,7 @@
(is (= 2 (storm-num-workers state "mystorm"))) ;; because only 2 executors
)))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
--- End diff --
"A proper" means?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50754397
--- Diff: storm-core/test/clj/org/apache/storm/security/serialization/BlowfishTupleSerializer_test.clj ---
@@ -15,7 +15,7 @@
;; limitations under the License.
(ns org.apache.storm.security.serialization.BlowfishTupleSerializer-test
(:use [clojure test]
- [org.apache.storm.util :only (exception-cause?)]
+ ;[org.apache.storm.util :only (exception-cause?)]
--- End diff --
Delete this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50893958
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -201,17 +231,28 @@
(.get_owner storm-base)
(clojurify-topology-action-options (.get_topology_action_options storm-base))
(convert-to-symbol-from-status (.get_prev_status storm-base))
+; (Utils/mapVal
+; (reify IFn (eval [this x] (clojurify-debugoptions x)))
+; (.get_component_debug storm-base)))))
--- End diff --
and here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/1043#issuecomment-175205323
First off RAT is complaining about 3 files. Please take a look into that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50897886
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1382,5 +1422,438 @@ public static TopologyInfo getTopologyInfo(String name, String asUser, Map storm
public static int toPositive(int number) {
return number & Integer.MAX_VALUE;
}
-}
+
+
+ //Everything from here on is translated from the old util.clj (storm-core/src/clj/backtype.storm/util.clj)
+
+ //Wraps an exception in a RuntimeException if needed
+ public static Exception wrapInRuntime (Exception e) {
+ if (e instanceof RuntimeException) {
+ return e;
+ } else {
+ return (new RuntimeException(e));
+ }
+ }
+
+ public static final boolean isOnWindows = "Windows_NT".equals(System.getenv("OS"));
+
+ public static final String filePathSeparator = System.getProperty("file.separator");
+
+ public static final String classPathSeparator = System.getProperty("path.separator");
+
+
+ /*
+ Returns the first item of coll for which (pred item) returns logical true.
+ Consumes sequences up to the first match, will consume the entire sequence
+ and return nil if no match is found.
+ */
+ public static Object findFirst (IPredicate pred, Collection coll) {
+ if (coll == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = coll.iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+
+ public static Object findFirst (IPredicate pred, Map map) {
+ if (map == null || pred == null) {
+ return null;
+ } else {
+ Iterator<Object> iter = map.entrySet().iterator();
+ if (iter==null || !iter.hasNext()) {
+ return null;
+ } else {
+ do {
+ Object obj = iter.next();
+ if (pred.test(obj)) {
+ return obj;
+ }
+ } while (iter.hasNext());
+ return null;
+ }
+ }
+ }
+ /*
+ Note: since the following functions are nowhere used in Storm, they were not translated:
+ dissoc-in
+ indexed
+ positions
+ assoc-conj
+ set-delta
+
+ clojurify-structure because it wouldn't make sense without clojure
+ */
+
+
+ public static String localHostname () throws UnknownHostException {
+ return _instance.localHostnameImpl();
+ }
+
+ protected String localHostnameImpl () throws UnknownHostException {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ }
+
+ private static String memoizedLocalHostnameString = null;
+
+ public static String memoizedLocalHostname () throws UnknownHostException {
+ if (memoizedLocalHostnameString == null) {
+ memoizedLocalHostnameString = localHostname();
+ }
+ return memoizedLocalHostnameString;
+ }
+
+ /*
+ checks conf for STORM_LOCAL_HOSTNAME.
+ when unconfigured, falls back to (memoized) guess by `local-hostname`.
+ */
+ public static String hostname (Map<String, Object> conf) throws UnknownHostException {
+ if (conf == null) {
+ return memoizedLocalHostname();
+ }
+ Object hostnameString = conf.get(Config.STORM_LOCAL_HOSTNAME);
+ if (hostnameString == null ) {
+ return memoizedLocalHostname();
+ }
+ if (hostnameString.equals("")) {
+ return memoizedLocalHostname();
+ }
+ return hostnameString.toString();
--- End diff --
Please just cast it to a string instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1043#discussion_r50894007
--- Diff: storm-core/src/clj/org/apache/storm/converter.clj ---
@@ -201,17 +231,28 @@
(.get_owner storm-base)
(clojurify-topology-action-options (.get_topology_action_options storm-base))
(convert-to-symbol-from-status (.get_prev_status storm-base))
+; (Utils/mapVal
+; (reify IFn (eval [this x] (clojurify-debugoptions x)))
+; (.get_component_debug storm-base)))))
(map-val clojurify-debugoptions (.get_component_debug storm-base)))))
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
+;TODO: when translating this function, you should replace the map-val with a proper for loop HERE
(defn thriftify-stats [stats]
(if stats
+; (Utils/mapVal
+; (reify IFn (eval [this x] (thriftify-executor-stats x)))
--- End diff --
and here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 1226
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/1043#issuecomment-175242193
Finished with my first pass through the code. For the most part things are looking very good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---