You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by harshach <gi...@git.apache.org> on 2015/03/31 20:42:19 UTC
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
GitHub user harshach opened a pull request:
https://github.com/apache/storm/pull/495
STORM-188. Allow user to specifiy full configuration path when running storm command
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harshach/incubator-storm STORM-188
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/495.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #495
----
commit 9a2c4129b145b901f320238ea36a282ff7f44d0c
Author: Sriharsha Chintalapani <ma...@harsha.io>
Date: 2015-03-31T18:37:30Z
STORM-188. Allow user to specifiy full configuration path when running storm command.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27561927
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
try {
- HashSet<URL> resources = new HashSet<URL>(findResources(name));
- if(resources.isEmpty()) {
- if(mustExist) throw new RuntimeException("Could not find config file on classpath " + name);
- else return new HashMap();
- }
- if(resources.size() > 1) {
- throw new RuntimeException("Found multiple " + name + " resources. You're probably bundling the Storm jars with your topology jar. "
- + resources);
- }
- URL resource = resources.iterator().next();
- Yaml yaml = new Yaml(new SafeConstructor());
- Map ret = null;
- InputStream input = resource.openStream();
- try {
- ret = (Map) yaml.load(new InputStreamReader(input));
- } finally {
- input.close();
+ in = getConfigFileInputStream(name);
+ if (null != in) {
+ Yaml yaml = new Yaml(new SafeConstructor());
+ Map ret = (Map) yaml.load(new InputStreamReader(in));
+ if (null != ret) {
+ return new HashMap(ret);
+ } else {
+ confFileEmpty = true;
+ }
}
- if(ret==null) ret = new HashMap();
-
- return new HashMap(ret);
-
+ if (mustExist) {
+ if(confFileEmpty)
+ throw new RuntimeException("Config file " + name + " doesn't have any valid storm configs");
+ else
+ throw new RuntimeException("Could not find config file on classpath " + name);
+ } else {
+ return new HashMap();
+ }
} catch (IOException e) {
throw new RuntimeException(e);
+ } finally {
+ if (null != in) {
+ try {
+ in.close();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
}
}
+ private static InputStream getConfigFileInputStream(String configFilePath)
+ throws IOException {
+ if (null == configFilePath) {
+ throw new IOException(
+ "Could not find config file, name not specified");
+ }
+
+ HashSet<URL> resources = new HashSet<URL>(findResources(configFilePath));
+ if (resources.isEmpty()) {
+ File configFile = new File(configFilePath);
+ if (configFile.exists()) {
+ return new FileInputStream(configFile);
+ }
--- End diff --
@lazyval
mustExist flag will control whether we should fail the method.
···public static Map findAndReadConfigFile(String name, boolean mustExist) ···
When we cannot find a resource, we will return null; then we check whether mustExist flag is enabled to determine whether we should throw RuntimeException.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by lazyval <gi...@git.apache.org>.
Github user lazyval commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27561516
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
try {
- HashSet<URL> resources = new HashSet<URL>(findResources(name));
- if(resources.isEmpty()) {
- if(mustExist) throw new RuntimeException("Could not find config file on classpath " + name);
- else return new HashMap();
- }
- if(resources.size() > 1) {
- throw new RuntimeException("Found multiple " + name + " resources. You're probably bundling the Storm jars with your topology jar. "
- + resources);
- }
- URL resource = resources.iterator().next();
- Yaml yaml = new Yaml(new SafeConstructor());
- Map ret = null;
- InputStream input = resource.openStream();
- try {
- ret = (Map) yaml.load(new InputStreamReader(input));
- } finally {
- input.close();
+ in = getConfigFileInputStream(name);
+ if (null != in) {
+ Yaml yaml = new Yaml(new SafeConstructor());
+ Map ret = (Map) yaml.load(new InputStreamReader(in));
+ if (null != ret) {
+ return new HashMap(ret);
+ } else {
+ confFileEmpty = true;
+ }
}
- if(ret==null) ret = new HashMap();
-
- return new HashMap(ret);
-
+ if (mustExist) {
+ if(confFileEmpty)
+ throw new RuntimeException("Config file " + name + " doesn't have any valid storm configs");
+ else
+ throw new RuntimeException("Could not find config file on classpath " + name);
+ } else {
+ return new HashMap();
+ }
} catch (IOException e) {
throw new RuntimeException(e);
+ } finally {
+ if (null != in) {
+ try {
+ in.close();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
}
}
+ private static InputStream getConfigFileInputStream(String configFilePath)
+ throws IOException {
+ if (null == configFilePath) {
+ throw new IOException(
+ "Could not find config file, name not specified");
+ }
+
+ HashSet<URL> resources = new HashSet<URL>(findResources(configFilePath));
+ if (resources.isEmpty()) {
+ File configFile = new File(configFilePath);
+ if (configFile.exists()) {
+ return new FileInputStream(configFile);
+ }
--- End diff --
Why not throw IOException, when path contains no actual file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by lazyval <gi...@git.apache.org>.
Github user lazyval commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27571878
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
--- End diff --
There are two variables types in Java: primitive (like boolean, int, long, short and so on), and their boxed counterparts (Boolean, Integer, Long, ...), which are seen as reference types. It's minor, but unless you're working with collections, there is no need to use later one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/495
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27571219
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
--- End diff --
@lazyval what you mean by boxed value?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on the pull request:
https://github.com/apache/storm/pull/495#issuecomment-88295467
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27584483
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
try {
- HashSet<URL> resources = new HashSet<URL>(findResources(name));
- if(resources.isEmpty()) {
- if(mustExist) throw new RuntimeException("Could not find config file on classpath " + name);
- else return new HashMap();
- }
- if(resources.size() > 1) {
- throw new RuntimeException("Found multiple " + name + " resources. You're probably bundling the Storm jars with your topology jar. "
- + resources);
- }
- URL resource = resources.iterator().next();
- Yaml yaml = new Yaml(new SafeConstructor());
- Map ret = null;
- InputStream input = resource.openStream();
- try {
- ret = (Map) yaml.load(new InputStreamReader(input));
- } finally {
- input.close();
+ in = getConfigFileInputStream(name);
+ if (null != in) {
+ Yaml yaml = new Yaml(new SafeConstructor());
+ Map ret = (Map) yaml.load(new InputStreamReader(in));
+ if (null != ret) {
+ return new HashMap(ret);
+ } else {
+ confFileEmpty = true;
+ }
}
- if(ret==null) ret = new HashMap();
-
- return new HashMap(ret);
-
+ if (mustExist) {
+ if(confFileEmpty)
+ throw new RuntimeException("Config file " + name + " doesn't have any valid storm configs");
+ else
+ throw new RuntimeException("Could not find config file on classpath " + name);
+ } else {
+ return new HashMap();
+ }
} catch (IOException e) {
throw new RuntimeException(e);
+ } finally {
+ if (null != in) {
+ try {
+ in.close();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
}
}
+ private static InputStream getConfigFileInputStream(String configFilePath)
+ throws IOException {
+ if (null == configFilePath) {
+ throw new IOException(
+ "Could not find config file, name not specified");
+ }
+
+ HashSet<URL> resources = new HashSet<URL>(findResources(configFilePath));
+ if (resources.isEmpty()) {
+ File configFile = new File(configFilePath);
+ if (configFile.exists()) {
+ return new FileInputStream(configFile);
+ }
--- End diff --
@lazyval there are cases where storm wants to use defaults.yaml if there is not storm.yaml. If one wants to do a quick test they can just run the storm cluster by not having any storm.yaml . Hence the reason we have mustexist as a flag to indicate if we want to throw an error or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27572921
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
--- End diff --
@lazyval I know about primitives and objects in java . Yes makes sense to use a boolean.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by lazyval <gi...@git.apache.org>.
Github user lazyval commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27562061
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
--- End diff --
no need in boxed value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by lazyval <gi...@git.apache.org>.
Github user lazyval commented on the pull request:
https://github.com/apache/storm/pull/495#issuecomment-88550651
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by lazyval <gi...@git.apache.org>.
Github user lazyval commented on a diff in the pull request:
https://github.com/apache/storm/pull/495#discussion_r27562802
--- Diff: storm-core/src/jvm/backtype/storm/utils/Utils.java ---
@@ -135,35 +137,68 @@ public static void sleep(long millis) {
}
public static Map findAndReadConfigFile(String name, boolean mustExist) {
+ InputStream in = null;
+ Boolean confFileEmpty = false;
try {
- HashSet<URL> resources = new HashSet<URL>(findResources(name));
- if(resources.isEmpty()) {
- if(mustExist) throw new RuntimeException("Could not find config file on classpath " + name);
- else return new HashMap();
- }
- if(resources.size() > 1) {
- throw new RuntimeException("Found multiple " + name + " resources. You're probably bundling the Storm jars with your topology jar. "
- + resources);
- }
- URL resource = resources.iterator().next();
- Yaml yaml = new Yaml(new SafeConstructor());
- Map ret = null;
- InputStream input = resource.openStream();
- try {
- ret = (Map) yaml.load(new InputStreamReader(input));
- } finally {
- input.close();
+ in = getConfigFileInputStream(name);
+ if (null != in) {
+ Yaml yaml = new Yaml(new SafeConstructor());
+ Map ret = (Map) yaml.load(new InputStreamReader(in));
+ if (null != ret) {
+ return new HashMap(ret);
+ } else {
+ confFileEmpty = true;
+ }
}
- if(ret==null) ret = new HashMap();
-
- return new HashMap(ret);
-
+ if (mustExist) {
+ if(confFileEmpty)
+ throw new RuntimeException("Config file " + name + " doesn't have any valid storm configs");
+ else
+ throw new RuntimeException("Could not find config file on classpath " + name);
+ } else {
+ return new HashMap();
+ }
} catch (IOException e) {
throw new RuntimeException(e);
+ } finally {
+ if (null != in) {
+ try {
+ in.close();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
}
}
+ private static InputStream getConfigFileInputStream(String configFilePath)
+ throws IOException {
+ if (null == configFilePath) {
+ throw new IOException(
+ "Could not find config file, name not specified");
+ }
+
+ HashSet<URL> resources = new HashSet<URL>(findResources(configFilePath));
+ if (resources.isEmpty()) {
+ File configFile = new File(configFilePath);
+ if (configFile.exists()) {
+ return new FileInputStream(configFile);
+ }
--- End diff --
Yeah, I just a bit concerned that flow is driven by both exceptions and flag/null.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/495#issuecomment-89409291
I think overall it is good and I am +1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/495#issuecomment-88204014
This PR is taken from https://github.com/apache/storm/pull/120#issuecomment-69435220
@clockfly @revans2 can you please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-188. Allow user to specifiy full configu...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/495#issuecomment-88537973
@clockfly @lazyval updated the patch for fixing the last comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---