You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Garcia-Contractor, Joseph (CORP)" <Jo...@ADP.com> on 2015/09/30 17:05:42 UTC

Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use -c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with -c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers="[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?

----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

RE: Storm and enviroment variables

Posted by "Garcia-Contractor, Joseph (CORP)" <Jo...@ADP.com>.
Storm version is 0.9.5

So I have been moving along with the storm stuff and I have decided to give “–config” a try.  I need to know if this is a bug or I am doing something wrong.

And it takes the –config part just fine as an argument however the java part is puking on it and failing, see below:



ls -ald /etc/storm/conf
lrwxrwxrwx 1 root root 28 Oct 27 14:32 /etc/storm/conf -> /etc/alternatives/storm-conf

ls -al /etc/alternatives/storm-conf
lrwxrwxrwx 1 root root 41 Oct 27 14:32 /etc/alternatives/storm-conf -> /opt/cloudera/parcels/STORM-0.9.5-u1/conf

ls -ald /opt/cloudera/parcels/STORM-0.9.5-u1/conf
drwxrwxr-x 2 storm storm 4096 Nov  2 12:01 /opt/cloudera/parcels/STORM-0.9.5-u1/conf

               ls -al /opt/cloudera/parcels/STORM-0.9.5-u1/conf
total 16
drwxrwxr-x  2 storm storm 4096 Nov  2 12:01 .
drwxrwxr-x 11 root  root  4096 Oct 26 12:00 ..
-rwxrwxr-x  1 root  root  1128 Sep 17 14:48 storm_env.ini
-rwxrwxr-x  1 storm storm  422 Nov  2 11:42 storm_nimbus.yaml

ls -al /etc/storm/conf/
total 16
drwxrwxr-x  2 storm storm 4096 Nov  2 12:01 .
drwxrwxr-x 11 root  root  4096 Oct 26 12:00 ..
-rwxrwxr-x  1 root  root  1128 Sep 17 14:48 storm_env.ini
-rwxrwxr-x  1 storm storm  422 Nov  2 11:42 storm_nimbus.yaml

storm nimbus --config /etc/storm/conf/storm_nimbus.yaml
Exception in thread "main" java.lang.ExceptionInInitializerError
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:195)
               at backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)
               at backtype.storm.config__init.load(Unknown Source)
               at backtype.storm.config__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.core$load_one.invoke(core.clj:5336)
               at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
               at clojure.core$load_lib.doInvoke(core.clj:5374)
               at clojure.lang.RestFn.applyTo(RestFn.java:142)
               at clojure.core$apply.invoke(core.clj:619)
               at clojure.core$load_libs.doInvoke(core.clj:5417)
               at clojure.lang.RestFn.applyTo(RestFn.java:137)
               at clojure.core$apply.invoke(core.clj:621)
               at clojure.core$use.doInvoke(core.clj:5507)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)
               at backtype.storm.command.config_value__init.load(Unknown Source)
               at backtype.storm.command.config_value__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.lang.Var.invoke(Var.java:415)
               at backtype.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath /etc/storm/conf/storm_nimbus.yaml
               at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
               at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
               at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
               ... 36 more
Exception in thread "main" java.lang.ExceptionInInitializerError
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:195)
               at backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)
               at backtype.storm.config__init.load(Unknown Source)
               at backtype.storm.config__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.core$load_one.invoke(core.clj:5336)
               at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
               at clojure.core$load_lib.doInvoke(core.clj:5374)
               at clojure.lang.RestFn.applyTo(RestFn.java:142)
               at clojure.core$apply.invoke(core.clj:619)
               at clojure.core$load_libs.doInvoke(core.clj:5417)
               at clojure.lang.RestFn.applyTo(RestFn.java:137)
               at clojure.core$apply.invoke(core.clj:621)
               at clojure.core$use.doInvoke(core.clj:5507)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)
               at backtype.storm.command.config_value__init.load(Unknown Source)
               at backtype.storm.command.config_value__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.lang.Var.invoke(Var.java:415)
               at backtype.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath /etc/storm/conf/storm_nimbus.yaml
               at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
               at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
               at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
               ... 36 more
Exception in thread "main" java.lang.ExceptionInInitializerError
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:195)
               at backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)
               at backtype.storm.config__init.load(Unknown Source)
               at backtype.storm.config__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.core$load_one.invoke(core.clj:5336)
               at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
               at clojure.core$load_lib.doInvoke(core.clj:5374)
               at clojure.lang.RestFn.applyTo(RestFn.java:142)
               at clojure.core$apply.invoke(core.clj:619)
               at clojure.core$load_libs.doInvoke(core.clj:5417)
               at clojure.lang.RestFn.applyTo(RestFn.java:137)
               at clojure.core$apply.invoke(core.clj:621)
               at clojure.core$use.doInvoke(core.clj:5507)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)
               at backtype.storm.command.config_value__init.load(Unknown Source)
               at backtype.storm.command.config_value__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.lang.Var.invoke(Var.java:415)
               at backtype.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath /etc/storm/conf/storm_nimbus.yaml
               at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
               at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
               at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
               ... 36 more
Running: java -server -Dstorm.options= -Dstorm.home=/opt/cloudera/parcels/STORM-0.9.5-u1 -Dstorm.log.dir= -Djava.library.path= -Dstorm.conf.file=/etc/storm/conf/storm_nimbus.yaml -cp /opt/cloudera/parcels/STORM-0.9.5-u1/lib/jgrapht-core-0.9.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/tools.cli-0.2.4.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-codec-1.6.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jline-2.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/asm-4.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/tools.macro-0.1.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/math.numeric-tower-0.0.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/joda-time-2.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/disruptor-2.10.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/objenesis-1.2.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-io-2.4.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clout-1.0.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clojure-1.5.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/chill-java-0.3.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-servlet-0.3.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/core.incubator-0.1.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-core-1.1.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/storm-core-0.9.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/log4j-over-slf4j-1.6.6.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/compojure-1.1.3.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/snakeyaml-1.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clj-time-0.4.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-lang-2.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-exec-1.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/kryo-2.21.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/clj-stacktrace-0.2.2.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-fileupload-1.2.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/logback-core-1.0.13.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/hiccup-0.3.6.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/json-simple-1.1.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/reflectasm-1.07-shaded.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/logback-classic-1.0.13.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/tools.logging-0.2.3.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-jetty-adapter-0.3.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/minlog-1.2.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/ring-devel-0.3.11.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/jetty-6.1.26.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/carbonite-1.4.0.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/STORM-0.9.5-u1/conf -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/opt/cloudera/parcels/STORM-0.9.5-u1/logback/cluster.xml backtype.storm.daemon.nimbus
Exception in thread "main" java.lang.ExceptionInInitializerError
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:195)
               at backtype.storm.util$loading__4910__auto__.invoke(util.clj:17)
               at backtype.storm.util__init.load(Unknown Source)
               at backtype.storm.util__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.core$load_one.invoke(core.clj:5336)
               at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
               at clojure.core$load_lib.doInvoke(core.clj:5374)
               at clojure.lang.RestFn.applyTo(RestFn.java:142)
               at clojure.core$apply.invoke(core.clj:619)
               at clojure.core$load_libs.doInvoke(core.clj:5417)
               at clojure.lang.RestFn.applyTo(RestFn.java:137)
               at clojure.core$apply.invoke(core.clj:621)
               at clojure.core$use.doInvoke(core.clj:5507)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at backtype.storm.scheduler.DefaultScheduler$loading__4910__auto__.invoke(DefaultScheduler.clj:16)
               at backtype.storm.scheduler.DefaultScheduler__init.load(Unknown Source)
               at backtype.storm.scheduler.DefaultScheduler__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.core$load_one.invoke(core.clj:5336)
               at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
               at clojure.core$load_lib.doInvoke(core.clj:5374)
               at clojure.lang.RestFn.applyTo(RestFn.java:142)
               at clojure.core$apply.invoke(core.clj:619)
               at clojure.core$load_libs.doInvoke(core.clj:5413)
               at clojure.lang.RestFn.applyTo(RestFn.java:137)
               at clojure.core$apply.invoke(core.clj:621)
               at clojure.core$use.doInvoke(core.clj:5507)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at backtype.storm.daemon.nimbus$loading__4910__auto__.invoke(nimbus.clj:16)
               at backtype.storm.daemon.nimbus__init.load(Unknown Source)
               at backtype.storm.daemon.nimbus__init.<clinit>(Unknown Source)
               at java.lang.Class.forName0(Native Method)
               at java.lang.Class.forName(Class.java:278)
               at clojure.lang.RT.loadClassForName(RT.java:2098)
               at clojure.lang.RT.load(RT.java:430)
               at clojure.lang.RT.load(RT.java:411)
               at clojure.core$load$fn__5018.invoke(core.clj:5530)
               at clojure.core$load.doInvoke(core.clj:5529)
               at clojure.lang.RestFn.invoke(RestFn.java:408)
               at clojure.lang.Var.invoke(Var.java:415)
               at backtype.storm.daemon.nimbus.<clinit>(Unknown Source)
Caused by: java.lang.RuntimeException: Could not find config file on classpath /etc/storm/conf/storm_nimbus.yaml
               at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
               at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
               at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
               ... 57 more
From: Garcia-Contractor, Joseph (CORP) [mailto:Joseph.Garcia-Contractor@ADP.com]
Sent: Thursday, October 08, 2015 3:34 PM
To: user@storm.apache.org
Subject: RE: Storm and enviroment variables

Yeap… it certainly does… I missed that.  Thanks for pointing it out!   ☺


From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Thursday, October 08, 2015 12:51 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

Glad that it fixed your issue. The python script has the —config option. Check https://github.com/apache/storm/blob/master/bin/storm.py#L632 out.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Thursday, October 8, 2015 at 6:52 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

I did not see a –config option in the storm python start script… I did however find in there that if I set the STORM_CONF_DIR environment variable it would look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you try —config <pathToYourYamlFile> command line option while running storm command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

RE: Storm and enviroment variables

Posted by "Garcia-Contractor, Joseph (CORP)" <Jo...@ADP.com>.
Yeap… it certainly does… I missed that.  Thanks for pointing it out!   ☺


From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Thursday, October 08, 2015 12:51 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

Glad that it fixed your issue. The python script has the —config option. Check https://github.com/apache/storm/blob/master/bin/storm.py#L632 out.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Thursday, October 8, 2015 at 6:52 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

I did not see a –config option in the storm python start script… I did however find in there that if I set the STORM_CONF_DIR environment variable it would look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you try —config <pathToYourYamlFile> command line option while running storm command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Re: Storm and enviroment variables

Posted by Priyank Shah <ps...@hortonworks.com>.
Glad that it fixed your issue. The python script has the —config option. Check https://github.com/apache/storm/blob/master/bin/storm.py#L632 out.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Thursday, October 8, 2015 at 6:52 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

I did not see a –config option in the storm python start script… I did however find in there that if I set the STORM_CONF_DIR environment variable it would look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you try —config <pathToYourYamlFile> command line option while running storm command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

RE: Storm and enviroment variables

Posted by "Garcia-Contractor, Joseph (CORP)" <Jo...@ADP.com>.
I did not see a –config option in the storm python start script… I did however find in there that if I set the STORM_CONF_DIR environment variable it would look for storm.yaml in that dir and this fixed my issue.

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, October 07, 2015 5:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

I think you can specify a config file that is other than the default. Can you try —config <pathToYourYamlFile> command line option while running storm command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Re: Storm and enviroment variables

Posted by Priyank Shah <ps...@hortonworks.com>.
I think you can specify a config file that is other than the default. Can you try —config <pathToYourYamlFile> command line option while running storm command ?

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, October 7, 2015 at 2:15 PM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: RE: Storm and enviroment variables

Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

RE: Storm and enviroment variables

Posted by "Garcia-Contractor, Joseph (CORP)" <Jo...@ADP.com>.
Ok so an additional question here…

Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm/conf/storm.yaml?

Joe

From: Priyank Shah [mailto:pshah@hortonworks.com]
Sent: Wednesday, September 30, 2015 2:31 PM
To: user@storm.apache.org
Subject: Re: Storm and enviroment variables

Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Re: Storm and enviroment variables

Posted by Priyank Shah <ps...@hortonworks.com>.
Hi Joseph,

I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings.  But not sure about other types. In the code its trying to parse it as a json. You can give it a try.

I don’t know about logback file . But you can configure your log directory using –c storm.log.dir=/var/log/ option.

From: "Garcia-Contractor, Joseph (CORP)"
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Wednesday, September 30, 2015 at 8:05 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Storm and enviroment variables

Hi All,

Another question.

Does storm accept environment variables that can be used in place of /etc/storm/conf/storm.yaml?

If so, what is the format of the environment variable it?  IE.: storm.local.dir in storm.yaml would be STORM_LOCAL_DIR in the environment.

I see that I can use –c at the command line but I would prefer using environment variable without having to build the command line args.  If I need to go with –c, what would the arguments still be in yaml format?  IE.: storm.zookeeper.servers=”[ zk_host1.foo, zk_host2.foo, zk_host3.foo ]

Is there a way to override the where the logback file is located and read from?
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.