You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Noam Cohen <No...@acuityads.com> on 2014/10/20 16:07:18 UTC
Hadoop and HBase configuration files location (inside the topology
jar?)
Hey guys -
Our topology writes data into both HBase and HDFS. Because of that - it has to find the location of the Hadoop/HBase masters in the "hadoop-site.xml" and "hbase-site.xml" configuration files.
For non-storm applications that we run - we simply make sure to load those files from the application's classpath. That way we can run the same program both on our lab and on our production environment and have it locate the right master servers based on the configuration files.
However - we couldn't find a way to do the same for Storm. We are forced to place the files inside the topology's JAR file and hence build two different JARs - one for production and one for lab.
Is there a way to make Storm load those files from the Worker server's local disks instead of packing them up inside the topology?
Thanks!
Noam
Re: Hadoop and HBase configuration files location (inside the
topology jar?)
Posted by Paul Poulosky <pp...@yahoo-inc.com>.
It’s part of the config that you pass in to StormSubmitter when you launch the topology.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/Config.java
It’s a hashmap mapping strings to Objects (most of which are strings, some of which are lists) that allow you to control configuration items for your topology.
public static final String TOPOLOGY_CLASSPATH="topology.classpath";
FYI,
Paul
From: Noam Cohen <No...@acuityads.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Monday, October 20, 2014 at 11:26 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Hadoop and HBase configuration files location (inside the topology jar?)
Hey Paul -
I'm not familiar with this parameter, on which config file does it go?
Instead, we tried setting the "java.library.path" parameter file in "storm.yaml" to point to "/etc/hadoop/conf" and "/etc/hbase/conf" (the location of the local Hadoop/HBase config files), but it didn't work...
Noam
On 20/10/14 17:26, Paul Poulosky wrote:
Couldn’t you set topology.classpath to include the location of the file? It would have to be installed in the same location on every worker node.
From: Noam Cohen <No...@acuityads.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Monday, October 20, 2014 at 9:07 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Hadoop and HBase configuration files location (inside the topology jar?)
Hey guys -
Our topology writes data into both HBase and HDFS. Because of that - it has to find the location of the Hadoop/HBase masters in the "hadoop-site.xml" and "hbase-site.xml" configuration files.
For non-storm applications that we run - we simply make sure to load those files from the application's classpath. That way we can run the same program both on our lab and on our production environment and have it locate the right master servers based on the configuration files.
However - we couldn't find a way to do the same for Storm. We are forced to place the files inside the topology's JAR file and hence build two different JARs - one for production and one for lab.
Is there a way to make Storm load those files from the Worker server's local disks instead of packing them up inside the topology?
Thanks!
Noam
Re: Hadoop and HBase configuration files location (inside the
topology jar?)
Posted by Noam Cohen <No...@acuityads.com>.
Hey Paul -
I'm not familiar with this parameter, on which config file does it go?
Instead, we tried setting the "java.library.path" parameter file in "storm.yaml" to point to "/etc/hadoop/conf" and "/etc/hbase/conf" (the location of the local Hadoop/HBase config files), but it didn't work...
Noam
On 20/10/14 17:26, Paul Poulosky wrote:
Couldn’t you set topology.classpath to include the location of the file? It would have to be installed in the same location on every worker node.
From: Noam Cohen <No...@acuityads.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Monday, October 20, 2014 at 9:07 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Hadoop and HBase configuration files location (inside the topology jar?)
Hey guys -
Our topology writes data into both HBase and HDFS. Because of that - it has to find the location of the Hadoop/HBase masters in the "hadoop-site.xml" and "hbase-site.xml" configuration files.
For non-storm applications that we run - we simply make sure to load those files from the application's classpath. That way we can run the same program both on our lab and on our production environment and have it locate the right master servers based on the configuration files.
However - we couldn't find a way to do the same for Storm. We are forced to place the files inside the topology's JAR file and hence build two different JARs - one for production and one for lab.
Is there a way to make Storm load those files from the Worker server's local disks instead of packing them up inside the topology?
Thanks!
Noam
Re: Hadoop and HBase configuration files location (inside the
topology jar?)
Posted by Noam Cohen <No...@acuityads.com>.
Thanks Samit! I'll try it and let you know if it works.
On 21/10/14 03:37, Samit Sasan wrote:
Place the config files in storm/conf folder and it would be on your typology classpath when your topology runs. Catch is that the config had to be deployed to all your supervisor machines.
-Samit
On Oct 20, 2014 7:56 PM, "Paul Poulosky" <pp...@yahoo-inc.com>> wrote:
Couldn’t you set topology.classpath to include the location of the file? It would have to be installed in the same location on every worker node.
From: Noam Cohen <No...@acuityads.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Monday, October 20, 2014 at 9:07 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Hadoop and HBase configuration files location (inside the topology jar?)
Hey guys -
Our topology writes data into both HBase and HDFS. Because of that - it has to find the location of the Hadoop/HBase masters in the "hadoop-site.xml" and "hbase-site.xml" configuration files.
For non-storm applications that we run - we simply make sure to load those files from the application's classpath. That way we can run the same program both on our lab and on our production environment and have it locate the right master servers based on the configuration files.
However - we couldn't find a way to do the same for Storm. We are forced to place the files inside the topology's JAR file and hence build two different JARs - one for production and one for lab.
Is there a way to make Storm load those files from the Worker server's local disks instead of packing them up inside the topology?
Thanks!
Noam
Re: Hadoop and HBase configuration files location (inside the
topology jar?)
Posted by Samit Sasan <sa...@gmail.com>.
Place the config files in storm/conf folder and it would be on your
typology classpath when your topology runs. Catch is that the config had to
be deployed to all your supervisor machines.
-Samit
On Oct 20, 2014 7:56 PM, "Paul Poulosky" <pp...@yahoo-inc.com> wrote:
> Couldn’t you set topology.classpath to include the location of the file?
> It would have to be installed in the same location on every worker node.
>
> From: Noam Cohen <No...@acuityads.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Monday, October 20, 2014 at 9:07 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Hadoop and HBase configuration files location (inside the
> topology jar?)
>
> Hey guys -
>
> Our topology writes data into both HBase and HDFS. Because of that - it
> has to find the location of the Hadoop/HBase masters in the
> "hadoop-site.xml" and "hbase-site.xml" configuration files.
>
> For non-storm applications that we run - we simply make sure to load those
> files from the application's classpath. That way we can run the same
> program both on our lab and on our production environment and have it
> locate the right master servers based on the configuration files.
>
> However - we couldn't find a way to do the same for Storm. We are forced
> to place the files inside the topology's JAR file and hence build two
> different JARs - one for production and one for lab.
>
> Is there a way to make Storm load those files from the Worker server's
> local disks instead of packing them up inside the topology?
>
> Thanks!
> Noam
>
Re: Hadoop and HBase configuration files location (inside the
topology jar?)
Posted by Paul Poulosky <pp...@yahoo-inc.com>.
Couldn’t you set topology.classpath to include the location of the file? It would have to be installed in the same location on every worker node.
From: Noam Cohen <No...@acuityads.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Monday, October 20, 2014 at 9:07 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Hadoop and HBase configuration files location (inside the topology jar?)
Hey guys -
Our topology writes data into both HBase and HDFS. Because of that - it has to find the location of the Hadoop/HBase masters in the "hadoop-site.xml" and "hbase-site.xml" configuration files.
For non-storm applications that we run - we simply make sure to load those files from the application's classpath. That way we can run the same program both on our lab and on our production environment and have it locate the right master servers based on the configuration files.
However - we couldn't find a way to do the same for Storm. We are forced to place the files inside the topology's JAR file and hence build two different JARs - one for production and one for lab.
Is there a way to make Storm load those files from the Worker server's local disks instead of packing them up inside the topology?
Thanks!
Noam