You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kushan Maskey <ku...@mmillerassociates.com> on 2014/08/20 23:20:18 UTC
Reading config.priperties file
I am quite new to cluster environment and so is the entire concept of
sotrm/kafka. I have everything running as it should. But I am struggling to
read config.properties file that has config information to
Kafka/Cassandra/Solr/Couch databases. I created a condif.properties file
with all these information. I load the config file on the topology. I set
the properties as a static variable in a class. Bolts call CassandraClient
class that I wrote to load any data that comes to KafkaSpout.
CassandraClient gets the cassandra host and other information from the
properties file.
This works perfectly fine locally. But when I deployed it on the server,
all these config variables are null, meaning that cassandra host and other
information are all null. If anyone has any idea how to tackle this would
be really great. Thanks.
--
Kushan Maskey
Re: Reading config.priperties file
Posted by Anand Nalya <an...@gmail.com>.
Hi Kushan,
A slight variation on second approach suggested by Parth is to read the
properties file in your driver class on the gateway and copy all the
properties to the backtype.storm.Config object. In this way, the properties
will be available in the prepare/open method of your bolt/spout.
Regards,
Anand
On 21 August 2014 03:51, Parth Brahmbhatt <pb...@hortonworks.com>
wrote:
> Are you packing the config file in the jar? Does the config file get
> loaded on the gateway , where you run storm command, or its suppose to be
> loaded as part of spout’s/bolt's prepare method? In the former case you
> need to ensure that your property file is part of your jar file. If you are
> using maven you can do so by adding the following to your build target:
> <resources>
> <resource>
> <directory>src/resource</directory>
> </resource>
> </resources>
>
> and then in your code you can get a handle on the config file by
>
>
> SomeClass.class.getClassLoader().getResourceAsStream(“yourconfig.properties")
>
>
> The other way would be to just read the config file at the gateway, read
> the properties and set the property value as instance variables in the
> appropriate bolt and spout object. Ensure that the instance variables are
> not marked as transient.
>
> Thanks
> Parth
>
> If the config file is read and loaded at the gateway then are you storing
> On Aug 20, 2014, at 2:50 PM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
> I pass the config file as an argument to the Topology.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Reading config.priperties file
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Are you packing the config file in the jar? Does the config file get loaded on the gateway , where you run storm command, or its suppose to be loaded as part of spout’s/bolt's prepare method? In the former case you need to ensure that your property file is part of your jar file. If you are using maven you can do so by adding the following to your build target:
<resources>
<resource>
<directory>src/resource</directory>
</resource>
</resources>
and then in your code you can get a handle on the config file by
SomeClass.class.getClassLoader().getResourceAsStream(“yourconfig.properties")
The other way would be to just read the config file at the gateway, read the properties and set the property value as instance variables in the appropriate bolt and spout object. Ensure that the instance variables are not marked as transient.
Thanks
Parth
If the config file is read and loaded at the gateway then are you storing
On Aug 20, 2014, at 2:50 PM, Kushan Maskey <ku...@mmillerassociates.com> wrote:
> I pass the config file as an argument to the Topology.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Reading config.priperties file
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
I pass the config file as an argument to the Topology.
--
Kushan Maskey
817.403.7500
On Wed, Aug 20, 2014 at 4:38 PM, Harsha <st...@harsha.io> wrote:
> Kushan,
> I guess its not able to find the config.properties file when you
> deploy the topology. How are you packaging it. One way I think of is to
> pass it as part of resources and where you calling properties.load() in
> your topology.
> -Harsha
>
>
> On Wed, Aug 20, 2014, at 02:20 PM, Kushan Maskey wrote:
>
> I am quite new to cluster environment and so is the entire concept of
> sotrm/kafka. I have everything running as it should. But I am struggling to
> read config.properties file that has config information to
> Kafka/Cassandra/Solr/Couch databases. I created a condif.properties file
> with all these information. I load the config file on the topology. I set
> the properties as a static variable in a class. Bolts call CassandraClient
> class that I wrote to load any data that comes to KafkaSpout.
> CassandraClient gets the cassandra host and other information from the
> properties file.
>
> This works perfectly fine locally. But when I deployed it on the server,
> all these config variables are null, meaning that cassandra host and other
> information are all null. If anyone has any idea how to tackle this would
> be really great. Thanks.
>
> --
> Kushan Maskey
>
>
>
Re: Reading config.priperties file
Posted by Harsha <st...@harsha.io>.
Kushan,
I guess its not able to find the config.properties file
when you deploy the topology. How are you packaging it. One way
I think of is to pass it as part of resources and where you
calling properties.load() in your topology.
-Harsha
On Wed, Aug 20, 2014, at 02:20 PM, Kushan Maskey wrote:
I am quite new to cluster environment and so is the entire
concept of sotrm/kafka. I have everything running as it should.
But I am struggling to read config.properties file that has
config information to Kafka/Cassandra/Solr/Couch databases. I
created a condif.properties file with all these information. I
load the config file on the topology. I set the properties as a
static variable in a class. Bolts call CassandraClient class
that I wrote to load any data that comes to KafkaSpout.
CassandraClient gets the cassandra host and other information
from the properties file.
This works perfectly fine locally. But when I deployed it on
the server, all these config variables are null, meaning that
cassandra host and other information are all null. If anyone
has any idea how to tackle this would be really great. Thanks.
--
Kushan Maskey