configuration file of storm cluster

[Nageswara Rao <na...@gmail.com>]

Hi All,

I have set-up storm cluster with one nimbus server and three supervisors.
Published three topologies which does different calculations as follows

- Topology1 : Reads raw data from MongoDB, do some calculations and finally
store back the result.

- Topology2 : Reads the result of topology1 and do some calculations and
publish results to a queue

- Topology3 : Consumes output of topology2 from the queue, calls a REST
Service, get reply from REST service, update result in MongoDB collection,
finally send an email.

I relatively new to storm, hence looking for an expert advice on the
following questions

1. Currently configuration to connect MongoDB, MySql, Mq, REST urls are
hard-coded in java file. The file is published to cluster along with the
topology. I found it is not good practice to customize source files for
each customer.

Is there a way to externalize all configurations, for example a
config.json, that can be referred by all elements of topology?

2. Wanted to extract logs at each stage [Spouts and Bolts], how do I
configure log4j.xml and place is it in the runtime of cluster?

3. Is it right to execute blocking call like REST call from a bolt?

Any help would be much appreciated.

-- 
Thanks,
Nageswara Rao.V

*"The LORD reigns"*

Re: configuration file of storm cluster

[Rajasekhar <ra...@gmail.com>]

Hi,

For question 1:
You can write such configs in property file and pass file as input to
topology.
Read the file using buffered reader (in main method) and pass the
parameters as needed.
Note : Predefine the layout of the property file.

For question 2:
Add log4j in Maven and create a object of it in the class as a field.
Write the logs where ever is needed. There is debug option for storm
topology but do not enable it for production purpose.
It really kills the performance and you will never catch with the input
feeds.




On Mon, Mar 21, 2016 at 6:30 AM, Nageswara Rao <na...@gmail.com>
wrote:

> Hi All,
>
> I have set-up storm cluster with one nimbus server and three supervisors.
> Published three topologies which does different calculations as follows
>
> - Topology1 : Reads raw data from MongoDB, do some calculations and
> finally store back the result.
>
> - Topology2 : Reads the result of topology1 and do some calculations and
> publish results to a queue
>
> - Topology3 : Consumes output of topology2 from the queue, calls a REST
> Service, get reply from REST service, update result in MongoDB collection,
> finally send an email.
>
> I relatively new to storm, hence looking for an expert advice on the
> following questions
>
> 1. Currently configuration to connect MongoDB, MySql, Mq, REST urls are
> hard-coded in java file. The file is published to cluster along with the
> topology. I found it is not good practice to customize source files for
> each customer.
>
> Is there a way to externalize all configurations, for example a
> config.json, that can be referred by all elements of topology?
>
> 2. Wanted to extract logs at each stage [Spouts and Bolts], how do I
> configure log4j.xml and place is it in the runtime of cluster?
>
> 3. Is it right to execute blocking call like REST call from a bolt?
>
> Any help would be much appreciated.
>
> --
> Thanks,
> Nageswara Rao.V
>
> *"The LORD reigns"*
>



-- 
Thanks & Regards
Rajasekhar