Where to maintain the list of KafkaSpouts' ZkHosts?

[I PVP <ip...@hotmail.com>]

What is the recommended approach to maintain the list of ZkHosts on a central “place" in which all KafkaSpouts can call it and it easy to updated  instead of having it  hard coded inside each the Topology?

Thanks

--
IPVP

Re: Where to maintain the list of KafkaSpouts' ZkHosts?

[Lohith Samaga M <Lo...@mphasis.com>]

Hi
You could try:
1 Hdfs location
2 Mounted shared drive
3 metadata DB like mysql
4 simple http server

I used a database

HTH

Lohith

Sent from my Sony Xperia™ smartphone


---- I PVP wrote ----

Is there a way to read from a .properties file that is located outside the Topology fat jar ?

Would still not be ideal to update the .properties file on each machine, but it is better than having to update the  ZkHosts inside the topology code.




From: Alexander T <mi...@gmail.com>
Reply: user@storm.apache.org <us...@storm.apache.org>
Date: March 24, 2016 at 5:05:52 AM
To: user@storm.apache.org <us...@storm.apache.org>
Subject:  Re: Where to maintain the list of KafkaSpouts' ZkHosts?


We also have this problem for Kinesis. I wondered if there is an API for spouts to read out the current zk hosts from Storm.

On Mar 24, 2016 01:08, "I PVP" <ip...@hotmail.com>> wrote:
What is the recommended approach to maintain the list of ZkHosts on a central “place" in which all KafkaSpouts can call it and it easy to updated  instead of having it  hard coded inside each the Topology?

Thanks

--
IPVP

Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended 
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or 
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded 
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly 
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.

Re: Where to maintain the list of KafkaSpouts' ZkHosts?

[I PVP <ip...@hotmail.com>]

Is there a way to read from a .properties file that is located outside the Topology fat jar ?

Would still not be ideal to update the .properties file on each machine, but it is better than having to update the  ZkHosts inside the topology code.




From: Alexander T <mi...@gmail.com>
Reply: user@storm.apache.org <us...@storm.apache.org>
Date: March 24, 2016 at 5:05:52 AM
To: user@storm.apache.org <us...@storm.apache.org>
Subject:  Re: Where to maintain the list of KafkaSpouts' ZkHosts?


We also have this problem for Kinesis. I wondered if there is an API for spouts to read out the current zk hosts from Storm.

On Mar 24, 2016 01:08, "I PVP" <ip...@hotmail.com>> wrote:
What is the recommended approach to maintain the list of ZkHosts on a central “place" in which all KafkaSpouts can call it and it easy to updated  instead of having it  hard coded inside each the Topology?

Thanks

--
IPVP

Re: Where to maintain the list of KafkaSpouts' ZkHosts?

[Alexander T <mi...@gmail.com>]

We also have this problem for Kinesis. I wondered if there is an API for
spouts to read out the current zk hosts from Storm.
On Mar 24, 2016 01:08, "I PVP" <ip...@hotmail.com> wrote:

> What is the recommended approach to maintain the list of ZkHosts on a
> central “place" in which all KafkaSpouts can call it and it easy to updated
>  instead of having it  hard coded inside each the Topology?
>
> Thanks
>
> --
> IPVP
>
>