You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by praveen puskar <pu...@gmail.com> on 2018/02/21 15:18:46 UTC

Apache storm Kafka SpoutConfig for Zookeeper quorum

Hi,

I have 2 topologies which processes live streams from more then 500
applications. It was working with one zookeeper  from the Zookeeper Quorum.
As the API don;t provide(not i know of) support for KafkaSpout
configuration. Intent here is to be highly available.


Find below the details :

To configure a kafkaSpout, it Takes BrokerHosts which in turn take one
zookeeper host.

BrokerHosts host = new ZkHosts("zookeeper-server-1:2181");
SpoutConfig spoutConfig = new SpoutConfig(host,
TopologyConstants.KAFKA_QUEUE.SOURCE,
            "/" + TopologyConstants.KAFKA_QUEUE.SOURCE, ID);
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

Problem is, If i have a zookeeper quorum(cluster of 3 zk servers), how do i
configure KafkaSpoutto take all the members of quorum instead of only 1. As
one zookeeper server may get down and the whole topology will be
unavailable.



-- 
Regards
Praveen Puskar





Please save TREE. Save the Environment. Make a contribution in retarding
Global Warming.

       Please do not print this email unless it is absolutely necessary.

Re: Apache storm Kafka SpoutConfig for Zookeeper quorum

Posted by praveen puskar <pu...@gmail.com>.
Got it. Thanks a ton.

On Wed, Feb 21, 2018 at 10:54 PM, Stig Rohde Døssing <sr...@apache.org>
wrote:

> I believe the connection string format is "host1:port1,host2:port2,host3:port3..."
> (storm-kafka uses Curator under the hood), so just supply multiple
> Zookeeper server urls to the ZkHosts constructor.
>
> 2018-02-21 16:18 GMT+01:00 praveen puskar <pu...@gmail.com>:
>
>>
>> Hi,
>>
>> I have 2 topologies which processes live streams from more then 500
>> applications. It was working with one zookeeper  from the Zookeeper Quorum.
>> As the API don;t provide(not i know of) support for KafkaSpout
>> configuration. Intent here is to be highly available.
>>
>>
>> Find below the details :
>>
>> To configure a kafkaSpout, it Takes BrokerHosts which in turn take one
>> zookeeper host.
>>
>> BrokerHosts host = new ZkHosts("zookeeper-server-1:2181");
>> SpoutConfig spoutConfig = new SpoutConfig(host, TopologyConstants.KAFKA_QUEUE.SOURCE,
>>             "/" + TopologyConstants.KAFKA_QUEUE.SOURCE, ID);
>> KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
>>
>> Problem is, If i have a zookeeper quorum(cluster of 3 zk servers), how do
>> i configure KafkaSpoutto take all the members of quorum instead of only
>> 1. As one zookeeper server may get down and the whole topology will be
>> unavailable.
>>
>>
>>
>> --
>> Regards
>> Praveen Puskar
>>
>>
>>
>>
>>
>> Please save TREE. Save the Environment. Make a contribution in retarding
>> Global Warming.
>>
>>        Please do not print this email unless it is absolutely necessary.
>>
>
>


-- 
Regards
Praveen Puskar
Grapecity Inc.
Noida - 62




Please save TREE. Save the Environment. Make a contribution in retarding
Global Warming.

       Please do not print this email unless it is absolutely necessary.

Re: Apache storm Kafka SpoutConfig for Zookeeper quorum

Posted by Stig Rohde Døssing <sr...@apache.org>.
I believe the connection string format is
"host1:port1,host2:port2,host3:port3..." (storm-kafka uses Curator under
the hood), so just supply multiple Zookeeper server urls to the ZkHosts
constructor.

2018-02-21 16:18 GMT+01:00 praveen puskar <pu...@gmail.com>:

>
> Hi,
>
> I have 2 topologies which processes live streams from more then 500
> applications. It was working with one zookeeper  from the Zookeeper Quorum.
> As the API don;t provide(not i know of) support for KafkaSpout
> configuration. Intent here is to be highly available.
>
>
> Find below the details :
>
> To configure a kafkaSpout, it Takes BrokerHosts which in turn take one
> zookeeper host.
>
> BrokerHosts host = new ZkHosts("zookeeper-server-1:2181");
> SpoutConfig spoutConfig = new SpoutConfig(host, TopologyConstants.KAFKA_QUEUE.SOURCE,
>             "/" + TopologyConstants.KAFKA_QUEUE.SOURCE, ID);
> KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
>
> Problem is, If i have a zookeeper quorum(cluster of 3 zk servers), how do
> i configure KafkaSpoutto take all the members of quorum instead of only
> 1. As one zookeeper server may get down and the whole topology will be
> unavailable.
>
>
>
> --
> Regards
> Praveen Puskar
>
>
>
>
>
> Please save TREE. Save the Environment. Make a contribution in retarding
> Global Warming.
>
>        Please do not print this email unless it is absolutely necessary.
>