Storm Zookeeper Client

[Myidealand account <my...@gmail.com>]

Hello ,

For my usecase I need to keep some information in zookeeper which I want my
transactional spout and bolts/ Stateful Processor nodes  to access , for
this I am thinking of using storm internal curator client to create a znode
and implement watcher so that all could read the status from that.

Let me know if my thought is right or any way forward .

Re: Storm Zookeeper Client

[Myidealand account <my...@gmail.com>]

Hello ,


While implementing this I want to know how the changes done in  the znode
created by my transactional spout are received by bolts/ Stateful Processor
nodes , means any light on watcher implementation ?


On Tue, Jan 14, 2014 at 11:14 AM, Ted Dunning <te...@gmail.com> wrote:

>
> That is a fine idea if the information is not expected to change very
> often.
>
> Zookeeper will not keep up with even a small Storm cluster that is trying
> to change things at a high rate.
>
> You have to multiply the change rate by the number of clients listening
> for changes.  If this product exceeds several thousand, I would suggest
> that you may want an alternative design where parallel data stores are
> coordinated by ZK rather than storing the information in ZK itself.
>
>
>
> On Mon, Jan 13, 2014 at 9:27 PM, Myidealand account <myidealand3@gmail.com
> > wrote:
>
>> Hello ,
>>
>> For my usecase I need to keep some information in zookeeper which I want
>> my transactional spout and bolts/ Stateful Processor nodes  to access , for
>> this I am thinking of using storm internal curator client to create a znode
>> and implement watcher so that all could read the status from that.
>>
>> Let me know if my thought is right or any way forward .
>>
>
>

Re: Storm Zookeeper Client

[Ted Dunning <te...@gmail.com>]

That is a fine idea if the information is not expected to change very often.

Zookeeper will not keep up with even a small Storm cluster that is trying
to change things at a high rate.

You have to multiply the change rate by the number of clients listening for
changes.  If this product exceeds several thousand, I would suggest that
you may want an alternative design where parallel data stores are
coordinated by ZK rather than storing the information in ZK itself.



On Mon, Jan 13, 2014 at 9:27 PM, Myidealand account
<my...@gmail.com>wrote:

> Hello ,
>
> For my usecase I need to keep some information in zookeeper which I want
> my transactional spout and bolts/ Stateful Processor nodes  to access , for
> this I am thinking of using storm internal curator client to create a znode
> and implement watcher so that all could read the status from that.
>
> Let me know if my thought is right or any way forward .
>