Using watcher for being notified of children addition/removal

[Steve Gury <st...@mimesis-republic.com>]

Hi,

I'm interested in being notified by addition/removal of subnode of a
watched node.

The method getChildren allows to be notified when "a successful
operation that deletes the node of the given path or creates/delete a
child under the node". But this notification only provides the path of
the parent, and I would like to have the path of the added/removed
child.

For instance:
   I create a node "BaseNode", and 1 million of EPHEMERAL_SEQUENTIAL
"ChildNode". I would like to be notified of any new creation/deletion
of "ChildNode" without having to list all "ChildNode" (1 million!)
   Ideally, I would like to be notified of any addition/deletion, the
Watcher being called with a WatchedEvent.getPath returning the path of
the ChildNode and not the path of the BaseNode.


Is there any recipe that would provide this feature (or a work around) ?

Thanks,
Steve

Re: Using watcher for being notified of children addition/removal

[Patrick Hunt <ph...@apache.org>]

You may want to consider adding a distributed queue to your use of ZK. 
As was mentioned previously, watches don't notify you of every change, 
just that a change was made. For example multiple changes may be 
"visible" when you get the notification.

A distributed queue would allow you to "log" every change, and have your 
"watcher" process easily process the result. The only issue I could see 
is one of atomicity, but depending on your use case(s) that may not be 
an issue, or perhaps one that can be worked around.

Patrick

On 08/02/2010 09:18 AM, Ted Dunning wrote:
> Another option besides Steve's excellent one would be to keep something like
> 1000 nodes in your list per znode.  Many update patterns will give you the
> same number of updates, but the ZK transactions that result (getChildren,
> read znode) will likely be more efficient, especially the getChildren call.
>
> Remember, it is not a requirement that you have a one-to-one mapping between
> your in-memory objects and in-zookeeper znodes.  If that works, fine.  If
> not, feel free to be creative.
>
> On Mon, Aug 2, 2010 at 7:45 AM, Steve Gury
> <st...@mimesis-republic.com>wrote:
>
>> Is there any recipe that would provide this feature (or a work around) ?
>>
>

Re: Using watcher for being notified of children addition/removal

[Ted Dunning <te...@gmail.com>]

Another option besides Steve's excellent one would be to keep something like
1000 nodes in your list per znode.  Many update patterns will give you the
same number of updates, but the ZK transactions that result (getChildren,
read znode) will likely be more efficient, especially the getChildren call.

Remember, it is not a requirement that you have a one-to-one mapping between
your in-memory objects and in-zookeeper znodes.  If that works, fine.  If
not, feel free to be creative.

On Mon, Aug 2, 2010 at 7:45 AM, Steve Gury
<st...@mimesis-republic.com>wrote:

> Is there any recipe that would provide this feature (or a work around) ?
>

Re: Using watcher for being notified of children addition/removal

[Dave Wright <wr...@gmail.com>]

Hi Steve,
While it would be theoretically possible to add the name of the node
that triggered the event to the watcher notification, the problem is
that this may cause you to "miss" events you wanted to see.
ZK does not guarantee you will get an event for "every" change. Any
changes that occur between the time you get a notification and do a
getChildren requesting a new notification do not send notifications.
That's why when you get the notification, you must call getChildren
and check for ALL changes against your "baseline" since there may be
multiple.
Obviously with 1m nodes there is a lot of overhead. I would suggest
splitting your list into smaller pieces, say 1000 nodes each, and
setting a watch on each of them. You mentioned using sequential nodes,
which wouldn't work the same way in this case, but either way a
redesign is probably in order.

-Dave Wright



On Mon, Aug 2, 2010 at 10:45 AM, Steve Gury
<st...@mimesis-republic.com> wrote:
> Hi,
>
> I'm interested in being notified by addition/removal of subnode of a
> watched node.
>
> The method getChildren allows to be notified when "a successful
> operation that deletes the node of the given path or creates/delete a
> child under the node". But this notification only provides the path of
> the parent, and I would like to have the path of the added/removed
> child.
>
> For instance:
>   I create a node "BaseNode", and 1 million of EPHEMERAL_SEQUENTIAL
> "ChildNode". I would like to be notified of any new creation/deletion
> of "ChildNode" without having to list all "ChildNode" (1 million!)
>   Ideally, I would like to be notified of any addition/deletion, the
> Watcher being called with a WatchedEvent.getPath returning the path of
> the ChildNode and not the path of the BaseNode.
>
>
> Is there any recipe that would provide this feature (or a work around) ?
>
> Thanks,
> Steve
>