Standard redistributable set of primitives?

[Tom Nichols <tm...@gmail.com>]

Hi,

I was wondering if there were plans to create a set of standard
ZooKeeper primitives, sort of like commons-collections.  I figure it
would be mostly based off of the recipes on the ZK wiki, but it would
provide users with a slightly easier starting point, not to mention it
would be well-tested and prevent users avoid "re-inventing the wheel."
 Does this sound like a reasonable idea?

Thanks.
-Tom

Re: Standard redistributable set of primitives?

[Tom Nichols <tm...@gmail.com>]

Hi everyone,

Attached are a couple primitives I've created for our use cases thus
far.  I'll try to explain my reasoning for some of the design
decisions below, then let you take a look and critique (they are
attached, w/ supporting classes and test cases).

* ROOT NODE
The idea of a root node is to identify to a pimitive where it can
manipulate nodes without possibly conflicting with another application
or primitive using the same ZK server instance.  So oftentimes it
might be something like:
/myApp/LeaderLock  - for a leader lock sync mechanism
/myApp/barrier1    - for someother place in the app that needs a barrier

* DEFAULT WATCHER
I have use cases where a single thread might use multiple different
sync strategies at different points in the application.  To give you
my use case:
- We have a pool of threads, one of which needs to be elected leader
(so we use the DistributedLeaderLock to determin who is the leader.)
- The leader polls a queue, then notifies the slaves (threads who did
not get elected leader) who then wake and consume new messages.  When
the slaves have exhausted the queue, they notify the leader to resume
polling.

So the first sync mechanism is obvious (a distributed leader lock/
leader election mechanism).  For the second sync/ notification
mechanism, I've created what I call a "relay."  Basically it toggles
creation of a single node that everyone is watching.  The leader
creates the node, telling all of the waiting slaves to "go."  When one
of the slaves realizes there are no more messages to consume, it will
attempt to delete the node.  This indicates to the slaves to stop, and
the leader to resume polling.  (Yes, in my case there is contention
for all of the clients to delete the same node potentially, but I've
designed it so you can use it in a "safe" or "unsafe" way.  See the
code.)  But basically it's a relay with an "up" and "down" state.
Assume for the time being that it is a sync mechanism used by the same
thread, but with a different purpose.

My point is, we have two sync mechanisms that can share the same ZK
instance by using different watchers.  So if you look at my code, all
of the "getChildren" or "exists" methods that want to be notified as a
watcher don't use the "true" argument to have the message passed to
the default watcher.  Instead, they pass a watcher instance -- "this"
-- so _that_ primitive gets notified of what _it_ is interested in,
and any other primitives using the same ZK client instance can do the
same without a notification intended for one primitive waking the
other.  Make sense?  I swear, it's really not that complicated, it
really does make sense if you understand the problem.  I'm just not
very good at explaining it :)

So my point of all that is, sync primitives don't really want to rely
on the default watcher.  Instead, they should register themselves as
the watcher to be notified for particular events, which allows the ZK
instance to be shared by multiple primitives in the same thread.


ZKUtil has some handy utility methods (and allows for configuration of
some defaults via System properties if desired) and SyncPrimitive
provides a base for the implementation classes.



On Sat, Jan 17, 2009 at 2:39 PM, Adam Rosien <ad...@rosien.net> wrote:
> Re: ZOOKEEPER-80:
>
> (I'm waiting for my password reset to the ASF JIRA so I thought I'd
> comment here in the mean time...)
>
> Maven has a nice directory layout structure that takes into account
> multiple languages, e.g., src/main/java, src/main/c, etc.
>
> I really dislike the terms recipes and contrib for a collection of
> high-level implementations that people will rely upon. Recipes don't
> sound like code, they sound like documentation to me. Do you ever
> really use contrib code directly in your programs? Usually one copies
> it and modifies it, like the classic petstore web app, rather than
> linking directly against it. I think if there are going to be
> implementations of the recipes then these should have a name, rather
> than being pushed into some contrib bucket that gives the impression
> that it's take-it-or-leave-it code.
>
> Something like zookeeper-commons or ...?
>
> .. Adam
>
> On Fri, Jan 16, 2009 at 11:22 AM, Benjamin Reed <br...@yahoo-inc.com> wrote:
>> This is exactly what we want. ZooKeeper was designed to provide kind of a coordination micro-kernel. the idea is that all sorts of more specialized primitives can be built at the client, thus keeping the core server simple and manageable.
>>
>> unfortunately, the thing that is holding us back is the directory structure. in some sense it's pretty silly, but we need a nice way to organize the recipe specifications and implementations so that multiplatform implementations will be compatible and users can understand the semantics. See ZOOKEEPER-80. Once that gets fixed we have a lock recipe, keptset recipe, and there is a file monitoring library that is ready to go in.
>>
>> the key to ZOOKEEPER-80 is to fix the structure and the build process.  if you have a flash inspiration on how to best implement ZOOKEEPER-80, we are all ears.
>>
>> ben
>> ________________________________________
>> From: Tom Nichols [tmnichols@gmail.com]
>> Sent: Friday, January 16, 2009 6:54 AM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: Standard redistributable set of primitives?
>>
>> Hi,
>>
>> I was wondering if there were plans to create a set of standard
>> ZooKeeper primitives, sort of like commons-collections.  I figure it
>> would be mostly based off of the recipes on the ZK wiki, but it would
>> provide users with a slightly easier starting point, not to mention it
>> would be well-tested and prevent users avoid "re-inventing the wheel."
>>  Does this sound like a reasonable idea?
>>
>> Thanks.
>> -Tom
>>
>

Re: Standard redistributable set of primitives?

[Adam Rosien <ad...@rosien.net>]

Re: ZOOKEEPER-80:

(I'm waiting for my password reset to the ASF JIRA so I thought I'd
comment here in the mean time...)

Maven has a nice directory layout structure that takes into account
multiple languages, e.g., src/main/java, src/main/c, etc.

I really dislike the terms recipes and contrib for a collection of
high-level implementations that people will rely upon. Recipes don't
sound like code, they sound like documentation to me. Do you ever
really use contrib code directly in your programs? Usually one copies
it and modifies it, like the classic petstore web app, rather than
linking directly against it. I think if there are going to be
implementations of the recipes then these should have a name, rather
than being pushed into some contrib bucket that gives the impression
that it's take-it-or-leave-it code.

Something like zookeeper-commons or ...?

.. Adam

On Fri, Jan 16, 2009 at 11:22 AM, Benjamin Reed <br...@yahoo-inc.com> wrote:
> This is exactly what we want. ZooKeeper was designed to provide kind of a coordination micro-kernel. the idea is that all sorts of more specialized primitives can be built at the client, thus keeping the core server simple and manageable.
>
> unfortunately, the thing that is holding us back is the directory structure. in some sense it's pretty silly, but we need a nice way to organize the recipe specifications and implementations so that multiplatform implementations will be compatible and users can understand the semantics. See ZOOKEEPER-80. Once that gets fixed we have a lock recipe, keptset recipe, and there is a file monitoring library that is ready to go in.
>
> the key to ZOOKEEPER-80 is to fix the structure and the build process.  if you have a flash inspiration on how to best implement ZOOKEEPER-80, we are all ears.
>
> ben
> ________________________________________
> From: Tom Nichols [tmnichols@gmail.com]
> Sent: Friday, January 16, 2009 6:54 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Standard redistributable set of primitives?
>
> Hi,
>
> I was wondering if there were plans to create a set of standard
> ZooKeeper primitives, sort of like commons-collections.  I figure it
> would be mostly based off of the recipes on the ZK wiki, but it would
> provide users with a slightly easier starting point, not to mention it
> would be well-tested and prevent users avoid "re-inventing the wheel."
>  Does this sound like a reasonable idea?
>
> Thanks.
> -Tom
>

RE: Standard redistributable set of primitives?

[Benjamin Reed <br...@yahoo-inc.com>]

This is exactly what we want. ZooKeeper was designed to provide kind of a coordination micro-kernel. the idea is that all sorts of more specialized primitives can be built at the client, thus keeping the core server simple and manageable.

unfortunately, the thing that is holding us back is the directory structure. in some sense it's pretty silly, but we need a nice way to organize the recipe specifications and implementations so that multiplatform implementations will be compatible and users can understand the semantics. See ZOOKEEPER-80. Once that gets fixed we have a lock recipe, keptset recipe, and there is a file monitoring library that is ready to go in.

the key to ZOOKEEPER-80 is to fix the structure and the build process.  if you have a flash inspiration on how to best implement ZOOKEEPER-80, we are all ears.

ben
________________________________________
From: Tom Nichols [tmnichols@gmail.com]
Sent: Friday, January 16, 2009 6:54 AM
To: zookeeper-user@hadoop.apache.org
Subject: Standard redistributable set of primitives?

Hi,

I was wondering if there were plans to create a set of standard
ZooKeeper primitives, sort of like commons-collections.  I figure it
would be mostly based off of the recipes on the ZK wiki, but it would
provide users with a slightly easier starting point, not to mention it
would be well-tested and prevent users avoid "re-inventing the wheel."
 Does this sound like a reasonable idea?

Thanks.
-Tom

Re: Standard redistributable set of primitives?

[Patrick Hunt <ph...@apache.org>]

There's been some interest to see this in 3.2.0, which I think is a good 
idea. We'll be firming up 3.2 plans once 3.1 is out the door.

There's open JIRA on this btw:
https://issues.apache.org/jira/browse/ZOOKEEPER-78

Feel free to work on this if you'd like, we certainly see the benefit of 
this and would like to include in our releases.

The basic recipes are documented here:
http://hadoop.apache.org/zookeeper/docs/r3.0.1/recipes.html

Patrick

Mahadev Konar wrote:
> Hi Tom,
>  IT does sound like a reasonable idea. If you want to go ahead and implement
> one of those, we would be happy to help out and get it into Zookeeper. We
> havent had the bandwidth to put in these recipes in Zookeeper code base.
> Please go ahead and create a jira if you want to work on it.
> 
> Thanks
> mahadev
> 
> 
> On 1/16/09 6:54 AM, "Tom Nichols" <tm...@gmail.com> wrote:
> 
>> Hi,
>>
>> I was wondering if there were plans to create a set of standard
>> ZooKeeper primitives, sort of like commons-collections.  I figure it
>> would be mostly based off of the recipes on the ZK wiki, but it would
>> provide users with a slightly easier starting point, not to mention it
>> would be well-tested and prevent users avoid "re-inventing the wheel."
>>  Does this sound like a reasonable idea?
>>
>> Thanks.
>> -Tom
> 

Re: Standard redistributable set of primitives?

[Mahadev Konar <ma...@yahoo-inc.com>]

Hi Tom,
 IT does sound like a reasonable idea. If you want to go ahead and implement
one of those, we would be happy to help out and get it into Zookeeper. We
havent had the bandwidth to put in these recipes in Zookeeper code base.
Please go ahead and create a jira if you want to work on it.

Thanks
mahadev


On 1/16/09 6:54 AM, "Tom Nichols" <tm...@gmail.com> wrote:

> Hi,
> 
> I was wondering if there were plans to create a set of standard
> ZooKeeper primitives, sort of like commons-collections.  I figure it
> would be mostly based off of the recipes on the ZK wiki, but it would
> provide users with a slightly easier starting point, not to mention it
> would be well-tested and prevent users avoid "re-inventing the wheel."
>  Does this sound like a reasonable idea?
> 
> Thanks.
> -Tom