You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Patrick Hunt (JIRA)" <ji...@apache.org> on 2008/07/17 19:39:31 UTC
[jira] Created: (ZOOKEEPER-80) Document process for client recipe
contributions
Document process for client recipe contributions
------------------------------------------------
Key: ZOOKEEPER-80
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
Project: Zookeeper
Issue Type: Task
Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
How do we accept zk client recipe contributions? Initiated by the following discussion on the mailing list:
------ ben reed wrote --------
Excellent proposal. The only thing I would add is that there should be
an english description of the recipe in subversion. That way if someone
wanted to do a compatible binding they can do it. If the recipe is on
the wiki it would be hard to keep it in sync, so it is important that it
is in subversion. My preference would be that the doc would be in the
same contrib subdirectory as the source for ease of maintenance.
ben
Patrick Hunt wrote:
> > James, thanks for the contribution! Tests and everything. :-)
> >
> > Jacob sent some mail to the list recently (attached) that details a
> > protocol that he's used successfully (and picked up by some zk users).
> > I have a todo item to document this protocol on the recipes wiki page,
> > haven't gotten to it yet. Not sure how/if this matches what you've
> > done but we should sync up (also see below).
> > https://issues.apache.org/jira/browse/ZOOKEEPER-79
> >
> > There has been some discussion on client side helper code in the past
> > however this is the first contribution. We need to make some decisions
> > and outline what/how we will accept.
> >
> > 1) I think we should have a
> > "contrib/recipes/{java/{main,test}/org/apache/zookeeper/... ,c/,...}"
> > hierarchy for contributions that implement recipes, including any
> > helper code
> >
> > 2) We should first document recipes on the wiki, then implement them
> > in the code
> > http://wiki.apache.org/hadoop/ZooKeeper/ZooKeeperRecipes
> > The code should fully document the api/implementation, and refer to
> > wiki page for protocol specifics.
> >
> > 3) What should we do relative to ZK releases. Are recipes included in
> > a release? Will bugs in recipes hold up a release?
> >
> > My initial thought is that contrib is available through svn, but not
> > included in the release. If users want to access/use this code they
> > will be required to checkout/build themselves. (at least initially)
> >
> > 4) We will not require "parody" btw the various client languages.
> > Currently we support Java/C clients, we will be adding various
> > scripting languages soon. Contributions will be submitted for various
> > clients (James' submission is for java), that will be placed into
> > contrib, if someone else contributes C bindings (etc...) we will add
> > those to contrib/recipes as well.
> >
> > 5) Implementations should strive to implement similar recipe protocols
> > (see 2 above, a good reason to document before implement). There may
> > be multiple, different, protocols each with their own implementation,
> > but for a particular protocol the implementations should be the same.
> >
> > We may want to stress 5 even more - if multiple clients
> > implementations (c/java/...) are participating in a single instance of
> > leader election it will be CRITICAL for them to be inter-operable.
> >
> >
> > Comments, questions, suggestion?
> >
> > Patrick
> >
> > James Strachan wrote:
>> >> So having recently discovered ZooKeeper, I'm really liking it - good
>> >> job folks!
>> >>
>> >> I've seen discussions of building high level features from the core ZK
>> >> library and had not seen any available on the interweb so figured I'd
>> >> have a try creating a simple one. Feel free to ignore it if a ZK ninja
>> >> can think of a neater way of doing it - I've basically followed the
>> >> protocol defined in the recent ZK presentation...
>> >> http://developer.yahoo.com/blogs/hadoop/2008/03/intro-to-zookeeper-video.html
>> >>
>> >>
>> >> I've submitted the code as a patch here...
>> >> https://issues.apache.org/jira/browse/ZOOKEEPER-78
>> >>
>> >> I figured the Java Client might as well come with some helper code to
>> >> make doing things like exclusive locks or leader elections easier; we
>> >> could always spin them out into a separate library if and when
>> >> required etc. Right now its one fairly simple class :)
>> >>
>> >> Currently its a simple class where you can register a Runnable to be
>> >> invoked when you have the lock; or you can just keep asking if you
>> >> have the lock now and again as you see fit etc.
>> >>
>> >> WriteLock locker = new WriteLock(zookeeper, "/foo/bar");
>> >> locker.setWhenOwner(new Runnable() {...}); // fire this code when
>> >> owner...
>> >>
>> >> // lets try own it
>> >> locker.acquire();
>> >>
>> >> // I may or may not have the lock now
>> >> if (locker.isOwner()) {....}
>> >>
>> >> // time passes
>> >> locker.close();
>> >>
>> >>
>> >> Thoughts?
>> >>
> >
> > ------------------------------------------------------------------------
> >
> > Subject:
> > Re: [Zookeeper-user] Leader election
> > From:
> > "Jacob Levy" <jy...@yahoo-inc.com>
> > Date:
> > Fri, 11 Jul 2008 10:42:33 -0700
> > To:
> > "Flavio Junqueira" <fp...@yahoo.com>,
> > <zo...@lists.sourceforge.net>, <av...@gmail.com>
> >
> > To:
> > "Flavio Junqueira" <fp...@yahoo.com>,
> > <zo...@lists.sourceforge.net>, <av...@gmail.com>
> > CC:
> > zookeeper-user@hadoop.apache.org
> >
> >
> > *Avinash*
> >
> >
> >
> > The following protocol will help you fix the observed misbehavior. As
> > Flavio points out, you cannot rely on the order of nodes in
> > getChildren, you must use an intrinsic property of each node to
> > determine who is the leader. The protocol devised by Runping Qi and
> > described here will do that.
> >
> >
> >
> > First of all, when you create child nodes of the node that holds the
> > leadership bids, you must create them with the EPHEMERAL and SEQUENCE
> > flag. ZooKeeper guarantees to give you an ephemeral node named
> > uniquely and with a sequence number larger by at least one than any
> > previously created node in the sequence. You provide a prefix, like
> > "L_" or your own choice, and ZooKeeper creates nodes named "L_23",
> > "L_24", etc. The sequence number starts at 0 and increases monotonously.
> >
> >
> >
> > Once you've placed your leadership bid, you search backwards from the
> > sequence number of **your** node to see if there are any preceding (in
> > terms of the sequence number) nodes. When you find one, you place a
> > watch on it and wait for it to disappear. When you get the watch
> > notification, you search again, until you do not find a preceding
> > node, then you know you're the leader. This protocol guarantees that
> > there is at any time only one node that thinks it is the leader. But
> > it does not disseminate information about who is the leader. If you
> > want everyone to know who is the leader, you can have an additional
> > Znode whose value is the name of the current leader (or some
> > identifying information on how to contact the leader, etc.). Note that
> > this cannot be done atomically, so by the time other nodes find out
> > who the leader is, the leadership may already have passed on to a
> > different node.
> >
> >
> >
> > *Flavio*
> >
> >
> >
> > Might it make sense to provide a standardized implementation of leader
> > election in the library code in Java?
> >
> >
> >
> > --Jacob
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > *From:* zookeeper-user-bounces@lists.sourceforge.net
> > [mailto:zookeeper-user-bounces@lists.sourceforge.net] *On Behalf Of
> > *Flavio Junqueira
> > *Sent:* Friday, July 11, 2008 1:02 AM
> > *To:* zookeeper-user@lists.sourceforge.net
> > *Cc:* zookeeper-user@hadoop.apache.org
> > *Subject:* Re: [Zookeeper-user] Leader election
> >
> >
> >
> > Hi Avinash, getChildren returns a list in lexicographic order, so if
> > you are updating the children of the election node concurrently, then
> > you may get a different first node with different clients. If you are
> > using the sequence flag to create nodes, then you may consider
> > stripping the prefix of the node name and using the sufix value to
> > determine order.
> >
> > Hope it helps.
> >
> > -Flavio
> >
> >
> >
> > ----- Original Message ----
> > From: Avinash Lakshman <av...@gmail.com>
> > To: zookeeper-user@lists.sourceforge.net
> > Sent: Friday, July 11, 2008 7:20:06 AM
> > Subject: [Zookeeper-user] Leader election
> >
> > Hi
> >
> > I am trying to elect leader among 50 nodes. There is always one odd
> > guy who seems to think that someone else distinct from what some other
> > nodes see as leader. Could someone please tell me what is wrong with
> > the following code for leader election:
> >
> > public void electLeader()
> > {
> > ZooKeeper zk = StorageService.instance().getZooKeeperHandle();
> > String path = "/Leader";
> > try
> > {
> > String createPath = path +
> > "/L-";
> > LeaderElector.createLock_.lock();
> > while( true )
> > {
> > /* Get all znodes under the Leader znode */
> > List<String> values = zk.getChildren(path, false);
> > /*
> > * Get the first znode and if it is the
> > * pathCreated created above then the data
> > * in that znode is the leader's identity.
> > */
> > if ( leader_ == null )
> > {
> > leader_ = new AtomicReference<EndPoint>(
> > EndPoint.fromBytes( zk.getData(path + "/" + values.get(0), false,
> > null) ) );
> > }
> > else
> > {
> > leader_.set( EndPoint.fromBytes(
> > zk.getData(path + "/" + values .get(0), false, null) ) );
> > /* Disseminate the state as to who the leader
> > is. */
> > onLeaderElection();
> > }
> > logger_.debug("Elected leader is " + leader_ + " @
> > znode " + ( path + "/" + values.get(0) ) );
> > Collections.sort(values);
> > /* We need only the last portion of this znode */
> > String[] peices = pathCreated_.split("/");
> > int index = Collections.binarySearch(values,
> > peices[peices.length - 1]);
> > if ( index > 0 )
> > {
> > String pathToCheck = path + "/" +
> > values.get(index - 1);
> > Stat stat = zk.exists(pathToCheck, true);
> > if ( stat != null )
> > {
> > logger_.debug("Awaiting my turn ...");
> > condition_.await();
> > logger_.debug("Checking to see if leader
> > is around ...");
> > }
> > }
> > else
> > {
> > break;
> > }
> > }
> > }
> > catch ( InterruptedException ex )
> > {
> > logger_.warn(LogUtil.throwableToString(ex));
> > }
> > catch ( KeeperException ex )
> > {
> > logger_.warn(LogUtil.throwableToString(ex));
> > }
> > finally
> > {
> > LeaderElector.createLock_.unlock();
> > }
> > }
> > }
> >
> > Thanks
> > Avinash
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > -------------------------------------------------------------------------
> > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> > Studies have shown that voting for your favorite open source project,
> > along with a healthy diet, reduces your potential for chronic lameness
> > and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Zookeeper-user mailing list
> > Zookeeper-user@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/zookeeper-user
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-80:
----------------------------------
Description:
Per Doug's suggestion I'll use a link instead of copy/paste:
http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
was:
How do we accept zk client recipe contributions? Initiated by the following discussion on the mailing list:
------ ben reed wrote --------
Excellent proposal. The only thing I would add is that there should be
an english description of the recipe in subversion. That way if someone
wanted to do a compatible binding they can do it. If the recipe is on
the wiki it would be hard to keep it in sync, so it is important that it
is in subversion. My preference would be that the doc would be in the
same contrib subdirectory as the source for ease of maintenance.
ben
Patrick Hunt wrote:
> > James, thanks for the contribution! Tests and everything. :-)
> >
> > Jacob sent some mail to the list recently (attached) that details a
> > protocol that he's used successfully (and picked up by some zk users).
> > I have a todo item to document this protocol on the recipes wiki page,
> > haven't gotten to it yet. Not sure how/if this matches what you've
> > done but we should sync up (also see below).
> > https://issues.apache.org/jira/browse/ZOOKEEPER-79
> >
> > There has been some discussion on client side helper code in the past
> > however this is the first contribution. We need to make some decisions
> > and outline what/how we will accept.
> >
> > 1) I think we should have a
> > "contrib/recipes/{java/{main,test}/org/apache/zookeeper/... ,c/,...}"
> > hierarchy for contributions that implement recipes, including any
> > helper code
> >
> > 2) We should first document recipes on the wiki, then implement them
> > in the code
> > http://wiki.apache.org/hadoop/ZooKeeper/ZooKeeperRecipes
> > The code should fully document the api/implementation, and refer to
> > wiki page for protocol specifics.
> >
> > 3) What should we do relative to ZK releases. Are recipes included in
> > a release? Will bugs in recipes hold up a release?
> >
> > My initial thought is that contrib is available through svn, but not
> > included in the release. If users want to access/use this code they
> > will be required to checkout/build themselves. (at least initially)
> >
> > 4) We will not require "parody" btw the various client languages.
> > Currently we support Java/C clients, we will be adding various
> > scripting languages soon. Contributions will be submitted for various
> > clients (James' submission is for java), that will be placed into
> > contrib, if someone else contributes C bindings (etc...) we will add
> > those to contrib/recipes as well.
> >
> > 5) Implementations should strive to implement similar recipe protocols
> > (see 2 above, a good reason to document before implement). There may
> > be multiple, different, protocols each with their own implementation,
> > but for a particular protocol the implementations should be the same.
> >
> > We may want to stress 5 even more - if multiple clients
> > implementations (c/java/...) are participating in a single instance of
> > leader election it will be CRITICAL for them to be inter-operable.
> >
> >
> > Comments, questions, suggestion?
> >
> > Patrick
> >
> > James Strachan wrote:
>> >> So having recently discovered ZooKeeper, I'm really liking it - good
>> >> job folks!
>> >>
>> >> I've seen discussions of building high level features from the core ZK
>> >> library and had not seen any available on the interweb so figured I'd
>> >> have a try creating a simple one. Feel free to ignore it if a ZK ninja
>> >> can think of a neater way of doing it - I've basically followed the
>> >> protocol defined in the recent ZK presentation...
>> >> http://developer.yahoo.com/blogs/hadoop/2008/03/intro-to-zookeeper-video.html
>> >>
>> >>
>> >> I've submitted the code as a patch here...
>> >> https://issues.apache.org/jira/browse/ZOOKEEPER-78
>> >>
>> >> I figured the Java Client might as well come with some helper code to
>> >> make doing things like exclusive locks or leader elections easier; we
>> >> could always spin them out into a separate library if and when
>> >> required etc. Right now its one fairly simple class :)
>> >>
>> >> Currently its a simple class where you can register a Runnable to be
>> >> invoked when you have the lock; or you can just keep asking if you
>> >> have the lock now and again as you see fit etc.
>> >>
>> >> WriteLock locker = new WriteLock(zookeeper, "/foo/bar");
>> >> locker.setWhenOwner(new Runnable() {...}); // fire this code when
>> >> owner...
>> >>
>> >> // lets try own it
>> >> locker.acquire();
>> >>
>> >> // I may or may not have the lock now
>> >> if (locker.isOwner()) {....}
>> >>
>> >> // time passes
>> >> locker.close();
>> >>
>> >>
>> >> Thoughts?
>> >>
> >
> > ------------------------------------------------------------------------
> >
> > Subject:
> > Re: [Zookeeper-user] Leader election
> > From:
> > "Jacob Levy" <jy...@yahoo-inc.com>
> > Date:
> > Fri, 11 Jul 2008 10:42:33 -0700
> > To:
> > "Flavio Junqueira" <fp...@yahoo.com>,
> > <zo...@lists.sourceforge.net>, <av...@gmail.com>
> >
> > To:
> > "Flavio Junqueira" <fp...@yahoo.com>,
> > <zo...@lists.sourceforge.net>, <av...@gmail.com>
> > CC:
> > zookeeper-user@hadoop.apache.org
> >
> >
> > *Avinash*
> >
> >
> >
> > The following protocol will help you fix the observed misbehavior. As
> > Flavio points out, you cannot rely on the order of nodes in
> > getChildren, you must use an intrinsic property of each node to
> > determine who is the leader. The protocol devised by Runping Qi and
> > described here will do that.
> >
> >
> >
> > First of all, when you create child nodes of the node that holds the
> > leadership bids, you must create them with the EPHEMERAL and SEQUENCE
> > flag. ZooKeeper guarantees to give you an ephemeral node named
> > uniquely and with a sequence number larger by at least one than any
> > previously created node in the sequence. You provide a prefix, like
> > "L_" or your own choice, and ZooKeeper creates nodes named "L_23",
> > "L_24", etc. The sequence number starts at 0 and increases monotonously.
> >
> >
> >
> > Once you've placed your leadership bid, you search backwards from the
> > sequence number of **your** node to see if there are any preceding (in
> > terms of the sequence number) nodes. When you find one, you place a
> > watch on it and wait for it to disappear. When you get the watch
> > notification, you search again, until you do not find a preceding
> > node, then you know you're the leader. This protocol guarantees that
> > there is at any time only one node that thinks it is the leader. But
> > it does not disseminate information about who is the leader. If you
> > want everyone to know who is the leader, you can have an additional
> > Znode whose value is the name of the current leader (or some
> > identifying information on how to contact the leader, etc.). Note that
> > this cannot be done atomically, so by the time other nodes find out
> > who the leader is, the leadership may already have passed on to a
> > different node.
> >
> >
> >
> > *Flavio*
> >
> >
> >
> > Might it make sense to provide a standardized implementation of leader
> > election in the library code in Java?
> >
> >
> >
> > --Jacob
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > *From:* zookeeper-user-bounces@lists.sourceforge.net
> > [mailto:zookeeper-user-bounces@lists.sourceforge.net] *On Behalf Of
> > *Flavio Junqueira
> > *Sent:* Friday, July 11, 2008 1:02 AM
> > *To:* zookeeper-user@lists.sourceforge.net
> > *Cc:* zookeeper-user@hadoop.apache.org
> > *Subject:* Re: [Zookeeper-user] Leader election
> >
> >
> >
> > Hi Avinash, getChildren returns a list in lexicographic order, so if
> > you are updating the children of the election node concurrently, then
> > you may get a different first node with different clients. If you are
> > using the sequence flag to create nodes, then you may consider
> > stripping the prefix of the node name and using the sufix value to
> > determine order.
> >
> > Hope it helps.
> >
> > -Flavio
> >
> >
> >
> > ----- Original Message ----
> > From: Avinash Lakshman <av...@gmail.com>
> > To: zookeeper-user@lists.sourceforge.net
> > Sent: Friday, July 11, 2008 7:20:06 AM
> > Subject: [Zookeeper-user] Leader election
> >
> > Hi
> >
> > I am trying to elect leader among 50 nodes. There is always one odd
> > guy who seems to think that someone else distinct from what some other
> > nodes see as leader. Could someone please tell me what is wrong with
> > the following code for leader election:
> >
> > public void electLeader()
> > {
> > ZooKeeper zk = StorageService.instance().getZooKeeperHandle();
> > String path = "/Leader";
> > try
> > {
> > String createPath = path +
> > "/L-";
> > LeaderElector.createLock_.lock();
> > while( true )
> > {
> > /* Get all znodes under the Leader znode */
> > List<String> values = zk.getChildren(path, false);
> > /*
> > * Get the first znode and if it is the
> > * pathCreated created above then the data
> > * in that znode is the leader's identity.
> > */
> > if ( leader_ == null )
> > {
> > leader_ = new AtomicReference<EndPoint>(
> > EndPoint.fromBytes( zk.getData(path + "/" + values.get(0), false,
> > null) ) );
> > }
> > else
> > {
> > leader_.set( EndPoint.fromBytes(
> > zk.getData(path + "/" + values .get(0), false, null) ) );
> > /* Disseminate the state as to who the leader
> > is. */
> > onLeaderElection();
> > }
> > logger_.debug("Elected leader is " + leader_ + " @
> > znode " + ( path + "/" + values.get(0) ) );
> > Collections.sort(values);
> > /* We need only the last portion of this znode */
> > String[] peices = pathCreated_.split("/");
> > int index = Collections.binarySearch(values,
> > peices[peices.length - 1]);
> > if ( index > 0 )
> > {
> > String pathToCheck = path + "/" +
> > values.get(index - 1);
> > Stat stat = zk.exists(pathToCheck, true);
> > if ( stat != null )
> > {
> > logger_.debug("Awaiting my turn ...");
> > condition_.await();
> > logger_.debug("Checking to see if leader
> > is around ...");
> > }
> > }
> > else
> > {
> > break;
> > }
> > }
> > }
> > catch ( InterruptedException ex )
> > {
> > logger_.warn(LogUtil.throwableToString(ex));
> > }
> > catch ( KeeperException ex )
> > {
> > logger_.warn(LogUtil.throwableToString(ex));
> > }
> > finally
> > {
> > LeaderElector.createLock_.unlock();
> > }
> > }
> > }
> >
> > Thanks
> > Avinash
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > -------------------------------------------------------------------------
> > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> > Studies have shown that voting for your favorite open source project,
> > along with a healthy diet, reduces your potential for chronic lameness
> > and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Zookeeper-user mailing list
> > Zookeeper-user@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/zookeeper-user
> >
I edited the description to add a link rather than copy/paste the discussion.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614466#action_12614466 ]
Doug Cutting commented on ZOOKEEPER-80:
---------------------------------------
Whoa! Big issue description! Perhaps you could have gone with a link to the mail archive? Descriptions are included in every message about the issue...
In any case, I think perhaps each recipe deserves its own code-tree, and should hence be a separate contrib module. Perhaps, instead of 'contrib/' these should just be under 'recipes/', with a separate src/, lib/, doc/, build.xml, README.txt, etc. for each? Multiple language implementations would go in different src/ subdirectories. Does that work?
Also, I am -1 for making these subversion-only. Only released software is fully covered by the Apache license. Subversion is for internal exchange by Apache of works-in-progress, not for end users.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> How do we accept zk client recipe contributions? Initiated by the following discussion on the mailing list:
> ------ ben reed wrote --------
> Excellent proposal. The only thing I would add is that there should be
> an english description of the recipe in subversion. That way if someone
> wanted to do a compatible binding they can do it. If the recipe is on
> the wiki it would be hard to keep it in sync, so it is important that it
> is in subversion. My preference would be that the doc would be in the
> same contrib subdirectory as the source for ease of maintenance.
> ben
> Patrick Hunt wrote:
> > > James, thanks for the contribution! Tests and everything. :-)
> > >
> > > Jacob sent some mail to the list recently (attached) that details a
> > > protocol that he's used successfully (and picked up by some zk users).
> > > I have a todo item to document this protocol on the recipes wiki page,
> > > haven't gotten to it yet. Not sure how/if this matches what you've
> > > done but we should sync up (also see below).
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-79
> > >
> > > There has been some discussion on client side helper code in the past
> > > however this is the first contribution. We need to make some decisions
> > > and outline what/how we will accept.
> > >
> > > 1) I think we should have a
> > > "contrib/recipes/{java/{main,test}/org/apache/zookeeper/... ,c/,...}"
> > > hierarchy for contributions that implement recipes, including any
> > > helper code
> > >
> > > 2) We should first document recipes on the wiki, then implement them
> > > in the code
> > > http://wiki.apache.org/hadoop/ZooKeeper/ZooKeeperRecipes
> > > The code should fully document the api/implementation, and refer to
> > > wiki page for protocol specifics.
> > >
> > > 3) What should we do relative to ZK releases. Are recipes included in
> > > a release? Will bugs in recipes hold up a release?
> > >
> > > My initial thought is that contrib is available through svn, but not
> > > included in the release. If users want to access/use this code they
> > > will be required to checkout/build themselves. (at least initially)
> > >
> > > 4) We will not require "parody" btw the various client languages.
> > > Currently we support Java/C clients, we will be adding various
> > > scripting languages soon. Contributions will be submitted for various
> > > clients (James' submission is for java), that will be placed into
> > > contrib, if someone else contributes C bindings (etc...) we will add
> > > those to contrib/recipes as well.
> > >
> > > 5) Implementations should strive to implement similar recipe protocols
> > > (see 2 above, a good reason to document before implement). There may
> > > be multiple, different, protocols each with their own implementation,
> > > but for a particular protocol the implementations should be the same.
> > >
> > > We may want to stress 5 even more - if multiple clients
> > > implementations (c/java/...) are participating in a single instance of
> > > leader election it will be CRITICAL for them to be inter-operable.
> > >
> > >
> > > Comments, questions, suggestion?
> > >
> > > Patrick
> > >
> > > James Strachan wrote:
> >> >> So having recently discovered ZooKeeper, I'm really liking it - good
> >> >> job folks!
> >> >>
> >> >> I've seen discussions of building high level features from the core ZK
> >> >> library and had not seen any available on the interweb so figured I'd
> >> >> have a try creating a simple one. Feel free to ignore it if a ZK ninja
> >> >> can think of a neater way of doing it - I've basically followed the
> >> >> protocol defined in the recent ZK presentation...
> >> >> http://developer.yahoo.com/blogs/hadoop/2008/03/intro-to-zookeeper-video.html
> >> >>
> >> >>
> >> >> I've submitted the code as a patch here...
> >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-78
> >> >>
> >> >> I figured the Java Client might as well come with some helper code to
> >> >> make doing things like exclusive locks or leader elections easier; we
> >> >> could always spin them out into a separate library if and when
> >> >> required etc. Right now its one fairly simple class :)
> >> >>
> >> >> Currently its a simple class where you can register a Runnable to be
> >> >> invoked when you have the lock; or you can just keep asking if you
> >> >> have the lock now and again as you see fit etc.
> >> >>
> >> >> WriteLock locker = new WriteLock(zookeeper, "/foo/bar");
> >> >> locker.setWhenOwner(new Runnable() {...}); // fire this code when
> >> >> owner...
> >> >>
> >> >> // lets try own it
> >> >> locker.acquire();
> >> >>
> >> >> // I may or may not have the lock now
> >> >> if (locker.isOwner()) {....}
> >> >>
> >> >> // time passes
> >> >> locker.close();
> >> >>
> >> >>
> >> >> Thoughts?
> >> >>
> > >
> > > ------------------------------------------------------------------------
> > >
> > > Subject:
> > > Re: [Zookeeper-user] Leader election
> > > From:
> > > "Jacob Levy" <jy...@yahoo-inc.com>
> > > Date:
> > > Fri, 11 Jul 2008 10:42:33 -0700
> > > To:
> > > "Flavio Junqueira" <fp...@yahoo.com>,
> > > <zo...@lists.sourceforge.net>, <av...@gmail.com>
> > >
> > > To:
> > > "Flavio Junqueira" <fp...@yahoo.com>,
> > > <zo...@lists.sourceforge.net>, <av...@gmail.com>
> > > CC:
> > > zookeeper-user@hadoop.apache.org
> > >
> > >
> > > *Avinash*
> > >
> > >
> > >
> > > The following protocol will help you fix the observed misbehavior. As
> > > Flavio points out, you cannot rely on the order of nodes in
> > > getChildren, you must use an intrinsic property of each node to
> > > determine who is the leader. The protocol devised by Runping Qi and
> > > described here will do that.
> > >
> > >
> > >
> > > First of all, when you create child nodes of the node that holds the
> > > leadership bids, you must create them with the EPHEMERAL and SEQUENCE
> > > flag. ZooKeeper guarantees to give you an ephemeral node named
> > > uniquely and with a sequence number larger by at least one than any
> > > previously created node in the sequence. You provide a prefix, like
> > > "L_" or your own choice, and ZooKeeper creates nodes named "L_23",
> > > "L_24", etc. The sequence number starts at 0 and increases monotonously.
> > >
> > >
> > >
> > > Once you've placed your leadership bid, you search backwards from the
> > > sequence number of **your** node to see if there are any preceding (in
> > > terms of the sequence number) nodes. When you find one, you place a
> > > watch on it and wait for it to disappear. When you get the watch
> > > notification, you search again, until you do not find a preceding
> > > node, then you know you're the leader. This protocol guarantees that
> > > there is at any time only one node that thinks it is the leader. But
> > > it does not disseminate information about who is the leader. If you
> > > want everyone to know who is the leader, you can have an additional
> > > Znode whose value is the name of the current leader (or some
> > > identifying information on how to contact the leader, etc.). Note that
> > > this cannot be done atomically, so by the time other nodes find out
> > > who the leader is, the leadership may already have passed on to a
> > > different node.
> > >
> > >
> > >
> > > *Flavio*
> > >
> > >
> > >
> > > Might it make sense to provide a standardized implementation of leader
> > > election in the library code in Java?
> > >
> > >
> > >
> > > --Jacob
> > >
> > >
> > >
> > > ------------------------------------------------------------------------
> > >
> > > *From:* zookeeper-user-bounces@lists.sourceforge.net
> > > [mailto:zookeeper-user-bounces@lists.sourceforge.net] *On Behalf Of
> > > *Flavio Junqueira
> > > *Sent:* Friday, July 11, 2008 1:02 AM
> > > *To:* zookeeper-user@lists.sourceforge.net
> > > *Cc:* zookeeper-user@hadoop.apache.org
> > > *Subject:* Re: [Zookeeper-user] Leader election
> > >
> > >
> > >
> > > Hi Avinash, getChildren returns a list in lexicographic order, so if
> > > you are updating the children of the election node concurrently, then
> > > you may get a different first node with different clients. If you are
> > > using the sequence flag to create nodes, then you may consider
> > > stripping the prefix of the node name and using the sufix value to
> > > determine order.
> > >
> > > Hope it helps.
> > >
> > > -Flavio
> > >
> > >
> > >
> > > ----- Original Message ----
> > > From: Avinash Lakshman <av...@gmail.com>
> > > To: zookeeper-user@lists.sourceforge.net
> > > Sent: Friday, July 11, 2008 7:20:06 AM
> > > Subject: [Zookeeper-user] Leader election
> > >
> > > Hi
> > >
> > > I am trying to elect leader among 50 nodes. There is always one odd
> > > guy who seems to think that someone else distinct from what some other
> > > nodes see as leader. Could someone please tell me what is wrong with
> > > the following code for leader election:
> > >
> > > public void electLeader()
> > > {
> > > ZooKeeper zk = StorageService.instance().getZooKeeperHandle();
> > > String path = "/Leader";
> > > try
> > > {
> > > String createPath = path +
> > > "/L-";
> > > LeaderElector.createLock_.lock();
> > > while( true )
> > > {
> > > /* Get all znodes under the Leader znode */
> > > List<String> values = zk.getChildren(path, false);
> > > /*
> > > * Get the first znode and if it is the
> > > * pathCreated created above then the data
> > > * in that znode is the leader's identity.
> > > */
> > > if ( leader_ == null )
> > > {
> > > leader_ = new AtomicReference<EndPoint>(
> > > EndPoint.fromBytes( zk.getData(path + "/" + values.get(0), false,
> > > null) ) );
> > > }
> > > else
> > > {
> > > leader_.set( EndPoint.fromBytes(
> > > zk.getData(path + "/" + values .get(0), false, null) ) );
> > > /* Disseminate the state as to who the leader
> > > is. */
> > > onLeaderElection();
> > > }
> > > logger_.debug("Elected leader is " + leader_ + " @
> > > znode " + ( path + "/" + values.get(0) ) );
> > > Collections.sort(values);
> > > /* We need only the last portion of this znode */
> > > String[] peices = pathCreated_.split("/");
> > > int index = Collections.binarySearch(values,
> > > peices[peices.length - 1]);
> > > if ( index > 0 )
> > > {
> > > String pathToCheck = path + "/" +
> > > values.get(index - 1);
> > > Stat stat = zk.exists(pathToCheck, true);
> > > if ( stat != null )
> > > {
> > > logger_.debug("Awaiting my turn ...");
> > > condition_.await();
> > > logger_.debug("Checking to see if leader
> > > is around ...");
> > > }
> > > }
> > > else
> > > {
> > > break;
> > > }
> > > }
> > > }
> > > catch ( InterruptedException ex )
> > > {
> > > logger_.warn(LogUtil.throwableToString(ex));
> > > }
> > > catch ( KeeperException ex )
> > > {
> > > logger_.warn(LogUtil.throwableToString(ex));
> > > }
> > > finally
> > > {
> > > LeaderElector.createLock_.unlock();
> > > }
> > > }
> > > }
> > >
> > > Thanks
> > > Avinash
> > >
> > >
> > >
> > > ------------------------------------------------------------------------
> > >
> > > -------------------------------------------------------------------------
> > > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> > > Studies have shown that voting for your favorite open source project,
> > > along with a healthy diet, reduces your potential for chronic lameness
> > > and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> > > ------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Zookeeper-user mailing list
> > > Zookeeper-user@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/zookeeper-user
> > >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (ZOOKEEPER-80) Document process for client
recipe contributions
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on ZOOKEEPER-80 started by Patrick Hunt.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work stopped: (ZOOKEEPER-80) Document process for client
recipe contributions
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on ZOOKEEPER-80 stopped by Patrick Hunt.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617861#action_12617861 ]
Patrick Hunt commented on ZOOKEEPER-80:
---------------------------------------
My personal pref is to call them recipes, perhaps this is due to my background in networking.... also recipes is a bit more "fun". :-)
Similar comment as 103 - doesn't this result in a large number of directories in trunk? Is that a good thing?
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617868#action_12617868 ]
Hiram Chirino commented on ZOOKEEPER-80:
----------------------------------------
Patrick, no.. as the recipes will be grouped by language.
I only see a few top level dirs being created.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615817#action_12615817 ]
Patrick Hunt commented on ZOOKEEPER-80:
---------------------------------------
Ok, based on Jira comments and thread discussion I propose the following:
1) we'll add a new contrib directory as zookeeper/trunk/src/contrib/<module>
this directory will contain user contributions - such as recipes and the pending fuse module, ex:
zookeeper/trunk/src/contrib/recipes
zookeeper/trunk/src/contrib/zkfuse (see ZOOKEEPER-25)
let's call these sub-dirs "contrib modules"
2) each module will contain readme/build.xml (or autoconf etc...)/ src, docs, lib, etc.... subdirectories.
if a module contains more than one lang implemention we would have src/c src/java etc...
3) recipes (any contrib module really) will have associated documentation in it's contrib/<module>/docs directory. Note that Hadoop, and ZooKeeper, use Apache forrest for their documentation and for generating the ASF hadoop.apache.org web site.
4) recipes will have a single source hierarchy:
contrib/recipes/src/java/main/org/apache/zookeeper/recipes/leaderelection/...
contrib/recipes/src/java/main/org/apache/zookeeper/recipes/locks/...
etc...
5) build.xml for recipes will generate a release jar file "zookeeper-recipes-<version>.jar", api docs, etc... which will be tar'd and included in the ZooKeeper release
6) recipe users will be encouraged to use the released jar files in their client code (no checking out from svn as originally proposed)
7) contrib module issues will be tracked on the zookeeper jira
8) contrib modules are maintained on an "as is, best effort, work in progress" basis. The contents of these contrib modules are tagged, versioned, built, and released along with the ZooKeeper clients/server release.
Vote early vote often ;-)
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Hiram Chirino (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617827#action_12617827 ]
Hiram Chirino commented on ZOOKEEPER-80:
----------------------------------------
Hi,
Here are my 2 cents for how to implement this so it co-exists nicely with ZOOKEEPER-103
For java at least, you want to group lots of small disjoint contributions into one jar. If they become more massive, it may be worth splitting it out to independent jars, but simplicities sake/maintenance needed to manage small contributions, one jar should be enough. The only tricky bit is that we also want to support implementing the recipe in other languages, so we want to also support multiple module directories.
So my proposal is to have a module which would mainly hold the general documentation for the protocol which should be language agnostic:
trunk/zookeeper-recipes/
trunk/zookeeper-recipes/src/docs
And then we just have sub modules for the implementation of those recipes for the different languages.
The java module would use maven directory layouts
trunk/zookeeper-recipes/zookeeper-java-recipes
trunk/zookeeper-recipes/zookeeper-java-recipes/src/main/java
The c stuff would standard GNU c source layout
trunk/zookeeper-recipes/zookeeper-c-recipes
trunk/zookeeper-recipes/zookeeper-c-recipes/src
ruby would use what every ruby folks are used to.
trunk/zookeeper-recipes/zookeeper-ruby-recipes
Also note that it might be better to replace 'recipes' term with 'protocols'.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-80) Document process for client recipe
contributions
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664645#action_12664645 ]
Benjamin Reed commented on ZOOKEEPER-80:
----------------------------------------
My perspective on the issue of the directory layout is that different language bindings for a given recipe should interoperate. to that end it seems best to have a directory representing a recipe, have the recipe specification in that directory and have the different language bindings as subdirectories of that directory. that way if the recipe needs to change, because of a bug for example, it is clear exactly what implementations need to be updated.
> Document process for client recipe contributions
> ------------------------------------------------
>
> Key: ZOOKEEPER-80
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-80
> Project: Zookeeper
> Issue Type: Task
> Components: documentation
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
>
> Per Doug's suggestion I'll use a link instead of copy/paste:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-dev/200807.mbox/%3C487F8262.9020001@yahoo-inc.com%3E
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.