You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Benjamin Reed <br...@apache.org> on 2011/07/12 07:37:47 UTC
dev meeting minutes
i've published the notes i took from the dev meeting at
https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011DeveloperMeeting
please feel free to update if you think that i missed anything.
thanx
ben
Re: dev meeting minutes
Posted by Benjamin Reed <br...@apache.org>.
unfortunately, i stepped out during the discussion. i was only there
at the beginning and the end, so i don't know the details. perhaps
someone else can fill in.
ben
On Tue, Jul 12, 2011 at 8:08 AM, Vishal Kher <vi...@gmail.com> wrote:
> Hi Ben,
>
> Thanks for posting this.
>
>> Reconfiguration: alex talked about the work for online cluster
> reconfiguration. (addition and removal of servers.) there was some
> skepticism that the solution was too complicated. alex will
>> continue work and document the approach.
>
> Can you elaborate a bit more on what part of the protocol was considered
> complex and whether there were discussions to simplify it?
>
> -Vishal
>
> On Tue, Jul 12, 2011 at 1:37 AM, Benjamin Reed <br...@apache.org> wrote:
>
>> i've published the notes i took from the dev meeting at
>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011DeveloperMeeting
>>
>> please feel free to update if you think that i missed anything.
>>
>> thanx
>> ben
>>
>
RE: dev meeting minutes
Posted by Vishal Kathuria <vi...@fb.com>.
Thanks for the suggestion Patrick,
I posted it as an attachment to the wiki for the Zookeeper meet and included the link in the minutes.
Here is the link too
https://cwiki.apache.org/confluence/download/attachments/26805288/Hadoop+Zookeeper+Summit.pptx
-----Original Message-----
From: Patrick Hunt [mailto:phunt@apache.org]
Sent: Wednesday, July 13, 2011 10:31 AM
To: dev@zookeeper.apache.org; Vishal Kathuria
Subject: Re: dev meeting minutes
Vishal (Kathuria) would you mind posting your ZK presentation? Perhaps here as wiki attachment:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeperPresentations
with a reference from minutes?
thanks,
Patrick
On Wed, Jul 13, 2011 at 8:56 AM, Benjamin Reed <br...@apache.org> wrote:
>> I don't know your approach to cluster reconfiguration, but I had a
>> thought about it lately that may be rather trivial, if I didn't oversee anything.
>> Cluster reconfiguration would need to be an operation that goes
>> through the leader and gets acknowledged by a quorum. All operations
>> following the cluster reconfiguration operation would need to be committed by the new quorum.
>>
>> I hope this idea might help and I didn't make a fool of myself. :-)
>>
>> Best regards,
>>
>> Thomas Koch, http://www.koch.ro
>>
>
> it would be nice if things were that simple. your idea only works if
> no failures happen. (lots of things are simple if no failure happens.
> code paths become nice and clean.) since we would like to deal with
> failures, things are a bit more complicated. at the heart of it is a
> reconfiguration request that gets committed using Zab.
>
> ben
>
Re: dev meeting minutes
Posted by Patrick Hunt <ph...@apache.org>.
Vishal (Kathuria) would you mind posting your ZK presentation? Perhaps
here as wiki attachment:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeperPresentations
with a reference from minutes?
thanks,
Patrick
On Wed, Jul 13, 2011 at 8:56 AM, Benjamin Reed <br...@apache.org> wrote:
>> I don't know your approach to cluster reconfiguration, but I had a thought
>> about it lately that may be rather trivial, if I didn't oversee anything.
>> Cluster reconfiguration would need to be an operation that goes through the
>> leader and gets acknowledged by a quorum. All operations following the cluster
>> reconfiguration operation would need to be committed by the new quorum.
>>
>> I hope this idea might help and I didn't make a fool of myself. :-)
>>
>> Best regards,
>>
>> Thomas Koch, http://www.koch.ro
>>
>
> it would be nice if things were that simple. your idea only works if
> no failures happen. (lots of things are simple if no failure happens.
> code paths become nice and clean.) since we would like to deal with
> failures, things are a bit more complicated. at the heart of it is a
> reconfiguration request that gets committed using Zab.
>
> ben
>
Re: dev meeting minutes
Posted by Benjamin Reed <br...@apache.org>.
> I don't know your approach to cluster reconfiguration, but I had a thought
> about it lately that may be rather trivial, if I didn't oversee anything.
> Cluster reconfiguration would need to be an operation that goes through the
> leader and gets acknowledged by a quorum. All operations following the cluster
> reconfiguration operation would need to be committed by the new quorum.
>
> I hope this idea might help and I didn't make a fool of myself. :-)
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>
it would be nice if things were that simple. your idea only works if
no failures happen. (lots of things are simple if no failure happens.
code paths become nice and clean.) since we would like to deal with
failures, things are a bit more complicated. at the heart of it is a
reconfiguration request that gets committed using Zab.
ben
Re: dev meeting minutes
Posted by Alexander Shraer <sh...@yahoo-inc.com>.
Thats part of it. Turned out that its a bit more complicated but you're right - Zookeeper does have a lot of things already in place that make reconfiguration pretty easy.
Thanks,
Alex
On Jul 12, 2011, at 7:01 PM, "Thomas Koch" <th...@koch.ro> wrote:
> Vishal Kher:
>> Thanks. Looking forward to reading the paper.
>>
>> On Tue, Jul 12, 2011 at 1:11 PM, Alexander Shraer <shralex@yahoo-
> inc.com>wrote:
>>> Hi Vishal,
>>>
>>> At the meeting I presented the general approach I'm following to
>>> implement reconfiguration in ZK. If I understood correctly, the concern
>>> was broadly that there are a lot of cases to consider and that
>>> correctness is non-trivial. We agreed that I'll prepare a detailed
>>> writeup of the protocol and changes made to ZK. I believe that this
>>> should address most of the concerns.
>>>
>>> The positive points raised in the meeting were that the method offers
>>> minimal interruption to service during reconfiguration, uses normal ZK
>>> operation flow, and does not require modifications to other ZK
>>> operations.
>>>
>>> Best Regards,
>>> Alex
> Hi Alex,
>
> I don't know your approach to cluster reconfiguration, but I had a thought
> about it lately that may be rather trivial, if I didn't oversee anything.
> Cluster reconfiguration would need to be an operation that goes through the
> leader and gets acknowledged by a quorum. All operations following the cluster
> reconfiguration operation would need to be committed by the new quorum.
>
> I hope this idea might help and I didn't make a fool of myself. :-)
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
Re: dev meeting minutes
Posted by Thomas Koch <th...@koch.ro>.
Vishal Kher:
> Thanks. Looking forward to reading the paper.
>
> On Tue, Jul 12, 2011 at 1:11 PM, Alexander Shraer <shralex@yahoo-
inc.com>wrote:
> > Hi Vishal,
> >
> > At the meeting I presented the general approach I'm following to
> > implement reconfiguration in ZK. If I understood correctly, the concern
> > was broadly that there are a lot of cases to consider and that
> > correctness is non-trivial. We agreed that I'll prepare a detailed
> > writeup of the protocol and changes made to ZK. I believe that this
> > should address most of the concerns.
> >
> > The positive points raised in the meeting were that the method offers
> > minimal interruption to service during reconfiguration, uses normal ZK
> > operation flow, and does not require modifications to other ZK
> > operations.
> >
> > Best Regards,
> > Alex
Hi Alex,
I don't know your approach to cluster reconfiguration, but I had a thought
about it lately that may be rather trivial, if I didn't oversee anything.
Cluster reconfiguration would need to be an operation that goes through the
leader and gets acknowledged by a quorum. All operations following the cluster
reconfiguration operation would need to be committed by the new quorum.
I hope this idea might help and I didn't make a fool of myself. :-)
Best regards,
Thomas Koch, http://www.koch.ro
Re: dev meeting minutes
Posted by Eugene Koontz <ek...@hiro-tan.org>.
On 07/12/2011 01:38 PM, Vishal Kher wrote:
> Thanks. Looking forward to reading the paper.
>
> On Tue, Jul 12, 2011 at 1:11 PM, Alexander Shraer<sh...@yahoo-inc.com>wrote:
>
>> Hi Vishal,
>>
>> At the meeting I presented the general approach I'm following to implement
>> reconfiguration in ZK. If I understood correctly, the concern was broadly
>> that there are a lot of cases to consider and that correctness is
>> non-trivial. We agreed that I'll prepare a detailed writeup of the protocol
>> and changes made to ZK. I believe that this should address most of the
>> concerns.
>>
>> The positive points raised in the meeting were that the method offers
>> minimal interruption to service during reconfiguration, uses normal ZK
>> operation flow, and does not require modifications to other ZK operations.
>>
>> Best Regards,
>> Alex
>>
Thank you Alex for working on this - sounds really useful. Another thing
I remember from the dev meeting was Ted Dunning mentioning porting
everything to Clojure rather than Scala (since the original meeting
notes mentioned, in jest, porting to Scala). A Clojure port would allow
storing Zookeeper configuration state using Clojure's software
transactional memory system (see
http://clojure.org/concurrent_programming) I thought that was a very
interesting idea!
-Eugene
Re: dev meeting minutes
Posted by Vishal Kher <vi...@gmail.com>.
Thanks. Looking forward to reading the paper.
On Tue, Jul 12, 2011 at 1:11 PM, Alexander Shraer <sh...@yahoo-inc.com>wrote:
> Hi Vishal,
>
> At the meeting I presented the general approach I'm following to implement
> reconfiguration in ZK. If I understood correctly, the concern was broadly
> that there are a lot of cases to consider and that correctness is
> non-trivial. We agreed that I'll prepare a detailed writeup of the protocol
> and changes made to ZK. I believe that this should address most of the
> concerns.
>
> The positive points raised in the meeting were that the method offers
> minimal interruption to service during reconfiguration, uses normal ZK
> operation flow, and does not require modifications to other ZK operations.
>
> Best Regards,
> Alex
>
> > -----Original Message-----
> > From: Vishal Kher [mailto:vishalmlst@gmail.com]
> > Sent: Tuesday, July 12, 2011 8:09 AM
> > To: dev@zookeeper.apache.org
> > Subject: Re: dev meeting minutes
> >
> > Hi Ben,
> >
> > Thanks for posting this.
> >
> > > Reconfiguration: alex talked about the work for online cluster
> > reconfiguration. (addition and removal of servers.) there was some
> > skepticism that the solution was too complicated. alex will
> > > continue work and document the approach.
> >
> > Can you elaborate a bit more on what part of the protocol was
> > considered
> > complex and whether there were discussions to simplify it?
> >
> > -Vishal
> >
> > On Tue, Jul 12, 2011 at 1:37 AM, Benjamin Reed <br...@apache.org>
> > wrote:
> >
> > > i've published the notes i took from the dev meeting at
> > >
> > >
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011Developer
> > Meeting
> > >
> > > please feel free to update if you think that i missed anything.
> > >
> > > thanx
> > > ben
> > >
>
RE: dev meeting minutes
Posted by Alexander Shraer <sh...@yahoo-inc.com>.
Hi Vishal,
At the meeting I presented the general approach I'm following to implement reconfiguration in ZK. If I understood correctly, the concern was broadly that there are a lot of cases to consider and that correctness is non-trivial. We agreed that I'll prepare a detailed writeup of the protocol and changes made to ZK. I believe that this should address most of the concerns.
The positive points raised in the meeting were that the method offers minimal interruption to service during reconfiguration, uses normal ZK operation flow, and does not require modifications to other ZK operations.
Best Regards,
Alex
> -----Original Message-----
> From: Vishal Kher [mailto:vishalmlst@gmail.com]
> Sent: Tuesday, July 12, 2011 8:09 AM
> To: dev@zookeeper.apache.org
> Subject: Re: dev meeting minutes
>
> Hi Ben,
>
> Thanks for posting this.
>
> > Reconfiguration: alex talked about the work for online cluster
> reconfiguration. (addition and removal of servers.) there was some
> skepticism that the solution was too complicated. alex will
> > continue work and document the approach.
>
> Can you elaborate a bit more on what part of the protocol was
> considered
> complex and whether there were discussions to simplify it?
>
> -Vishal
>
> On Tue, Jul 12, 2011 at 1:37 AM, Benjamin Reed <br...@apache.org>
> wrote:
>
> > i've published the notes i took from the dev meeting at
> >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011Developer
> Meeting
> >
> > please feel free to update if you think that i missed anything.
> >
> > thanx
> > ben
> >
Re: dev meeting minutes
Posted by Vishal Kher <vi...@gmail.com>.
Hi Ben,
Thanks for posting this.
> Reconfiguration: alex talked about the work for online cluster
reconfiguration. (addition and removal of servers.) there was some
skepticism that the solution was too complicated. alex will
> continue work and document the approach.
Can you elaborate a bit more on what part of the protocol was considered
complex and whether there were discussions to simplify it?
-Vishal
On Tue, Jul 12, 2011 at 1:37 AM, Benjamin Reed <br...@apache.org> wrote:
> i've published the notes i took from the dev meeting at
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011DeveloperMeeting
>
> please feel free to update if you think that i missed anything.
>
> thanx
> ben
>