You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Michi Mutsuzaki <mi...@cs.stanford.edu> on 2014/07/16 07:03:50 UTC
Re: intern project idea: decouple zab from zookeeper
Hi,
Yisheng has been working on this project for about 5 weeks for his
12-week internship. Here is the current status:
- First of all, let me thank Flavio and Hongchao for their help. I
don't think the project would be where it is right now without their
support.
- We have more or less functional implementation of zab in java. You
can checkout the code here: https://github.com/ZK-1931/javazab
- There is a simple reference server. It's an http based key-value
store that uses javazab for replicating state:
https://github.com/ZK-1931/zabkv
- The implementation is missing 2 major features, dynamic
reconfiguration and snapshotting. Yisheng is about to start working on
dynamic reconfiguration.
It's fairly easy to run the reference server. It would be great if you
can play around with it and give us feedback.
Thanks!
--Michi
On Tue, Jun 3, 2014 at 11:46 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
> The intern hasn't started yet, but here is the github repo in case
> anybody is interested.
>
> https://github.com/ZK-1931/javazab
>
> On Tue, Jun 3, 2014 at 3:20 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
>> Thanks Flavio and Raul. I feel much more confident with your support.
>> Also, it would be a good learning experience for the intern and me.
>> Let's do this from scratch. I'll set up a github repo.
>>
>> On Tue, Jun 3, 2014 at 12:51 PM, Raúl Gutiérrez Segalés
>> <rg...@itevenworks.net> wrote:
>>> On 3 June 2014 12:44, Flavio Junqueira <fp...@yahoo.com.invalid>
>>> wrote:
>>>>
>>>> It would be great to do a clean implementation of Zab. We have added a lot
>>>> crap for backward compatibility, and the reconfig stuff, although a great
>>>> feature properly implemented, didn't improve the state of the code. Also, an
>>>> implementation of the Zab protocol perhaps putting snapshots aside for v0.1,
>>>> shouldn't take more than just a few weeks.
>>>>
>>>> If you do it openly say on github, then I volunteer to help.
>>>
>>>
>>> A clean-room implementation of ZAB could indeed be awesome for multiple
>>> purposes. Reasoning around the current implementation is some times
>>> challenging for us missing the historical context.
>>>
>>> Would be more than happy to help with reviews and such as well.
>>>
>>>
>>> -rgs
RE: intern project idea: decouple zab from zookeeper
Posted by Hongchao Deng <fe...@hotmail.com>.
The WAL are used to restore db operations. IMHO, the WAL is the executed log entries of ZAB.
I believe that a btree-based, MVCC bitcask could handle ZK's interfaces much better than hashtable and another hashtable for children. In theory, each operation shouldn't require more than one disk seek such that performance will be the same.
I recently heard etcd team talking about this. It would be great if we can catch up with a protocol ready for the change. :)
- Hongchao Deng
> Date: Mon, 22 Sep 2014 22:56:16 -0700
> Subject: Re: intern project idea: decouple zab from zookeeper
> From: michi@cs.stanford.edu
> To: dev@zookeeper.apache.org
>
> Sorry for the late response Hongchao. That's definitely an interesting
> idea. What's the difference between WAL and local consistent log?
>
> On Fri, Sep 19, 2014 at 8:45 AM, Hongchao Deng <fe...@hotmail.com> wrote:
> > Consider separating a storage backend (e.g. take a look at bitcask)?Snapshot, data store, WAL are in backend now.Zab still maintains a local consistent log.
> > What's nice about this? Extending kv size beyond mem.The performance should be the same, which is the disk IO (batch) thruput -- the bottleneck.
> >
> > - Hongchao Deng
> >
> >> Date: Thu, 18 Sep 2014 22:48:36 -0700
> >> Subject: Re: intern project idea: decouple zab from zookeeper
> >> From: michi@cs.stanford.edu
> >> To: fpjunqueira@yahoo.com
> >> CC: dev@zookeeper.apache.org
> >>
> >> Hi,
> >>
> >> Yisheng extended his internship for another three month. Here is the
> >> current status of the project:
> >>
> >> - Released 0.1.0. This is an alpha quality release, and it supports
> >> all the basic functionalities including snapshot, reconfiguration, and
> >> SSL. http://search.maven.org/#artifactdetails|com.github.zk1931|jzab|0.1.0|jar
> >> - Performance looks reasonable. It can do about 22k requests per
> >> second on a 3-node cluster even though our log implementation is
> >> fairly naive. https://github.com/zk1931/jzab/wiki
> >>
> >> Yisheng started working on a Curator-like library on top of jzab that
> >> can support data structures like map and queue. Let me know if you
> >> have any other interesting project ideas.
> >>
> >> Thanks!
> >> --Michi
> >>
> >> On Wed, Jul 16, 2014 at 1:37 AM, Flavio Junqueira <fp...@yahoo.com> wrote:
> >> > Great job, Yisheng, Michi!
> >> >
> >> > -Flavio
> >> >
> >> >
> >> > On Wednesday, July 16, 2014 6:04 AM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> >> > wrote:
> >> >
> >> >
> >> >
> >> > Hi,
> >> >
> >> > Yisheng has been working on this project for about 5 weeks for his
> >> > 12-week internship. Here is the current status:
> >> >
> >> > - First of all, let me thank Flavio and Hongchao for their help. I
> >> > don't think the project would be where it is right now without their
> >> > support.
> >> > - We have more or less functional implementation of zab in java. You
> >> > can checkout the code here: https://github.com/ZK-1931/javazab
> >> > - There is a simple reference server. It's an http based key-value
> >> > store that uses javazab for replicating state:
> >> > https://github.com/ZK-1931/zabkv
> >> > - The implementation is missing 2 major features, dynamic
> >> > reconfiguration and snapshotting. Yisheng is about to start working on
> >> > dynamic reconfiguration.
> >> >
> >> > It's fairly easy to run the reference server. It would be great if you
> >> > can play around with it and give us feedback.
> >> >
> >> > Thanks!
> >> > --Michi
> >> >
> >> > On Tue, Jun 3, 2014 at 11:46 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> >> > wrote:
> >> >> The intern hasn't started yet, but here is the github repo in case
> >> >> anybody is interested.
> >> >>
> >> >> https://github.com/ZK-1931/javazab
> >> >>
> >> >> On Tue, Jun 3, 2014 at 3:20 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> >> >> wrote:
> >> >>> Thanks Flavio and Raul. I feel much more confident with your support.
> >> >>> Also, it would be a good learning experience for the intern and me.
> >> >>> Let's do this from scratch. I'll set up a github repo.
> >> >>>
> >> >>> On Tue, Jun 3, 2014 at 12:51 PM, Raúl Gutiérrez Segalés
> >> >>> <rg...@itevenworks.net> wrote:
> >> >>>> On 3 June 2014 12:44, Flavio Junqueira <fp...@yahoo.com.invalid>
> >> >>>> wrote:
> >> >>>>>
> >> >>>>> It would be great to do a clean implementation of Zab. We have added a
> >> >>>>> lot
> >> >>>>> crap for backward compatibility, and the reconfig stuff, although a
> >> >>>>> great
> >> >>>>> feature properly implemented, didn't improve the state of the code.
> >> >>>>> Also, an
> >> >>>>> implementation of the Zab protocol perhaps putting snapshots aside for
> >> >>>>> v0.1,
> >> >>>>> shouldn't take more than just a few weeks.
> >> >>>>>
> >> >>>>> If you do it openly say on github, then I volunteer to help.
> >> >>>>
> >> >>>>
> >> >>>> A clean-room implementation of ZAB could indeed be awesome for multiple
> >> >>>> purposes. Reasoning around the current implementation is some times
> >> >>>> challenging for us missing the historical context.
> >> >>>>
> >> >>>> Would be more than happy to help with reviews and such as well.
> >> >>>>
> >> >>>>
> >> >>>> -rgs
> >> >
> >
Re: intern project idea: decouple zab from zookeeper
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
Sorry for the late response Hongchao. That's definitely an interesting
idea. What's the difference between WAL and local consistent log?
On Fri, Sep 19, 2014 at 8:45 AM, Hongchao Deng <fe...@hotmail.com> wrote:
> Consider separating a storage backend (e.g. take a look at bitcask)?Snapshot, data store, WAL are in backend now.Zab still maintains a local consistent log.
> What's nice about this? Extending kv size beyond mem.The performance should be the same, which is the disk IO (batch) thruput -- the bottleneck.
>
> - Hongchao Deng
>
>> Date: Thu, 18 Sep 2014 22:48:36 -0700
>> Subject: Re: intern project idea: decouple zab from zookeeper
>> From: michi@cs.stanford.edu
>> To: fpjunqueira@yahoo.com
>> CC: dev@zookeeper.apache.org
>>
>> Hi,
>>
>> Yisheng extended his internship for another three month. Here is the
>> current status of the project:
>>
>> - Released 0.1.0. This is an alpha quality release, and it supports
>> all the basic functionalities including snapshot, reconfiguration, and
>> SSL. http://search.maven.org/#artifactdetails|com.github.zk1931|jzab|0.1.0|jar
>> - Performance looks reasonable. It can do about 22k requests per
>> second on a 3-node cluster even though our log implementation is
>> fairly naive. https://github.com/zk1931/jzab/wiki
>>
>> Yisheng started working on a Curator-like library on top of jzab that
>> can support data structures like map and queue. Let me know if you
>> have any other interesting project ideas.
>>
>> Thanks!
>> --Michi
>>
>> On Wed, Jul 16, 2014 at 1:37 AM, Flavio Junqueira <fp...@yahoo.com> wrote:
>> > Great job, Yisheng, Michi!
>> >
>> > -Flavio
>> >
>> >
>> > On Wednesday, July 16, 2014 6:04 AM, Michi Mutsuzaki <mi...@cs.stanford.edu>
>> > wrote:
>> >
>> >
>> >
>> > Hi,
>> >
>> > Yisheng has been working on this project for about 5 weeks for his
>> > 12-week internship. Here is the current status:
>> >
>> > - First of all, let me thank Flavio and Hongchao for their help. I
>> > don't think the project would be where it is right now without their
>> > support.
>> > - We have more or less functional implementation of zab in java. You
>> > can checkout the code here: https://github.com/ZK-1931/javazab
>> > - There is a simple reference server. It's an http based key-value
>> > store that uses javazab for replicating state:
>> > https://github.com/ZK-1931/zabkv
>> > - The implementation is missing 2 major features, dynamic
>> > reconfiguration and snapshotting. Yisheng is about to start working on
>> > dynamic reconfiguration.
>> >
>> > It's fairly easy to run the reference server. It would be great if you
>> > can play around with it and give us feedback.
>> >
>> > Thanks!
>> > --Michi
>> >
>> > On Tue, Jun 3, 2014 at 11:46 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
>> > wrote:
>> >> The intern hasn't started yet, but here is the github repo in case
>> >> anybody is interested.
>> >>
>> >> https://github.com/ZK-1931/javazab
>> >>
>> >> On Tue, Jun 3, 2014 at 3:20 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
>> >> wrote:
>> >>> Thanks Flavio and Raul. I feel much more confident with your support.
>> >>> Also, it would be a good learning experience for the intern and me.
>> >>> Let's do this from scratch. I'll set up a github repo.
>> >>>
>> >>> On Tue, Jun 3, 2014 at 12:51 PM, Raúl Gutiérrez Segalés
>> >>> <rg...@itevenworks.net> wrote:
>> >>>> On 3 June 2014 12:44, Flavio Junqueira <fp...@yahoo.com.invalid>
>> >>>> wrote:
>> >>>>>
>> >>>>> It would be great to do a clean implementation of Zab. We have added a
>> >>>>> lot
>> >>>>> crap for backward compatibility, and the reconfig stuff, although a
>> >>>>> great
>> >>>>> feature properly implemented, didn't improve the state of the code.
>> >>>>> Also, an
>> >>>>> implementation of the Zab protocol perhaps putting snapshots aside for
>> >>>>> v0.1,
>> >>>>> shouldn't take more than just a few weeks.
>> >>>>>
>> >>>>> If you do it openly say on github, then I volunteer to help.
>> >>>>
>> >>>>
>> >>>> A clean-room implementation of ZAB could indeed be awesome for multiple
>> >>>> purposes. Reasoning around the current implementation is some times
>> >>>> challenging for us missing the historical context.
>> >>>>
>> >>>> Would be more than happy to help with reviews and such as well.
>> >>>>
>> >>>>
>> >>>> -rgs
>> >
>
RE: intern project idea: decouple zab from zookeeper
Posted by Hongchao Deng <fe...@hotmail.com>.
Consider separating a storage backend (e.g. take a look at bitcask)?Snapshot, data store, WAL are in backend now.Zab still maintains a local consistent log.
What's nice about this? Extending kv size beyond mem.The performance should be the same, which is the disk IO (batch) thruput -- the bottleneck.
- Hongchao Deng
> Date: Thu, 18 Sep 2014 22:48:36 -0700
> Subject: Re: intern project idea: decouple zab from zookeeper
> From: michi@cs.stanford.edu
> To: fpjunqueira@yahoo.com
> CC: dev@zookeeper.apache.org
>
> Hi,
>
> Yisheng extended his internship for another three month. Here is the
> current status of the project:
>
> - Released 0.1.0. This is an alpha quality release, and it supports
> all the basic functionalities including snapshot, reconfiguration, and
> SSL. http://search.maven.org/#artifactdetails|com.github.zk1931|jzab|0.1.0|jar
> - Performance looks reasonable. It can do about 22k requests per
> second on a 3-node cluster even though our log implementation is
> fairly naive. https://github.com/zk1931/jzab/wiki
>
> Yisheng started working on a Curator-like library on top of jzab that
> can support data structures like map and queue. Let me know if you
> have any other interesting project ideas.
>
> Thanks!
> --Michi
>
> On Wed, Jul 16, 2014 at 1:37 AM, Flavio Junqueira <fp...@yahoo.com> wrote:
> > Great job, Yisheng, Michi!
> >
> > -Flavio
> >
> >
> > On Wednesday, July 16, 2014 6:04 AM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> > wrote:
> >
> >
> >
> > Hi,
> >
> > Yisheng has been working on this project for about 5 weeks for his
> > 12-week internship. Here is the current status:
> >
> > - First of all, let me thank Flavio and Hongchao for their help. I
> > don't think the project would be where it is right now without their
> > support.
> > - We have more or less functional implementation of zab in java. You
> > can checkout the code here: https://github.com/ZK-1931/javazab
> > - There is a simple reference server. It's an http based key-value
> > store that uses javazab for replicating state:
> > https://github.com/ZK-1931/zabkv
> > - The implementation is missing 2 major features, dynamic
> > reconfiguration and snapshotting. Yisheng is about to start working on
> > dynamic reconfiguration.
> >
> > It's fairly easy to run the reference server. It would be great if you
> > can play around with it and give us feedback.
> >
> > Thanks!
> > --Michi
> >
> > On Tue, Jun 3, 2014 at 11:46 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> > wrote:
> >> The intern hasn't started yet, but here is the github repo in case
> >> anybody is interested.
> >>
> >> https://github.com/ZK-1931/javazab
> >>
> >> On Tue, Jun 3, 2014 at 3:20 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> >> wrote:
> >>> Thanks Flavio and Raul. I feel much more confident with your support.
> >>> Also, it would be a good learning experience for the intern and me.
> >>> Let's do this from scratch. I'll set up a github repo.
> >>>
> >>> On Tue, Jun 3, 2014 at 12:51 PM, Raúl Gutiérrez Segalés
> >>> <rg...@itevenworks.net> wrote:
> >>>> On 3 June 2014 12:44, Flavio Junqueira <fp...@yahoo.com.invalid>
> >>>> wrote:
> >>>>>
> >>>>> It would be great to do a clean implementation of Zab. We have added a
> >>>>> lot
> >>>>> crap for backward compatibility, and the reconfig stuff, although a
> >>>>> great
> >>>>> feature properly implemented, didn't improve the state of the code.
> >>>>> Also, an
> >>>>> implementation of the Zab protocol perhaps putting snapshots aside for
> >>>>> v0.1,
> >>>>> shouldn't take more than just a few weeks.
> >>>>>
> >>>>> If you do it openly say on github, then I volunteer to help.
> >>>>
> >>>>
> >>>> A clean-room implementation of ZAB could indeed be awesome for multiple
> >>>> purposes. Reasoning around the current implementation is some times
> >>>> challenging for us missing the historical context.
> >>>>
> >>>> Would be more than happy to help with reviews and such as well.
> >>>>
> >>>>
> >>>> -rgs
> >
Re: intern project idea: decouple zab from zookeeper
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
Hi,
Yisheng extended his internship for another three month. Here is the
current status of the project:
- Released 0.1.0. This is an alpha quality release, and it supports
all the basic functionalities including snapshot, reconfiguration, and
SSL. http://search.maven.org/#artifactdetails|com.github.zk1931|jzab|0.1.0|jar
- Performance looks reasonable. It can do about 22k requests per
second on a 3-node cluster even though our log implementation is
fairly naive. https://github.com/zk1931/jzab/wiki
Yisheng started working on a Curator-like library on top of jzab that
can support data structures like map and queue. Let me know if you
have any other interesting project ideas.
Thanks!
--Michi
On Wed, Jul 16, 2014 at 1:37 AM, Flavio Junqueira <fp...@yahoo.com> wrote:
> Great job, Yisheng, Michi!
>
> -Flavio
>
>
> On Wednesday, July 16, 2014 6:04 AM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> wrote:
>
>
>
> Hi,
>
> Yisheng has been working on this project for about 5 weeks for his
> 12-week internship. Here is the current status:
>
> - First of all, let me thank Flavio and Hongchao for their help. I
> don't think the project would be where it is right now without their
> support.
> - We have more or less functional implementation of zab in java. You
> can checkout the code here: https://github.com/ZK-1931/javazab
> - There is a simple reference server. It's an http based key-value
> store that uses javazab for replicating state:
> https://github.com/ZK-1931/zabkv
> - The implementation is missing 2 major features, dynamic
> reconfiguration and snapshotting. Yisheng is about to start working on
> dynamic reconfiguration.
>
> It's fairly easy to run the reference server. It would be great if you
> can play around with it and give us feedback.
>
> Thanks!
> --Michi
>
> On Tue, Jun 3, 2014 at 11:46 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
> wrote:
>> The intern hasn't started yet, but here is the github repo in case
>> anybody is interested.
>>
>> https://github.com/ZK-1931/javazab
>>
>> On Tue, Jun 3, 2014 at 3:20 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
>> wrote:
>>> Thanks Flavio and Raul. I feel much more confident with your support.
>>> Also, it would be a good learning experience for the intern and me.
>>> Let's do this from scratch. I'll set up a github repo.
>>>
>>> On Tue, Jun 3, 2014 at 12:51 PM, Raúl Gutiérrez Segalés
>>> <rg...@itevenworks.net> wrote:
>>>> On 3 June 2014 12:44, Flavio Junqueira <fp...@yahoo.com.invalid>
>>>> wrote:
>>>>>
>>>>> It would be great to do a clean implementation of Zab. We have added a
>>>>> lot
>>>>> crap for backward compatibility, and the reconfig stuff, although a
>>>>> great
>>>>> feature properly implemented, didn't improve the state of the code.
>>>>> Also, an
>>>>> implementation of the Zab protocol perhaps putting snapshots aside for
>>>>> v0.1,
>>>>> shouldn't take more than just a few weeks.
>>>>>
>>>>> If you do it openly say on github, then I volunteer to help.
>>>>
>>>>
>>>> A clean-room implementation of ZAB could indeed be awesome for multiple
>>>> purposes. Reasoning around the current implementation is some times
>>>> challenging for us missing the historical context.
>>>>
>>>> Would be more than happy to help with reviews and such as well.
>>>>
>>>>
>>>> -rgs
>
Re: intern project idea: decouple zab from zookeeper
Posted by Flavio Junqueira <fp...@yahoo.com.INVALID>.
Great job, Yisheng, Michi!
-Flavio
On Wednesday, July 16, 2014 6:04 AM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
>
>
>Hi,
>
>Yisheng has been working on this project for about 5 weeks for his
>12-week internship. Here is the current status:
>
>- First of all, let me thank Flavio and Hongchao for their help. I
>don't think the project would be where it is right now without their
>support.
>- We have more or less functional implementation of zab in java. You
>can checkout the code here: https://github.com/ZK-1931/javazab
>- There is a simple reference server. It's an http based key-value
>store that uses javazab for replicating state:
>https://github.com/ZK-1931/zabkv
>- The implementation is missing 2 major features, dynamic
>reconfiguration and snapshotting. Yisheng is about to start working on
>dynamic reconfiguration.
>
>It's fairly easy to run the reference server. It would be great if you
>can play around with it and give us feedback.
>
>Thanks!
>--Michi
>
>On Tue, Jun 3, 2014 at 11:46 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
>> The intern hasn't started yet, but here is the github repo in case
>> anybody is interested.
>>
>> https://github.com/ZK-1931/javazab
>>
>> On Tue, Jun 3, 2014 at 3:20 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
>>> Thanks Flavio and Raul. I feel much more confident with your support.
>>> Also, it would be a good learning experience for the intern and me.
>>> Let's do this from scratch. I'll set up a github repo.
>>>
>>> On Tue, Jun 3, 2014 at 12:51 PM, Raúl Gutiérrez Segalés
>>> <rg...@itevenworks.net> wrote:
>>>> On 3 June 2014 12:44, Flavio Junqueira <fp...@yahoo.com.invalid>
>>>> wrote:
>>>>>
>>>>> It would be great to do a clean implementation of Zab. We have added a lot
>>>>> crap for backward compatibility, and the reconfig stuff, although a great
>>>>> feature properly implemented, didn't improve the state of the code. Also, an
>>>>> implementation of the Zab protocol perhaps putting snapshots aside for v0.1,
>>>>> shouldn't take more than just a few weeks.
>>>>>
>>>>> If you do it openly say on github, then I volunteer to help.
>>>>
>>>>
>>>> A clean-room implementation of ZAB could indeed be awesome for multiple
>>>> purposes. Reasoning around the current implementation is some times
>>>> challenging for us missing the historical context.
>>>>
>>>> Would be more than happy to help with reviews and such as well.
>>>>
>>>>
>>>> -rgs
>
>