You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by André Oriani <ra...@students.ic.unicamp.br> on 2011/04/18 04:07:39 UTC
Extracting Zab from Zookeeper and Request class
Hi Folks,
As I mentioned in an early post , I am working on a personal project to
extract Zab (zab <https://github.com/aoriani/zab>) from ZooKeeper for my MS
project. I am doing that by bring one class from Zookeeper to my code at
time. I am not planning to have clients , so the only proposer are the
servers in the quorum.
Now that I have finish bringing the leader election (almost , still need to
bring NEWLEADER proposal) , I am starting with I think is the hard part:
Leader, Leaner e Processor classes. They are harder because I need to
change code dealing with Zookeeper requests to work in a abcast/adeliver
fashion.
Well, stopping the chitchatting, my question is : the class Request has the
fields cxn, sessionId,cxid and TnxHeader has also clientId. I did a grep on
the code and little operation is done over those fields. Most of the time
they are copied from a data structure to another. My guess is that they are
used by the quorum members to identify if committed proposal belongs to one
of its clients. Since my code is not going to deal with clients, can I
remove those fields? Or do they play some role in the atomic broadcast
protocol that I haven`t realized yet ?
Thanks,
André
Re: Extracting Zab from Zookeeper and Request class
Posted by André Oriani <ao...@gmail.com>.
Hi guys,
Sorry for taking so long to return, but I needed to study more the
code to come up more questions in a single bundle.
I am freaking out here : placing big chunks of code together without
testing and expecting everything to work is the worst software
engineering approach ever. But i do not see how to to different since
most of the job now was to comment code related the data tree and
client management. I cannot reuse the unit tests.
Implementing log will be another problem.
So Questions ;
1)For what are used Leader.PING and Leader.REVALIDATE. Are they
related to the broadcast protocol ?
2) The PreRequestProcessor and FinalRequestProcessor only work to keep
the data tree. They don't play special in the atomic broadcast
protocol right ?
3) @Mahadev, I have already taken a look at that Jira, but I never
have take a deep look into the code
(https://svn.cs.hmc.edu/svn/linkedin08/zab-multibranch/) . I thought
it was an unfinished work. But, now after studying it seems pretty
close. Do you know how functional it is ? I mean, can I use it for
application that needs atomic broadcast ? It seems at least a good
start. It has a test suit. My biggest fear with my current work is
that because I have written none of the lines, I will have a lot of
trouble making it working. That code can be used right ? I mean, They
have a LICENSE.txt file in the root with the Apache license
4) @Benjamin
They way I am extracting Zab is very similar to the one taken by the
Harvey Mudd Clinic Team. The processor are part of the protocol.
Thanks and Regards,
André
On Mon, Apr 18, 2011 at 13:01, Benjamin Reed <br...@apache.org> wrote:
> you will not use those fields in the atomic broadcast. you can look at
> Leader.propose() and see that only the request.hdr and request.txn get
> sent through the atomic broadcast.
>
> when you separate zab from zookeeper and then integrate back in, you
> should end up with something like this in
> LeaderZooKeeper.setupRequestProcessors():
>
> @Override
> protected void setupRequestProcessors() {
> RequestProcessor finalProcessor = new FinalRequestProcessor(this);
> RequestProcessor toBeAppliedProcessor = new
> Leader.ToBeAppliedRequestProcessor(
> finalProcessor, getLeader().toBeApplied);
> zabProcessor = new ZabProcessor(toBeAppliedProcessor);
> zabProcessor.start();
> firstProcessor = new PrepRequestProcessor(this, zablProcessor);
> ((PrepRequestProcessor)firstProcessor).start();
> }
>
> where ZabProcessor is a new request processor you write that uses the
> Zab implementation.
>
> ben
>
>
> On Sun, Apr 17, 2011 at 7:07 PM, André Oriani
> <ra...@students.ic.unicamp.br> wrote:
>> Hi Folks,
>>
>>
>> As I mentioned in an early post , I am working on a personal project to
>> extract Zab (zab <https://github.com/aoriani/zab>) from ZooKeeper for my MS
>> project. I am doing that by bring one class from Zookeeper to my code at
>> time. I am not planning to have clients , so the only proposer are the
>> servers in the quorum.
>>
>> Now that I have finish bringing the leader election (almost , still need to
>> bring NEWLEADER proposal) , I am starting with I think is the hard part:
>> Leader, Leaner e Processor classes. They are harder because I need to
>> change code dealing with Zookeeper requests to work in a abcast/adeliver
>> fashion.
>>
>> Well, stopping the chitchatting, my question is : the class Request has the
>> fields cxn, sessionId,cxid and TnxHeader has also clientId. I did a grep on
>> the code and little operation is done over those fields. Most of the time
>> they are copied from a data structure to another. My guess is that they are
>> used by the quorum members to identify if committed proposal belongs to one
>> of its clients. Since my code is not going to deal with clients, can I
>> remove those fields? Or do they play some role in the atomic broadcast
>> protocol that I haven`t realized yet ?
>>
>>
>> Thanks,
>> André
>>
>
Re: Extracting Zab from Zookeeper and Request class
Posted by André Oriani <ra...@students.ic.unicamp.br>.
Hi guys,
Sorry for taking so long to return, but I needed to study more the
code to come up more questions in a single bundle.
I am freaking out here : placing big chunks of code together without
testing and expecting everything to work is the worst software
engineering approach ever. But i do not see how to to different since
most of the job now was to comment code related the data tree and
client management. I cannot reuse the unit tests.
Implementing log will be another problem.
So Questions ;
1)For what are used Leader.PING and Leader.REVALIDATE. Are they
related to the broadcast protocol ?
2) The PreRequestProcessor and FinalRequestProcessor only work to keep
the data tree. They don't play special in the atomic broadcast
protocol right ?
3) @Mahadev, I have already taken a look at that Jira, but I never
have take a deep look into the code
(https://svn.cs.hmc.edu/svn/linkedin08/zab-multibranch/) . I thought
it was an unfinished work. But, now after studying it seems pretty
close. Do you know how functional it is ? I mean, can I use it for
application that needs atomic broadcast ? It seems at least a good
start. It has a test suit. My biggest fear with my current work is
that because I have written none of the lines, I will have a lot of
trouble making it working. That code can be used right ? I mean, They
have a LICENSE.txt file in the root with the Apache license
4) @Benjamin
They way I am extracting Zab is very similar to the one taken by the
Harvey Mudd Clinic Team. The processor are part of the protocol.
Thanks and Regards,
André
On Mon, Apr 18, 2011 at 13:01, Benjamin Reed <br...@apache.org> wrote:
> you will not use those fields in the atomic broadcast. you can look at
> Leader.propose() and see that only the request.hdr and request.txn get
> sent through the atomic broadcast.
>
> when you separate zab from zookeeper and then integrate back in, you
> should end up with something like this in
> LeaderZooKeeper.setupRequestProcessors():
>
> @Override
> protected void setupRequestProcessors() {
> RequestProcessor finalProcessor = new FinalRequestProcessor(this);
> RequestProcessor toBeAppliedProcessor = new
> Leader.ToBeAppliedRequestProcessor(
> finalProcessor, getLeader().toBeApplied);
> zabProcessor = new ZabProcessor(toBeAppliedProcessor);
> zabProcessor.start();
> firstProcessor = new PrepRequestProcessor(this, zablProcessor);
> ((PrepRequestProcessor)firstProcessor).start();
> }
>
> where ZabProcessor is a new request processor you write that uses the
> Zab implementation.
>
> ben
>
>
> On Sun, Apr 17, 2011 at 7:07 PM, André Oriani
> <ra...@students.ic.unicamp.br> wrote:
>> Hi Folks,
>>
>>
>> As I mentioned in an early post , I am working on a personal project to
>> extract Zab (zab <https://github.com/aoriani/zab>) from ZooKeeper for my MS
>> project. I am doing that by bring one class from Zookeeper to my code at
>> time. I am not planning to have clients , so the only proposer are the
>> servers in the quorum.
>>
>> Now that I have finish bringing the leader election (almost , still need to
>> bring NEWLEADER proposal) , I am starting with I think is the hard part:
>> Leader, Leaner e Processor classes. They are harder because I need to
>> change code dealing with Zookeeper requests to work in a abcast/adeliver
>> fashion.
>>
>> Well, stopping the chitchatting, my question is : the class Request has the
>> fields cxn, sessionId,cxid and TnxHeader has also clientId. I did a grep on
>> the code and little operation is done over those fields. Most of the time
>> they are copied from a data structure to another. My guess is that they are
>> used by the quorum members to identify if committed proposal belongs to one
>> of its clients. Since my code is not going to deal with clients, can I
>> remove those fields? Or do they play some role in the atomic broadcast
>> protocol that I haven`t realized yet ?
>>
>>
>> Thanks,
>> André
>>
>
Re: Extracting Zab from Zookeeper and Request class
Posted by Benjamin Reed <br...@apache.org>.
you will not use those fields in the atomic broadcast. you can look at
Leader.propose() and see that only the request.hdr and request.txn get
sent through the atomic broadcast.
when you separate zab from zookeeper and then integrate back in, you
should end up with something like this in
LeaderZooKeeper.setupRequestProcessors():
@Override
protected void setupRequestProcessors() {
RequestProcessor finalProcessor = new FinalRequestProcessor(this);
RequestProcessor toBeAppliedProcessor = new
Leader.ToBeAppliedRequestProcessor(
finalProcessor, getLeader().toBeApplied);
zabProcessor = new ZabProcessor(toBeAppliedProcessor);
zabProcessor.start();
firstProcessor = new PrepRequestProcessor(this, zablProcessor);
((PrepRequestProcessor)firstProcessor).start();
}
where ZabProcessor is a new request processor you write that uses the
Zab implementation.
ben
On Sun, Apr 17, 2011 at 7:07 PM, André Oriani
<ra...@students.ic.unicamp.br> wrote:
> Hi Folks,
>
>
> As I mentioned in an early post , I am working on a personal project to
> extract Zab (zab <https://github.com/aoriani/zab>) from ZooKeeper for my MS
> project. I am doing that by bring one class from Zookeeper to my code at
> time. I am not planning to have clients , so the only proposer are the
> servers in the quorum.
>
> Now that I have finish bringing the leader election (almost , still need to
> bring NEWLEADER proposal) , I am starting with I think is the hard part:
> Leader, Leaner e Processor classes. They are harder because I need to
> change code dealing with Zookeeper requests to work in a abcast/adeliver
> fashion.
>
> Well, stopping the chitchatting, my question is : the class Request has the
> fields cxn, sessionId,cxid and TnxHeader has also clientId. I did a grep on
> the code and little operation is done over those fields. Most of the time
> they are copied from a data structure to another. My guess is that they are
> used by the quorum members to identify if committed proposal belongs to one
> of its clients. Since my code is not going to deal with clients, can I
> remove those fields? Or do they play some role in the atomic broadcast
> protocol that I haven`t realized yet ?
>
>
> Thanks,
> André
>
Re: Extracting Zab from Zookeeper and Request class
Posted by Mahadev Konar <ma...@apache.org>.
Hi Andre,
YOu might want to take a look at:
https://issues.apache.org/jira/browse/ZOOKEEPER-30
There is already some work that was done as part of extracting ZAB.
thanks
mahadev
On Sun, Apr 17, 2011 at 7:07 PM, André Oriani
<ra...@students.ic.unicamp.br> wrote:
> Hi Folks,
>
>
> As I mentioned in an early post , I am working on a personal project to
> extract Zab (zab <https://github.com/aoriani/zab>) from ZooKeeper for my MS
> project. I am doing that by bring one class from Zookeeper to my code at
> time. I am not planning to have clients , so the only proposer are the
> servers in the quorum.
>
> Now that I have finish bringing the leader election (almost , still need to
> bring NEWLEADER proposal) , I am starting with I think is the hard part:
> Leader, Leaner e Processor classes. They are harder because I need to
> change code dealing with Zookeeper requests to work in a abcast/adeliver
> fashion.
>
> Well, stopping the chitchatting, my question is : the class Request has the
> fields cxn, sessionId,cxid and TnxHeader has also clientId. I did a grep on
> the code and little operation is done over those fields. Most of the time
> they are copied from a data structure to another. My guess is that they are
> used by the quorum members to identify if committed proposal belongs to one
> of its clients. Since my code is not going to deal with clients, can I
> remove those fields? Or do they play some role in the atomic broadcast
> protocol that I haven`t realized yet ?
>
>
> Thanks,
> André
>
--
thanks
mahadev
@mahadevkonar