You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mesos.apache.org by Jie Yu <yu...@gmail.com> on 2013/11/01 01:18:46 UTC

Re: Review Request 14631: Catch-up Replicated Log 1: decoupled coordinator logics and made them asynchronous.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14631/
-----------------------------------------------------------

(Updated Nov. 1, 2013, 12:18 a.m.)


Review request for mesos and Benjamin Hindman.


Changes
-------

Using the new Shared::upgrade().

Tests:
bin/mesos-tests.sh --gtest_filter=LogTest.*:ReplicaTest.*:CoordinatorTest.* --gtest_repeat=100


Bugs: MESOS-736
    https://issues.apache.org/jira/browse/MESOS-736


Repository: mesos-git


Description
-------

This is the first patch of a series of patches that implement a catch-up mechanism for replicated log. See the following ticket for more details:
https://issues.apache.org/jira/browse/MESOS-736

Here is a brief summary of this patch: (Sorry for the fact that we are not able to break it into smaller patches :()

1) Pulled the original Coordinator logic out and divides it into several Paxos phases (see src/log/consensus.hpp). Instead of using a blocking semantics, we implemented all the logics asynchronously.

2) In order to ensure the liveness of a catch-uper, we implemented a retry logic by bumping the proposal number. This also requires us to slightly change the existing replica protocol.

3) Made the "fill" operation independent of the underlying replica. Instead, introduced a catchup (see src/log/catchup.hpp) function to make sure the underlying local replica has learned each write.

4) Modified the log tests to adapt to the new semantics (see (3) above)

This is a joint work with Yan Xu.


Diffs (updated)
-----

  src/Makefile.am a11c76b 
  src/log/catchup.hpp PRE-CREATION 
  src/log/catchup.cpp PRE-CREATION 
  src/log/consensus.hpp PRE-CREATION 
  src/log/consensus.cpp PRE-CREATION 
  src/log/coordinator.hpp 3f6fb7c 
  src/log/coordinator.cpp 6e6466f 
  src/log/log.hpp 77edc7a 
  src/log/network.hpp d34cf78 
  src/log/replica.hpp d1f5ead 
  src/log/replica.cpp 59a6ff3 
  src/messages/log.proto 3d5859f 
  src/tests/log_tests.cpp ff5f86c 

Diff: https://reviews.apache.org/r/14631/diff/


Testing
-------

bin/mesos-tests.sh --gtest_filter=*CoordinatorTest*:*LogTest*:*ReplicaTest* --gtest_repeat=100


Thanks,

Jie Yu

Re: Review Request 14631: Catch-up Replicated Log 1: decoupled coordinator logics and made them asynchronous.

Posted by Benjamin Hindman <be...@berkeley.edu>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14631/#review28258
-----------------------------------------------------------



src/log/catchup.hpp
<https://reviews.apache.org/r/14631/#comment55002>

    Let's describe the parameters in a bit more detail. In particular, I think that 'proposal' needs some explaining (i.e., why couldn't catchup figure out a proposal number to use itself?).



src/log/catchup.cpp
<https://reviews.apache.org/r/14631/#comment55001>

    Kill "We use this function to make sure that a log position is learned locally.".



src/tests/log_tests.cpp
<https://reviews.apache.org/r/14631/#comment56105>

    How about we just terminate 'replica2' instead? That is actually more in line with what we're trying to test too!
    
    Maybe something like:
    
    process::terminate(replica2->pid());
    process::wait(replica2->pid());
    replica2.reset();


- Benjamin Hindman


On Jan. 16, 2014, 6:37 a.m., Jie Yu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14631/
> -----------------------------------------------------------
> 
> (Updated Jan. 16, 2014, 6:37 a.m.)
> 
> 
> Review request for mesos and Benjamin Hindman.
> 
> 
> Bugs: MESOS-736
>     https://issues.apache.org/jira/browse/MESOS-736
> 
> 
> Repository: mesos-git
> 
> 
> Description
> -------
> 
> This is the first patch of a series of patches that implement a catch-up mechanism for replicated log. See the following ticket for more details:
> https://issues.apache.org/jira/browse/MESOS-736
> 
> Here is a brief summary of this patch: (Sorry for the fact that we are not able to break it into smaller patches :()
> 
> 1) Pulled the original Coordinator logic out and divides it into several Paxos phases (see src/log/consensus.hpp). Instead of using a blocking semantics, we implemented all the logics asynchronously.
> 
> 2) In order to ensure the liveness of a catch-uper, we implemented a retry logic by bumping the proposal number. This also requires us to slightly change the existing replica protocol.
> 
> 3) Made the "fill" operation independent of the underlying replica. Instead, introduced a catchup (see src/log/catchup.hpp) function to make sure the underlying local replica has learned each write.
> 
> 4) Modified the log tests to adapt to the new semantics (see (3) above)
> 
> This is a joint work with Yan Xu.
> 
> 
> Diffs
> -----
> 
>   src/Makefile.am cf0c8c6 
>   src/log/catchup.hpp PRE-CREATION 
>   src/log/catchup.cpp PRE-CREATION 
>   src/log/consensus.hpp PRE-CREATION 
>   src/log/consensus.cpp PRE-CREATION 
>   src/log/coordinator.hpp 3f6fb7c 
>   src/log/coordinator.cpp 6e6466f 
>   src/log/log.hpp 77edc7a 
>   src/log/network.hpp d34cf78 
>   src/log/replica.hpp d1f5ead 
>   src/log/replica.cpp 82c2157 
>   src/messages/log.proto e6460ab 
>   src/tests/log_tests.cpp ff5f86c 
> 
> Diff: https://reviews.apache.org/r/14631/diff/
> 
> 
> Testing
> -------
> 
> bin/mesos-tests.sh --gtest_filter=*CoordinatorTest*:*LogTest*:*ReplicaTest* --gtest_repeat=100
> 
> 
> Thanks,
> 
> Jie Yu
> 
>