You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Jay JN Guo <gu...@cn.ibm.com> on 2016/07/08 09:26:06 UTC
[Replicated Log] Enable Mesos to use etcd for replicated_log
Hi,
We are working on a Mesos module to substitute Zookeeper with Etcd.
Contender and detector are done through modulerized interfaces, however,
replicated_log is still coupled with ZK. Here are my questions:
#1 What's the difference between replicated_log with/without ZK? Without
flag --zk, Log is constructed with hardcoded quorum of 1. Does it assume
master to be running in non-HA mode? Otherwise, we observed that znodes are
created in ZK to store log_replica information, does it help Paxos
coordination in some way?
#2 We hope to make replicated_log pluggable. Some code change need to
happen in Mesos upstream (interface modulerization, extra flags, etc). So
we wonder if someone could shepherd them? Also, it would be great if we
could get some help on better understanding replicated_log internals.
#3 Is there a plan to use replicated_log to do master contend/detect
instead of ZK? If yes, what's the status?
Your help and suggestions are highly appreciated!!
Thanks,
/Jay
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Joseph Wu <jo...@mesosphere.io>.
Jay, I'll shepherd the two detector/contender/network module changes.
On Mon, Jul 11, 2016 at 2:44 AM, Jay JN Guo <gu...@cn.ibm.com> wrote:
>
> Shuai,
>
> I think you are right. AFAIK, replicated_log interacts with zk in only
> following two ways:
>
> 1) replicated_log creates znode and maintain 'membership' via
> zookeeper::Group. It actually stores pid under zk_url/log_replicas.
> 2) replicated_log detects other replicas using pid created in previous step
> and store them in set<UPID> pids [1]. This is done via ZookeeperNetwork.
>
> Other than these, replicas inter-communicate with others through protobuf
> processes. Since there's only one active master at a time, that master will
> be the proposer (coordinator) and others being acceptor. So there is no
> need for leader election in replicated_log.
>
> Please correct me if I'm wrong.
>
> Thanks,
> /J
>
> [1] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L316
>
> Shuai Lin <li...@gmail.com> wrote on 07/11/2016 00:13:12:
>
> > From: Shuai Lin <li...@gmail.com>
> > To: dev <de...@mesos.apache.org>
> > Cc: Jie Yu <ji...@mesosphere.io>, Kapil Arya <ka...@mesosphere.io>
> > Date: 07/11/2016 00:13
> > Subject: Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
> >
> > >
> > > i.e. The MasterContender is the piece that decides the "coordinator" of
> the
> > > replicated log.
> >
> >
> > IINM master contender/detector is not related to replicated logs. The
> only
> > thing they have in common (when using zookeeper) is they both get the
> > zookeeper servers list from the `--zk` flag.
> >
> >
> > On Sat, Jul 9, 2016 at 1:54 AM, Joseph Wu <jo...@mesosphere.io> wrote:
> >
> > > Jay,
> > >
> > > (1) Looks like we missed this when we modularized the
> > > MasterDetector/Contender [1]. We need to expand on src/master/main.cpp
> a
> > > bit.
> > > Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't have
> the
> > > cycles.
> > >
> > > (2) The bit of the replicated log which relies on ZK is a small portion
> > > called the ZookeeperNetwork [2]. The job of this component is to watch
> the
> > > ZK group for membership changes. Log replication messages are
> broadcasted
> > > to all members in this "network abstraction".
> > > This is also a piece that needs to be modularized. (Can you file
> another
> > > bug? :)
> > >
> > > (3) The replicated log is something stored locally on the master (i.e.
> > > LevelDB). The network abstraction has some similarity with the
> > > MasterDetector, but those pieces are otherwise unrelated.
> > > i.e. The MasterContender is the piece that decides the "coordinator" of
> the
> > > replicated log. But the replicated log uses it's own implementation of
> > > Paxos after the coordinator is chosen.
> > >
> > > [1] https://issues.apache.org/jira/browse/MESOS-4610
> > > [2]
> https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
> > >
> > > On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan
> <av...@mesosphere.io>
> > > wrote:
> > >
> > > > +Jie
> > > >
> > > > I think replicated log uses ZK only for leader election. Hence,
> without
> > > ZK
> > > > the quorum is hard-coded to 1.
> > > >
> > > > For (#2), trying to understand what you mean by replicated log being
> > > > pluggable? You mean turning of replicated log on the Master for
> storing
> > > > Registrar information?
> > > >
> > > > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com>
> > > wrote:
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > > > > Contender and detector are done through modulerized interfaces,
> > > however,
> > > > > replicated_log is still coupled with ZK. Here are my questions:
> > > > >
> > > > > #1 What's the difference between replicated_log with/without ZK?
> > > Without
> > > > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
> > > assume
> > > > > master to be running in non-HA mode? Otherwise, we observed that
> znodes
> > > > are
> > > > > created in ZK to store log_replica information, does it help Paxos
> > > > > coordination in some way?
> > > > > #2 We hope to make replicated_log pluggable. Some code change need
> to
> > > > > happen in Mesos upstream (interface modulerization, extra flags,
> etc).
> > > So
> > > > > we wonder if someone could shepherd them? Also, it would be great
> if we
> > > > > could get some help on better understanding replicated_log
> internals.
> > > > > #3 Is there a plan to use replicated_log to do master
> contend/detect
> > > > > instead of ZK? If yes, what's the status?
> > > > >
> > > > > Your help and suggestions are highly appreciated!!
> > > > >
> > > > > Thanks,
> > > > > /Jay
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Avinash Sridharan, Mesosphere
> > > > +1 (323) 702 5245
> > > >
> > >
>
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by tommy xiao <xi...@gmail.com>.
the summary is clear, so the etcd how long to get a run.
2016-07-13 23:40 GMT+08:00 Shuai Lin <li...@gmail.com>:
> Great summary, thx!
>
> On Mon, Jul 11, 2016 at 5:44 PM, Jay JN Guo <gu...@cn.ibm.com> wrote:
>
> >
> > Shuai,
> >
> > I think you are right. AFAIK, replicated_log interacts with zk in only
> > following two ways:
> >
> > 1) replicated_log creates znode and maintain 'membership' via
> > zookeeper::Group. It actually stores pid under zk_url/log_replicas.
> > 2) replicated_log detects other replicas using pid created in previous
> step
> > and store them in set<UPID> pids [1]. This is done via ZookeeperNetwork.
> >
> > Other than these, replicas inter-communicate with others through protobuf
> > processes. Since there's only one active master at a time, that master
> will
> > be the proposer (coordinator) and others being acceptor. So there is no
> > need for leader election in replicated_log.
> >
> > Please correct me if I'm wrong.
> >
> > Thanks,
> > /J
> >
> > [1] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L316
> >
> > Shuai Lin <li...@gmail.com> wrote on 07/11/2016 00:13:12:
> >
> > > From: Shuai Lin <li...@gmail.com>
> > > To: dev <de...@mesos.apache.org>
> > > Cc: Jie Yu <ji...@mesosphere.io>, Kapil Arya <ka...@mesosphere.io>
> > > Date: 07/11/2016 00:13
> > > Subject: Re: [Replicated Log] Enable Mesos to use etcd for
> replicated_log
> > >
> > > >
> > > > i.e. The MasterContender is the piece that decides the "coordinator"
> of
> > the
> > > > replicated log.
> > >
> > >
> > > IINM master contender/detector is not related to replicated logs. The
> > only
> > > thing they have in common (when using zookeeper) is they both get the
> > > zookeeper servers list from the `--zk` flag.
> > >
> > >
> > > On Sat, Jul 9, 2016 at 1:54 AM, Joseph Wu <jo...@mesosphere.io>
> wrote:
> > >
> > > > Jay,
> > > >
> > > > (1) Looks like we missed this when we modularized the
> > > > MasterDetector/Contender [1]. We need to expand on
> src/master/main.cpp
> > a
> > > > bit.
> > > > Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't
> have
> > the
> > > > cycles.
> > > >
> > > > (2) The bit of the replicated log which relies on ZK is a small
> portion
> > > > called the ZookeeperNetwork [2]. The job of this component is to
> watch
> > the
> > > > ZK group for membership changes. Log replication messages are
> > broadcasted
> > > > to all members in this "network abstraction".
> > > > This is also a piece that needs to be modularized. (Can you file
> > another
> > > > bug? :)
> > > >
> > > > (3) The replicated log is something stored locally on the master
> (i.e.
> > > > LevelDB). The network abstraction has some similarity with the
> > > > MasterDetector, but those pieces are otherwise unrelated.
> > > > i.e. The MasterContender is the piece that decides the "coordinator"
> of
> > the
> > > > replicated log. But the replicated log uses it's own implementation
> of
> > > > Paxos after the coordinator is chosen.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/MESOS-4610
> > > > [2]
> > https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
> > > >
> > > > On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan
> > <av...@mesosphere.io>
> > > > wrote:
> > > >
> > > > > +Jie
> > > > >
> > > > > I think replicated log uses ZK only for leader election. Hence,
> > without
> > > > ZK
> > > > > the quorum is hard-coded to 1.
> > > > >
> > > > > For (#2), trying to understand what you mean by replicated log
> being
> > > > > pluggable? You mean turning of replicated log on the Master for
> > storing
> > > > > Registrar information?
> > > > >
> > > > > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com>
> > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are working on a Mesos module to substitute Zookeeper with
> Etcd.
> > > > > > Contender and detector are done through modulerized interfaces,
> > > > however,
> > > > > > replicated_log is still coupled with ZK. Here are my questions:
> > > > > >
> > > > > > #1 What's the difference between replicated_log with/without ZK?
> > > > Without
> > > > > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
> > > > assume
> > > > > > master to be running in non-HA mode? Otherwise, we observed that
> > znodes
> > > > > are
> > > > > > created in ZK to store log_replica information, does it help
> Paxos
> > > > > > coordination in some way?
> > > > > > #2 We hope to make replicated_log pluggable. Some code change
> need
> > to
> > > > > > happen in Mesos upstream (interface modulerization, extra flags,
> > etc).
> > > > So
> > > > > > we wonder if someone could shepherd them? Also, it would be great
> > if we
> > > > > > could get some help on better understanding replicated_log
> > internals.
> > > > > > #3 Is there a plan to use replicated_log to do master
> > contend/detect
> > > > > > instead of ZK? If yes, what's the status?
> > > > > >
> > > > > > Your help and suggestions are highly appreciated!!
> > > > > >
> > > > > > Thanks,
> > > > > > /Jay
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Avinash Sridharan, Mesosphere
> > > > > +1 (323) 702 5245
> > > > >
> > > >
> >
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Shuai Lin <li...@gmail.com>.
Great summary, thx!
On Mon, Jul 11, 2016 at 5:44 PM, Jay JN Guo <gu...@cn.ibm.com> wrote:
>
> Shuai,
>
> I think you are right. AFAIK, replicated_log interacts with zk in only
> following two ways:
>
> 1) replicated_log creates znode and maintain 'membership' via
> zookeeper::Group. It actually stores pid under zk_url/log_replicas.
> 2) replicated_log detects other replicas using pid created in previous step
> and store them in set<UPID> pids [1]. This is done via ZookeeperNetwork.
>
> Other than these, replicas inter-communicate with others through protobuf
> processes. Since there's only one active master at a time, that master will
> be the proposer (coordinator) and others being acceptor. So there is no
> need for leader election in replicated_log.
>
> Please correct me if I'm wrong.
>
> Thanks,
> /J
>
> [1] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L316
>
> Shuai Lin <li...@gmail.com> wrote on 07/11/2016 00:13:12:
>
> > From: Shuai Lin <li...@gmail.com>
> > To: dev <de...@mesos.apache.org>
> > Cc: Jie Yu <ji...@mesosphere.io>, Kapil Arya <ka...@mesosphere.io>
> > Date: 07/11/2016 00:13
> > Subject: Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
> >
> > >
> > > i.e. The MasterContender is the piece that decides the "coordinator" of
> the
> > > replicated log.
> >
> >
> > IINM master contender/detector is not related to replicated logs. The
> only
> > thing they have in common (when using zookeeper) is they both get the
> > zookeeper servers list from the `--zk` flag.
> >
> >
> > On Sat, Jul 9, 2016 at 1:54 AM, Joseph Wu <jo...@mesosphere.io> wrote:
> >
> > > Jay,
> > >
> > > (1) Looks like we missed this when we modularized the
> > > MasterDetector/Contender [1]. We need to expand on src/master/main.cpp
> a
> > > bit.
> > > Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't have
> the
> > > cycles.
> > >
> > > (2) The bit of the replicated log which relies on ZK is a small portion
> > > called the ZookeeperNetwork [2]. The job of this component is to watch
> the
> > > ZK group for membership changes. Log replication messages are
> broadcasted
> > > to all members in this "network abstraction".
> > > This is also a piece that needs to be modularized. (Can you file
> another
> > > bug? :)
> > >
> > > (3) The replicated log is something stored locally on the master (i.e.
> > > LevelDB). The network abstraction has some similarity with the
> > > MasterDetector, but those pieces are otherwise unrelated.
> > > i.e. The MasterContender is the piece that decides the "coordinator" of
> the
> > > replicated log. But the replicated log uses it's own implementation of
> > > Paxos after the coordinator is chosen.
> > >
> > > [1] https://issues.apache.org/jira/browse/MESOS-4610
> > > [2]
> https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
> > >
> > > On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan
> <av...@mesosphere.io>
> > > wrote:
> > >
> > > > +Jie
> > > >
> > > > I think replicated log uses ZK only for leader election. Hence,
> without
> > > ZK
> > > > the quorum is hard-coded to 1.
> > > >
> > > > For (#2), trying to understand what you mean by replicated log being
> > > > pluggable? You mean turning of replicated log on the Master for
> storing
> > > > Registrar information?
> > > >
> > > > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com>
> > > wrote:
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > > > > Contender and detector are done through modulerized interfaces,
> > > however,
> > > > > replicated_log is still coupled with ZK. Here are my questions:
> > > > >
> > > > > #1 What's the difference between replicated_log with/without ZK?
> > > Without
> > > > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
> > > assume
> > > > > master to be running in non-HA mode? Otherwise, we observed that
> znodes
> > > > are
> > > > > created in ZK to store log_replica information, does it help Paxos
> > > > > coordination in some way?
> > > > > #2 We hope to make replicated_log pluggable. Some code change need
> to
> > > > > happen in Mesos upstream (interface modulerization, extra flags,
> etc).
> > > So
> > > > > we wonder if someone could shepherd them? Also, it would be great
> if we
> > > > > could get some help on better understanding replicated_log
> internals.
> > > > > #3 Is there a plan to use replicated_log to do master
> contend/detect
> > > > > instead of ZK? If yes, what's the status?
> > > > >
> > > > > Your help and suggestions are highly appreciated!!
> > > > >
> > > > > Thanks,
> > > > > /Jay
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Avinash Sridharan, Mesosphere
> > > > +1 (323) 702 5245
> > > >
> > >
>
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Jay JN Guo <gu...@cn.ibm.com>.
Shuai,
I think you are right. AFAIK, replicated_log interacts with zk in only
following two ways:
1) replicated_log creates znode and maintain 'membership' via
zookeeper::Group. It actually stores pid under zk_url/log_replicas.
2) replicated_log detects other replicas using pid created in previous step
and store them in set<UPID> pids [1]. This is done via ZookeeperNetwork.
Other than these, replicas inter-communicate with others through protobuf
processes. Since there's only one active master at a time, that master will
be the proposer (coordinator) and others being acceptor. So there is no
need for leader election in replicated_log.
Please correct me if I'm wrong.
Thanks,
/J
[1] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L316
Shuai Lin <li...@gmail.com> wrote on 07/11/2016 00:13:12:
> From: Shuai Lin <li...@gmail.com>
> To: dev <de...@mesos.apache.org>
> Cc: Jie Yu <ji...@mesosphere.io>, Kapil Arya <ka...@mesosphere.io>
> Date: 07/11/2016 00:13
> Subject: Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
>
> >
> > i.e. The MasterContender is the piece that decides the "coordinator" of
the
> > replicated log.
>
>
> IINM master contender/detector is not related to replicated logs. The
only
> thing they have in common (when using zookeeper) is they both get the
> zookeeper servers list from the `--zk` flag.
>
>
> On Sat, Jul 9, 2016 at 1:54 AM, Joseph Wu <jo...@mesosphere.io> wrote:
>
> > Jay,
> >
> > (1) Looks like we missed this when we modularized the
> > MasterDetector/Contender [1]. We need to expand on src/master/main.cpp
a
> > bit.
> > Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't have
the
> > cycles.
> >
> > (2) The bit of the replicated log which relies on ZK is a small portion
> > called the ZookeeperNetwork [2]. The job of this component is to watch
the
> > ZK group for membership changes. Log replication messages are
broadcasted
> > to all members in this "network abstraction".
> > This is also a piece that needs to be modularized. (Can you file
another
> > bug? :)
> >
> > (3) The replicated log is something stored locally on the master (i.e.
> > LevelDB). The network abstraction has some similarity with the
> > MasterDetector, but those pieces are otherwise unrelated.
> > i.e. The MasterContender is the piece that decides the "coordinator" of
the
> > replicated log. But the replicated log uses it's own implementation of
> > Paxos after the coordinator is chosen.
> >
> > [1] https://issues.apache.org/jira/browse/MESOS-4610
> > [2]
https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
> >
> > On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan
<av...@mesosphere.io>
> > wrote:
> >
> > > +Jie
> > >
> > > I think replicated log uses ZK only for leader election. Hence,
without
> > ZK
> > > the quorum is hard-coded to 1.
> > >
> > > For (#2), trying to understand what you mean by replicated log being
> > > pluggable? You mean turning of replicated log on the Master for
storing
> > > Registrar information?
> > >
> > > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com>
> > wrote:
> > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > > > Contender and detector are done through modulerized interfaces,
> > however,
> > > > replicated_log is still coupled with ZK. Here are my questions:
> > > >
> > > > #1 What's the difference between replicated_log with/without ZK?
> > Without
> > > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
> > assume
> > > > master to be running in non-HA mode? Otherwise, we observed that
znodes
> > > are
> > > > created in ZK to store log_replica information, does it help Paxos
> > > > coordination in some way?
> > > > #2 We hope to make replicated_log pluggable. Some code change need
to
> > > > happen in Mesos upstream (interface modulerization, extra flags,
etc).
> > So
> > > > we wonder if someone could shepherd them? Also, it would be great
if we
> > > > could get some help on better understanding replicated_log
internals.
> > > > #3 Is there a plan to use replicated_log to do master
contend/detect
> > > > instead of ZK? If yes, what's the status?
> > > >
> > > > Your help and suggestions are highly appreciated!!
> > > >
> > > > Thanks,
> > > > /Jay
> > > >
> > >
> > >
> > >
> > > --
> > > Avinash Sridharan, Mesosphere
> > > +1 (323) 702 5245
> > >
> >
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Shuai Lin <li...@gmail.com>.
>
> i.e. The MasterContender is the piece that decides the "coordinator" of the
> replicated log.
IINM master contender/detector is not related to replicated logs. The only
thing they have in common (when using zookeeper) is they both get the
zookeeper servers list from the `--zk` flag.
On Sat, Jul 9, 2016 at 1:54 AM, Joseph Wu <jo...@mesosphere.io> wrote:
> Jay,
>
> (1) Looks like we missed this when we modularized the
> MasterDetector/Contender [1]. We need to expand on src/master/main.cpp a
> bit.
> Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't have the
> cycles.
>
> (2) The bit of the replicated log which relies on ZK is a small portion
> called the ZookeeperNetwork [2]. The job of this component is to watch the
> ZK group for membership changes. Log replication messages are broadcasted
> to all members in this "network abstraction".
> This is also a piece that needs to be modularized. (Can you file another
> bug? :)
>
> (3) The replicated log is something stored locally on the master (i.e.
> LevelDB). The network abstraction has some similarity with the
> MasterDetector, but those pieces are otherwise unrelated.
> i.e. The MasterContender is the piece that decides the "coordinator" of the
> replicated log. But the replicated log uses it's own implementation of
> Paxos after the coordinator is chosen.
>
> [1] https://issues.apache.org/jira/browse/MESOS-4610
> [2] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
>
> On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> > +Jie
> >
> > I think replicated log uses ZK only for leader election. Hence, without
> ZK
> > the quorum is hard-coded to 1.
> >
> > For (#2), trying to understand what you mean by replicated log being
> > pluggable? You mean turning of replicated log on the Master for storing
> > Registrar information?
> >
> > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com>
> wrote:
> >
> > >
> > >
> > > Hi,
> > >
> > > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > > Contender and detector are done through modulerized interfaces,
> however,
> > > replicated_log is still coupled with ZK. Here are my questions:
> > >
> > > #1 What's the difference between replicated_log with/without ZK?
> Without
> > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
> assume
> > > master to be running in non-HA mode? Otherwise, we observed that znodes
> > are
> > > created in ZK to store log_replica information, does it help Paxos
> > > coordination in some way?
> > > #2 We hope to make replicated_log pluggable. Some code change need to
> > > happen in Mesos upstream (interface modulerization, extra flags, etc).
> So
> > > we wonder if someone could shepherd them? Also, it would be great if we
> > > could get some help on better understanding replicated_log internals.
> > > #3 Is there a plan to use replicated_log to do master contend/detect
> > > instead of ZK? If yes, what's the status?
> > >
> > > Your help and suggestions are highly appreciated!!
> > >
> > > Thanks,
> > > /Jay
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Jay JN Guo <gu...@cn.ibm.com>.
Hi,
Thanks for your reply! I've created these two JIRA tickets to track the
work:
https://issues.apache.org/jira/browse/MESOS-5829
https://issues.apache.org/jira/browse/MESOS-5828
I've assigned them to myself and I would appreciate a shepherd to work
with.
Here's another question:
I see replicated_log is using Network instead of ZookeeperNetwork while
running in non-HA mode (where --quorum is hardcoded to '1'). However
replicated_log is stated to be coordinated with other replica through a
group of PID [1]. Does this imply that Paxos is running in multi-master
mode (every node assumes itself to be coordinator)?
@Avinash: I should've been more precise and said that replicated_log
Network being pluggable. Basically we want to plug in our own Network
implementation backed by etcd instead of zookeeper.
Thanks!
Jay
[1]
https://github.com/apache/mesos/blob/master/include/mesos/log/log.hpp#L189
Joseph Wu <jo...@mesosphere.io> wrote on 07/09/2016 01:54:14:
> From: Joseph Wu <jo...@mesosphere.io>
> To: dev <de...@mesos.apache.org>
> Cc: Jie Yu <ji...@mesosphere.io>, Kapil Arya <ka...@mesosphere.io>
> Date: 07/09/2016 01:54
> Subject: Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
>
> Jay,
>
> (1) Looks like we missed this when we modularized the
> MasterDetector/Contender [1]. We need to expand on src/master/main.cpp a
> bit.
> Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't have
the
> cycles.
>
> (2) The bit of the replicated log which relies on ZK is a small portion
> called the ZookeeperNetwork [2]. The job of this component is to watch
the
> ZK group for membership changes. Log replication messages are
broadcasted
> to all members in this "network abstraction".
> This is also a piece that needs to be modularized. (Can you file another
> bug? :)
>
> (3) The replicated log is something stored locally on the master (i.e.
> LevelDB). The network abstraction has some similarity with the
> MasterDetector, but those pieces are otherwise unrelated.
> i.e. The MasterContender is the piece that decides the "coordinator" of
the
> replicated log. But the replicated log uses it's own implementation of
> Paxos after the coordinator is chosen.
>
> [1] https://issues.apache.org/jira/browse/MESOS-4610
> [2] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
>
> On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> > +Jie
> >
> > I think replicated log uses ZK only for leader election. Hence, without
ZK
> > the quorum is hard-coded to 1.
> >
> > For (#2), trying to understand what you mean by replicated log being
> > pluggable? You mean turning of replicated log on the Master for storing
> > Registrar information?
> >
> > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com>
wrote:
> >
> > >
> > >
> > > Hi,
> > >
> > > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > > Contender and detector are done through modulerized interfaces,
however,
> > > replicated_log is still coupled with ZK. Here are my questions:
> > >
> > > #1 What's the difference between replicated_log with/without ZK?
Without
> > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
assume
> > > master to be running in non-HA mode? Otherwise, we observed that
znodes
> > are
> > > created in ZK to store log_replica information, does it help Paxos
> > > coordination in some way?
> > > #2 We hope to make replicated_log pluggable. Some code change need to
> > > happen in Mesos upstream (interface modulerization, extra flags,
etc). So
> > > we wonder if someone could shepherd them? Also, it would be great if
we
> > > could get some help on better understanding replicated_log internals.
> > > #3 Is there a plan to use replicated_log to do master contend/detect
> > > instead of ZK? If yes, what's the status?
> > >
> > > Your help and suggestions are highly appreciated!!
> > >
> > > Thanks,
> > > /Jay
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Joseph Wu <jo...@mesosphere.io>.
Jay,
(1) Looks like we missed this when we modularized the
MasterDetector/Contender [1]. We need to expand on src/master/main.cpp a
bit.
Can you file a bug? (cc: Kapil) I can shepherd if Kapil doesn't have the
cycles.
(2) The bit of the replicated log which relies on ZK is a small portion
called the ZookeeperNetwork [2]. The job of this component is to watch the
ZK group for membership changes. Log replication messages are broadcasted
to all members in this "network abstraction".
This is also a piece that needs to be modularized. (Can you file another
bug? :)
(3) The replicated log is something stored locally on the master (i.e.
LevelDB). The network abstraction has some similarity with the
MasterDetector, but those pieces are otherwise unrelated.
i.e. The MasterContender is the piece that decides the "coordinator" of the
replicated log. But the replicated log uses it's own implementation of
Paxos after the coordinator is chosen.
[1] https://issues.apache.org/jira/browse/MESOS-4610
[2] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> +Jie
>
> I think replicated log uses ZK only for leader election. Hence, without ZK
> the quorum is hard-coded to 1.
>
> For (#2), trying to understand what you mean by replicated log being
> pluggable? You mean turning of replicated log on the Master for storing
> Registrar information?
>
> On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com> wrote:
>
> >
> >
> > Hi,
> >
> > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > Contender and detector are done through modulerized interfaces, however,
> > replicated_log is still coupled with ZK. Here are my questions:
> >
> > #1 What's the difference between replicated_log with/without ZK? Without
> > flag --zk, Log is constructed with hardcoded quorum of 1. Does it assume
> > master to be running in non-HA mode? Otherwise, we observed that znodes
> are
> > created in ZK to store log_replica information, does it help Paxos
> > coordination in some way?
> > #2 We hope to make replicated_log pluggable. Some code change need to
> > happen in Mesos upstream (interface modulerization, extra flags, etc). So
> > we wonder if someone could shepherd them? Also, it would be great if we
> > could get some help on better understanding replicated_log internals.
> > #3 Is there a plan to use replicated_log to do master contend/detect
> > instead of ZK? If yes, what's the status?
> >
> > Your help and suggestions are highly appreciated!!
> >
> > Thanks,
> > /Jay
> >
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
Posted by Avinash Sridharan <av...@mesosphere.io>.
+Jie
I think replicated log uses ZK only for leader election. Hence, without ZK
the quorum is hard-coded to 1.
For (#2), trying to understand what you mean by replicated log being
pluggable? You mean turning of replicated log on the Master for storing
Registrar information?
On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo <gu...@cn.ibm.com> wrote:
>
>
> Hi,
>
> We are working on a Mesos module to substitute Zookeeper with Etcd.
> Contender and detector are done through modulerized interfaces, however,
> replicated_log is still coupled with ZK. Here are my questions:
>
> #1 What's the difference between replicated_log with/without ZK? Without
> flag --zk, Log is constructed with hardcoded quorum of 1. Does it assume
> master to be running in non-HA mode? Otherwise, we observed that znodes are
> created in ZK to store log_replica information, does it help Paxos
> coordination in some way?
> #2 We hope to make replicated_log pluggable. Some code change need to
> happen in Mesos upstream (interface modulerization, extra flags, etc). So
> we wonder if someone could shepherd them? Also, it would be great if we
> could get some help on better understanding replicated_log internals.
> #3 Is there a plan to use replicated_log to do master contend/detect
> instead of ZK? If yes, what's the status?
>
> Your help and suggestions are highly appreciated!!
>
> Thanks,
> /Jay
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245