You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Zhitao Li <zh...@gmail.com> on 2018/01/16 18:20:53 UTC
Sharing master detector to modules
Hi,
Some of our future development work on our custom modules requires them to
know the current leader of Mesos master. While it seems like we could
duplicate the logic in master/slave side to duplicate an instance of
`MasterDetector`, it seems more natural if we could figure out a clean way
to share the existing detector instance in slave to the module.
Any comment on this idea?
Thanks.
--
Cheers,
Zhitao Li
Re: Sharing master detector to modules
Posted by Ilya Pronin <ip...@twopensource.com>.
If the module needs to detect the same master as the agent does, I think
this feature can be implemented as a module hook.
On Tue, Jan 16, 2018 at 1:48 PM, Benjamin Mahler <bm...@apache.org> wrote:
> Just a note that if you increase the number of instances of detectors you
> wind up increasing the number of instances of our `Zookeeper` client class.
> This class was written a long time ago and so is still a blocking interface
> rather than a Future-based non-blocking interface. As a result, the minimum
> number of threads needed for libprocess to avoid deadlocking will increase:
>
> https://issues.apache.org/jira/browse/MESOS-8255
>
> On Tue, Jan 16, 2018 at 1:32 PM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> > That's correct. The number of watchers here would be doubled. Also, note
> > that the agent `MasterDetector` is actually checking the leader amongst
> the
> > `overlay-master` modules and not the Mesos master itself. Since these are
> > anonymous modules there is no way to share the slave/masters objects with
> > the modules (atleast the intention is not to have any sharing). That
> said,
> > we haven't seen any real impact on Zookeeper due to these extra
> > detectors/watchers being in place in practice. However, I would like to
> > post a disclaimer that this statement is more anecdotal rather than based
> > on some specific experiments. We haven't done any benchmarking ourselves
> to
> > verify the exact increase in load with these extra detectors in place.
> >
> > One thing to keep in mind is that Master failovers are relatively rare
> > events (or atleast they should be in a stable system). This implies that
> > while you might see a spike in the load for zookeeper watchers when the
> > event does occur, this is not something that should stress zookeeper out.
> >
> > On Tue, Jan 16, 2018 at 1:15 PM, Zhitao Li <zh...@gmail.com>
> wrote:
> >
> > > Hi Avinash,
> > >
> > > Thanks for the pointer. A quick scan seems to suggest that your
> > > implementation requires it to create two different instances of
> > > `MasterDetector`, one for master/agent itself, and one for your module.
> > > That probably means number of watchers on zookeeper would be doubled?
> > Would
> > > that create unnecessary load?
> > >
> > >
> > >
> > > On Tue, Jan 16, 2018 at 11:22 AM, Avinash Sridharan <
> > avinash@mesosphere.io
> > > >
> > > wrote:
> > >
> > > > Hi Zhitao,
> > > > We actually do this in the `DC/OS` overlay modules we use in DC/OS.
> > The
> > > > overlay modules run as Mesos modules both in the Master and the
> Agent.
> > > You
> > > > can see how we use the `MasterDetector` in the agent module:
> > > > https://github.com/dcos/dcos-mesos-modules/blob/master/
> > > > overlay/agent.cpp#L74
> > > >
> > > > https://github.com/dcos/dcos-mesos-modules/blob/master/overlay/
> should
> > > > have
> > > > more details of how the modules are actually used to build DC/OS
> > overlay.
> > > >
> > > > On Tue, Jan 16, 2018 at 10:20 AM, Zhitao Li <zh...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Some of our future development work on our custom modules requires
> > them
> > > > to
> > > > > know the current leader of Mesos master. While it seems like we
> could
> > > > > duplicate the logic in master/slave side to duplicate an instance
> of
> > > > > `MasterDetector`, it seems more natural if we could figure out a
> > clean
> > > > way
> > > > > to share the existing detector instance in slave to the module.
> > > > >
> > > > > Any comment on this idea?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > --
> > > > > Cheers,
> > > > >
> > > > > Zhitao Li
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Avinash Sridharan, Mesosphere
> > > > +1 (323) 702 5245
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > >
> > > Zhitao Li
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
--
Ilya Pronin
Re: Sharing master detector to modules
Posted by Benjamin Mahler <bm...@apache.org>.
Just a note that if you increase the number of instances of detectors you
wind up increasing the number of instances of our `Zookeeper` client class.
This class was written a long time ago and so is still a blocking interface
rather than a Future-based non-blocking interface. As a result, the minimum
number of threads needed for libprocess to avoid deadlocking will increase:
https://issues.apache.org/jira/browse/MESOS-8255
On Tue, Jan 16, 2018 at 1:32 PM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> That's correct. The number of watchers here would be doubled. Also, note
> that the agent `MasterDetector` is actually checking the leader amongst the
> `overlay-master` modules and not the Mesos master itself. Since these are
> anonymous modules there is no way to share the slave/masters objects with
> the modules (atleast the intention is not to have any sharing). That said,
> we haven't seen any real impact on Zookeeper due to these extra
> detectors/watchers being in place in practice. However, I would like to
> post a disclaimer that this statement is more anecdotal rather than based
> on some specific experiments. We haven't done any benchmarking ourselves to
> verify the exact increase in load with these extra detectors in place.
>
> One thing to keep in mind is that Master failovers are relatively rare
> events (or atleast they should be in a stable system). This implies that
> while you might see a spike in the load for zookeeper watchers when the
> event does occur, this is not something that should stress zookeeper out.
>
> On Tue, Jan 16, 2018 at 1:15 PM, Zhitao Li <zh...@gmail.com> wrote:
>
> > Hi Avinash,
> >
> > Thanks for the pointer. A quick scan seems to suggest that your
> > implementation requires it to create two different instances of
> > `MasterDetector`, one for master/agent itself, and one for your module.
> > That probably means number of watchers on zookeeper would be doubled?
> Would
> > that create unnecessary load?
> >
> >
> >
> > On Tue, Jan 16, 2018 at 11:22 AM, Avinash Sridharan <
> avinash@mesosphere.io
> > >
> > wrote:
> >
> > > Hi Zhitao,
> > > We actually do this in the `DC/OS` overlay modules we use in DC/OS.
> The
> > > overlay modules run as Mesos modules both in the Master and the Agent.
> > You
> > > can see how we use the `MasterDetector` in the agent module:
> > > https://github.com/dcos/dcos-mesos-modules/blob/master/
> > > overlay/agent.cpp#L74
> > >
> > > https://github.com/dcos/dcos-mesos-modules/blob/master/overlay/ should
> > > have
> > > more details of how the modules are actually used to build DC/OS
> overlay.
> > >
> > > On Tue, Jan 16, 2018 at 10:20 AM, Zhitao Li <zh...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Some of our future development work on our custom modules requires
> them
> > > to
> > > > know the current leader of Mesos master. While it seems like we could
> > > > duplicate the logic in master/slave side to duplicate an instance of
> > > > `MasterDetector`, it seems more natural if we could figure out a
> clean
> > > way
> > > > to share the existing detector instance in slave to the module.
> > > >
> > > > Any comment on this idea?
> > > >
> > > > Thanks.
> > > >
> > > > --
> > > > Cheers,
> > > >
> > > > Zhitao Li
> > > >
> > >
> > >
> > >
> > > --
> > > Avinash Sridharan, Mesosphere
> > > +1 (323) 702 5245
> > >
> >
> >
> >
> > --
> > Cheers,
> >
> > Zhitao Li
> >
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
Re: Sharing master detector to modules
Posted by Avinash Sridharan <av...@mesosphere.io>.
That's correct. The number of watchers here would be doubled. Also, note
that the agent `MasterDetector` is actually checking the leader amongst the
`overlay-master` modules and not the Mesos master itself. Since these are
anonymous modules there is no way to share the slave/masters objects with
the modules (atleast the intention is not to have any sharing). That said,
we haven't seen any real impact on Zookeeper due to these extra
detectors/watchers being in place in practice. However, I would like to
post a disclaimer that this statement is more anecdotal rather than based
on some specific experiments. We haven't done any benchmarking ourselves to
verify the exact increase in load with these extra detectors in place.
One thing to keep in mind is that Master failovers are relatively rare
events (or atleast they should be in a stable system). This implies that
while you might see a spike in the load for zookeeper watchers when the
event does occur, this is not something that should stress zookeeper out.
On Tue, Jan 16, 2018 at 1:15 PM, Zhitao Li <zh...@gmail.com> wrote:
> Hi Avinash,
>
> Thanks for the pointer. A quick scan seems to suggest that your
> implementation requires it to create two different instances of
> `MasterDetector`, one for master/agent itself, and one for your module.
> That probably means number of watchers on zookeeper would be doubled? Would
> that create unnecessary load?
>
>
>
> On Tue, Jan 16, 2018 at 11:22 AM, Avinash Sridharan <avinash@mesosphere.io
> >
> wrote:
>
> > Hi Zhitao,
> > We actually do this in the `DC/OS` overlay modules we use in DC/OS. The
> > overlay modules run as Mesos modules both in the Master and the Agent.
> You
> > can see how we use the `MasterDetector` in the agent module:
> > https://github.com/dcos/dcos-mesos-modules/blob/master/
> > overlay/agent.cpp#L74
> >
> > https://github.com/dcos/dcos-mesos-modules/blob/master/overlay/ should
> > have
> > more details of how the modules are actually used to build DC/OS overlay.
> >
> > On Tue, Jan 16, 2018 at 10:20 AM, Zhitao Li <zh...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Some of our future development work on our custom modules requires them
> > to
> > > know the current leader of Mesos master. While it seems like we could
> > > duplicate the logic in master/slave side to duplicate an instance of
> > > `MasterDetector`, it seems more natural if we could figure out a clean
> > way
> > > to share the existing detector instance in slave to the module.
> > >
> > > Any comment on this idea?
> > >
> > > Thanks.
> > >
> > > --
> > > Cheers,
> > >
> > > Zhitao Li
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Sharing master detector to modules
Posted by Zhitao Li <zh...@gmail.com>.
Hi Avinash,
Thanks for the pointer. A quick scan seems to suggest that your
implementation requires it to create two different instances of
`MasterDetector`, one for master/agent itself, and one for your module.
That probably means number of watchers on zookeeper would be doubled? Would
that create unnecessary load?
On Tue, Jan 16, 2018 at 11:22 AM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> Hi Zhitao,
> We actually do this in the `DC/OS` overlay modules we use in DC/OS. The
> overlay modules run as Mesos modules both in the Master and the Agent. You
> can see how we use the `MasterDetector` in the agent module:
> https://github.com/dcos/dcos-mesos-modules/blob/master/
> overlay/agent.cpp#L74
>
> https://github.com/dcos/dcos-mesos-modules/blob/master/overlay/ should
> have
> more details of how the modules are actually used to build DC/OS overlay.
>
> On Tue, Jan 16, 2018 at 10:20 AM, Zhitao Li <zh...@gmail.com> wrote:
>
> > Hi,
> >
> > Some of our future development work on our custom modules requires them
> to
> > know the current leader of Mesos master. While it seems like we could
> > duplicate the logic in master/slave side to duplicate an instance of
> > `MasterDetector`, it seems more natural if we could figure out a clean
> way
> > to share the existing detector instance in slave to the module.
> >
> > Any comment on this idea?
> >
> > Thanks.
> >
> > --
> > Cheers,
> >
> > Zhitao Li
> >
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
--
Cheers,
Zhitao Li
Re: Sharing master detector to modules
Posted by Avinash Sridharan <av...@mesosphere.io>.
Hi Zhitao,
We actually do this in the `DC/OS` overlay modules we use in DC/OS. The
overlay modules run as Mesos modules both in the Master and the Agent. You
can see how we use the `MasterDetector` in the agent module:
https://github.com/dcos/dcos-mesos-modules/blob/master/overlay/agent.cpp#L74
https://github.com/dcos/dcos-mesos-modules/blob/master/overlay/ should have
more details of how the modules are actually used to build DC/OS overlay.
On Tue, Jan 16, 2018 at 10:20 AM, Zhitao Li <zh...@gmail.com> wrote:
> Hi,
>
> Some of our future development work on our custom modules requires them to
> know the current leader of Mesos master. While it seems like we could
> duplicate the logic in master/slave side to duplicate an instance of
> `MasterDetector`, it seems more natural if we could figure out a clean way
> to share the existing detector instance in slave to the module.
>
> Any comment on this idea?
>
> Thanks.
>
> --
> Cheers,
>
> Zhitao Li
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245