You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Elouan Keryell-Even <el...@gmail.com> on 2015/11/27 16:16:44 UTC
Multiple active masters
Hi,
I don't think this is possible yet, but is it planned some day to be able
to work with multiple active Mesos masters?
I mean, not just a high-availability configuration with one elected master
and a few others idle masters (in case of failure), but really two masters
exploiting a shared pool of slaves.
Our use case:
We are using Mesos to manage resources of two separated clusters. For now,
one cluster acts as a "commandment center" (since it runs the Mesos
master), while the other only runs mesos slaves.
We would want to have a perfectly symetric architecture, where each cluster
could borrow a few slaves from the other cluster to run some of his jobs.
In my mind that implies each slave has to be managed by two mesos masters
at the same time.
I'm curious to have your opinion on this :)
Elouan Keryell-Even
Software Engineer @ Atos Integration
Toulouse, France
+33 6 64 61 29 56
Re: Multiple active masters
Posted by Klaus Ma <kl...@gmail.com>.
That's interesting :). but as far as I known, there's no such migration
feature in Mesos. Maybe you can start an EPIC for this requirement.
----
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
On Fri, Nov 27, 2015 at 11:16 PM, Elouan Keryell-Even <
elouan.keryell@gmail.com> wrote:
> Hi,
>
> I don't think this is possible yet, but is it planned some day to be able
> to work with multiple active Mesos masters?
>
> I mean, not just a high-availability configuration with one elected master
> and a few others idle masters (in case of failure), but really two masters
> exploiting a shared pool of slaves.
>
>
> Our use case:
>
> We are using Mesos to manage resources of two separated clusters. For now,
> one cluster acts as a "commandment center" (since it runs the Mesos
> master), while the other only runs mesos slaves.
>
> We would want to have a perfectly symetric architecture, where each
> cluster could borrow a few slaves from the other cluster to run some of his
> jobs.
>
> In my mind that implies each slave has to be managed by two mesos masters
> at the same time.
>
>
> I'm curious to have your opinion on this :)
>
> Elouan Keryell-Even
> Software Engineer @ Atos Integration
> Toulouse, France
> +33 6 64 61 29 56
>
Re: Multiple active masters
Posted by Guangya Liu <gy...@gmail.com>.
There is a JIRA ticket here https://issues.apache.org/jira/browse/MESOS-3548
, you can possibly append your thinking and use case there to push this
move forward? Thanks!
On Fri, Nov 27, 2015 at 11:16 PM, Elouan Keryell-Even <
elouan.keryell@gmail.com> wrote:
> Hi,
>
> I don't think this is possible yet, but is it planned some day to be able
> to work with multiple active Mesos masters?
>
> I mean, not just a high-availability configuration with one elected master
> and a few others idle masters (in case of failure), but really two masters
> exploiting a shared pool of slaves.
>
>
> Our use case:
>
> We are using Mesos to manage resources of two separated clusters. For now,
> one cluster acts as a "commandment center" (since it runs the Mesos
> master), while the other only runs mesos slaves.
>
> We would want to have a perfectly symetric architecture, where each
> cluster could borrow a few slaves from the other cluster to run some of his
> jobs.
>
> In my mind that implies each slave has to be managed by two mesos masters
> at the same time.
>
>
> I'm curious to have your opinion on this :)
>
> Elouan Keryell-Even
> Software Engineer @ Atos Integration
> Toulouse, France
> +33 6 64 61 29 56
>
Re: Multiple active masters
Posted by Elouan Keryell-Even <el...@gmail.com>.
@tommy Xiao the goal of our R&D work is precisely to have distant clusters
collaborating together, so we aren't really interested in having both
clusters on the same data center.
2015-11-30 14:26 GMT+01:00 Klaus Ma <kl...@gmail.com>:
> If the running two agent in slave host, the two clusters will share
> resource all the time. One option I can image is to use meta framework to
> connect to the two cluster, and balance the resources between the two
> cluster.
>
In our use case both clusters are to collaborate in a decentralized way.
One cluster could occasionally connect to the other to schedule some of his
jobs to run there, but in a normal situation, each side would only run jobs
locally with its own scheduling capabilities.
Thus, a centralized meta-cluster running on top of everything and
supporting all the scheduling is not really what we are interested in. I
see it more like a brief extension of the clusters' scheduling power.
>
> ----
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform Symphony/DCOS Development & Support, STG, IBM GCG
> +86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
>
> On Mon, Nov 30, 2015 at 8:01 PM, tommy xiao <xi...@gmail.com> wrote:
>
>> run two slave in same node is bad idea on production env. i only say, why
>> not install two cluster in same DC?
>>
>> 2015-11-30 14:20 GMT+08:00 Chengwei Yang <ch...@gmail.com>:
>>
>>> On Fri, Nov 27, 2015 at 04:16:44PM +0100, Elouan Keryell-Even wrote:
>>> > Hi,
>>> >
>>> > I don't think this is possible yet, but is it planned some day to be
>>> able to
>>> > work with multiple active Mesos masters?
>>> >
>>> > I mean, not just a high-availability configuration with one elected
>>> master and
>>> > a few others idle masters (in case of failure), but really two masters
>>> > exploiting a shared pool of slaves.
>>> >
>>> >
>>> > Our use case:
>>> >
>>> > We are using Mesos to manage resources of two separated clusters. For
>>> now, one
>>> > cluster acts as a "commandment center" (since it runs the Mesos
>>> master), while
>>> > the other only runs mesos slaves.
>>> >
>>> > We would want to have a perfectly symetric architecture, where each
>>> cluster
>>> > could borrow a few slaves from the other cluster to run some of his
>>> jobs.
>>> >
>>> > In my mind that implies each slave has to be managed by two mesos
>>> masters at
>>> > the same time.
>>>
>>> Try run two mesos-slave on the same node though I couldn't buy-in your
>>> use case?
>>>
>>> Of course, you have to configure different LIBPROCESS_PORT to avoid
>>> conflict
>>> and any other options you want, I didn't do this so I'm not sure if this
>>> works.
>>>
>>
Doesn't that imply to statically split the computing resources of the slave?
Anyway, thank you all very much for your feedback, it is really appreciated
:)
Elouan Keryell-Even
Software Engineer @ Atos Integration
Toulouse, France
>>>
>>> --
>>> Thanks,
>>> Chengwei
>>>
>>> >
>>> >
>>> > I'm curious to have your opinion on this :)
>>> >
>>> > Elouan Keryell-Even
>>> > Software Engineer @ Atos Integration
>>> > Toulouse, France
>>> > SECURITY NOTE: file ~/.netrc must not be accessible by others
>>>
>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>>
>
>
Re: Multiple active masters
Posted by Klaus Ma <kl...@gmail.com>.
If the running two agent in slave host, the two clusters will share
resource all the time. One option I can image is to use meta framework to
connect to the two cluster, and balance the resources between the two
cluster.
----
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982.cn@gmail.com | http://k82.me
On Mon, Nov 30, 2015 at 8:01 PM, tommy xiao <xi...@gmail.com> wrote:
> run two slave in same node is bad idea on production env. i only say, why
> not install two cluster in same DC?
>
> 2015-11-30 14:20 GMT+08:00 Chengwei Yang <ch...@gmail.com>:
>
>> On Fri, Nov 27, 2015 at 04:16:44PM +0100, Elouan Keryell-Even wrote:
>> > Hi,
>> >
>> > I don't think this is possible yet, but is it planned some day to be
>> able to
>> > work with multiple active Mesos masters?
>> >
>> > I mean, not just a high-availability configuration with one elected
>> master and
>> > a few others idle masters (in case of failure), but really two masters
>> > exploiting a shared pool of slaves.
>> >
>> >
>> > Our use case:
>> >
>> > We are using Mesos to manage resources of two separated clusters. For
>> now, one
>> > cluster acts as a "commandment center" (since it runs the Mesos
>> master), while
>> > the other only runs mesos slaves.
>> >
>> > We would want to have a perfectly symetric architecture, where each
>> cluster
>> > could borrow a few slaves from the other cluster to run some of his
>> jobs.
>> >
>> > In my mind that implies each slave has to be managed by two mesos
>> masters at
>> > the same time.
>>
>> Try run two mesos-slave on the same node though I couldn't buy-in your
>> use case?
>>
>> Of course, you have to configure different LIBPROCESS_PORT to avoid
>> conflict
>> and any other options you want, I didn't do this so I'm not sure if this
>> works.
>>
>>
>> --
>> Thanks,
>> Chengwei
>>
>> >
>> >
>> > I'm curious to have your opinion on this :)
>> >
>> > Elouan Keryell-Even
>> > Software Engineer @ Atos Integration
>> > Toulouse, France
>> > +33 6 64 61 29 56
>> > SECURITY NOTE: file ~/.netrc must not be accessible by others
>>
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>
Re: Multiple active masters
Posted by tommy xiao <xi...@gmail.com>.
run two slave in same node is bad idea on production env. i only say, why
not install two cluster in same DC?
2015-11-30 14:20 GMT+08:00 Chengwei Yang <ch...@gmail.com>:
> On Fri, Nov 27, 2015 at 04:16:44PM +0100, Elouan Keryell-Even wrote:
> > Hi,
> >
> > I don't think this is possible yet, but is it planned some day to be
> able to
> > work with multiple active Mesos masters?
> >
> > I mean, not just a high-availability configuration with one elected
> master and
> > a few others idle masters (in case of failure), but really two masters
> > exploiting a shared pool of slaves.
> >
> >
> > Our use case:
> >
> > We are using Mesos to manage resources of two separated clusters. For
> now, one
> > cluster acts as a "commandment center" (since it runs the Mesos master),
> while
> > the other only runs mesos slaves.
> >
> > We would want to have a perfectly symetric architecture, where each
> cluster
> > could borrow a few slaves from the other cluster to run some of his jobs.
> >
> > In my mind that implies each slave has to be managed by two mesos
> masters at
> > the same time.
>
> Try run two mesos-slave on the same node though I couldn't buy-in your use
> case?
>
> Of course, you have to configure different LIBPROCESS_PORT to avoid
> conflict
> and any other options you want, I didn't do this so I'm not sure if this
> works.
>
>
> --
> Thanks,
> Chengwei
>
> >
> >
> > I'm curious to have your opinion on this :)
> >
> > Elouan Keryell-Even
> > Software Engineer @ Atos Integration
> > Toulouse, France
> > +33 6 64 61 29 56
> > SECURITY NOTE: file ~/.netrc must not be accessible by others
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Multiple active masters
Posted by Chengwei Yang <ch...@gmail.com>.
On Fri, Nov 27, 2015 at 04:16:44PM +0100, Elouan Keryell-Even wrote:
> Hi,
>
> I don't think this is possible yet, but is it planned some day to be able to
> work with multiple active Mesos masters?
>
> I mean, not just a high-availability configuration with one elected master and
> a few others idle masters (in case of failure), but really two masters
> exploiting a shared pool of slaves.
>
>
> Our use case:
>
> We are using Mesos to manage resources of two separated clusters. For now, one
> cluster acts as a "commandment center" (since it runs the Mesos master), while
> the other only runs mesos slaves.
>
> We would want to have a perfectly symetric architecture, where each cluster
> could borrow a few slaves from the other cluster to run some of his jobs.
>
> In my mind that implies each slave has to be managed by two mesos masters at
> the same time.
Try run two mesos-slave on the same node though I couldn't buy-in your use case?
Of course, you have to configure different LIBPROCESS_PORT to avoid conflict
and any other options you want, I didn't do this so I'm not sure if this works.
--
Thanks,
Chengwei
>
>
> I'm curious to have your opinion on this :)
>
> Elouan Keryell-Even
> Software Engineer @ Atos Integration
> Toulouse, France
> +33 6 64 61 29 56
> SECURITY NOTE: file ~/.netrc must not be accessible by others