You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Kent Harris <ks...@bvvine.com> on 2016/05/24 19:14:57 UTC
Framework Scheduling on Slave Question
Pardon me if this is a newbie question. I’m trying to port an existing simulation system that is comprised of many processes. The master process is executed by a user (on a Mesos master node) and it has a custom Framework that simply launches a process, call it the “root” process”, via a CommandInfo executor on a slave node.
The root process represents a simulation system and it needs to further schedule many processes that represent various parts of the simulation (communicating via ZMQ primarily). To do this I have a second custom framework invoked by the root process that also uses CommandInfo executor semantics to launch a number of tasks I call agents. As an aside, all agents have a common base structure and a custom plugin structure where plugins represent different types of simulated hardware.
Thus I’m trying to run a framework scheduler on a Mesos slave node. I that possible? Does mesos-slave know how to pass scheduler requests back to a Mesos master node? Does one have to have mesos-master running on slave nodes to do this? Am I smoking bad stuff?
Advice appreciated.
- Kent
Re: Framework Scheduling on Slave Question
Posted by Adam Bordelon <ad...@mesosphere.io>.
> Thus I’m trying to run a framework scheduler on a Mesos slave node. I
that possible?
Absolutely. Marathon, Aurora, Singularity, and other "meta-frameworks" can
be used to launch other framework schedulers as Mesos tasks on agent nodes.
There's no reason you can't do the same thing for your own simulation
system.
> Does mesos-slave know how to pass scheduler requests back to a Mesos
master node?
Not really. Once the mesos-agent process has launched your sub-scheduler
process, it won't be involved in the rest of the scheduler API
communication between the sub-scheduler and mesos-master. To make that
work, the sub-scheduler must be started with information about how to
connect to the master, either via direct IPs, a ZK path, or a mesos-dns
name.
> Does one have to have mesos-master running on slave nodes to do this?
No. The process/container you run from the slave just has to be able to
reach the master, and the master must be able to reach the scheduler
process on its ip:port. You may want to override the auto-detected
LIBPROCESS_IP and LIBPROCESS_PORT, depending on your network configuration.
On Wed, May 25, 2016 at 7:40 AM, Shuai Lin <li...@gmail.com> wrote:
> Does mesos-slave know how to pass scheduler requests back to a Mesos
>> master node? Does one have to have mesos-master running on slave nodes to
>> do this? Am I smoking bad stuff?
>
>
> No problem at all. AFAIK it's a very common practice to have marathon
> running other frameworks.
>
> On Wed, May 25, 2016 at 9:52 AM, haosdent <ha...@gmail.com> wrote:
>
>> Hi, @Kent.
>>
>> > I’m trying to run a framework scheduler on a Mesos slave node.
>> > Does one have to have mesos-master running on slave nodes to do this?
>> If you mean run it manually, your could start your scheduler in any
>> machine, just make sure the network connection works between framework and
>> Mesos master.
>>
>> > Does mesos-slave know how to pass scheduler requests back to a Mesos
>> master node?
>> Framework only could communicate with Mesos Master, could not send
>> message to Mesos Slave directly unless forward via Mesos Master.
>>
>>
>> On Wed, May 25, 2016 at 3:14 AM, Kent Harris <ks...@bvvine.com> wrote:
>>
>>> Pardon me if this is a newbie question. I’m trying to port an existing
>>> simulation system that is comprised of many processes. The master process
>>> is executed by a user (on a Mesos master node) and it has a custom
>>> Framework that simply launches a process, call it the “root” process”, via
>>> a CommandInfo executor on a slave node.
>>>
>>> The root process represents a simulation system and it needs to further
>>> schedule many processes that represent various parts of the simulation
>>> (communicating via ZMQ primarily). To do this I have a second custom
>>> framework invoked by the root process that also uses CommandInfo executor
>>> semantics to launch a number of tasks I call agents. As an aside, all
>>> agents have a common base structure and a custom plugin structure where
>>> plugins represent different types of simulated hardware.
>>>
>>> Thus I’m trying to run a framework scheduler on a Mesos slave node. I
>>> that possible? Does mesos-slave know how to pass scheduler requests back
>>> to a Mesos master node? Does one have to have mesos-master running on
>>> slave nodes to do this? Am I smoking bad stuff?
>>>
>>> Advice appreciated.
>>>
>>> - Kent
>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
Re: Framework Scheduling on Slave Question
Posted by Shuai Lin <li...@gmail.com>.
>
> Does mesos-slave know how to pass scheduler requests back to a Mesos
> master node? Does one have to have mesos-master running on slave nodes to
> do this? Am I smoking bad stuff?
No problem at all. AFAIK it's a very common practice to have marathon
running other frameworks.
On Wed, May 25, 2016 at 9:52 AM, haosdent <ha...@gmail.com> wrote:
> Hi, @Kent.
>
> > I’m trying to run a framework scheduler on a Mesos slave node.
> > Does one have to have mesos-master running on slave nodes to do this?
> If you mean run it manually, your could start your scheduler in any
> machine, just make sure the network connection works between framework and
> Mesos master.
>
> > Does mesos-slave know how to pass scheduler requests back to a Mesos
> master node?
> Framework only could communicate with Mesos Master, could not send message
> to Mesos Slave directly unless forward via Mesos Master.
>
>
> On Wed, May 25, 2016 at 3:14 AM, Kent Harris <ks...@bvvine.com> wrote:
>
>> Pardon me if this is a newbie question. I’m trying to port an existing
>> simulation system that is comprised of many processes. The master process
>> is executed by a user (on a Mesos master node) and it has a custom
>> Framework that simply launches a process, call it the “root” process”, via
>> a CommandInfo executor on a slave node.
>>
>> The root process represents a simulation system and it needs to further
>> schedule many processes that represent various parts of the simulation
>> (communicating via ZMQ primarily). To do this I have a second custom
>> framework invoked by the root process that also uses CommandInfo executor
>> semantics to launch a number of tasks I call agents. As an aside, all
>> agents have a common base structure and a custom plugin structure where
>> plugins represent different types of simulated hardware.
>>
>> Thus I’m trying to run a framework scheduler on a Mesos slave node. I
>> that possible? Does mesos-slave know how to pass scheduler requests back
>> to a Mesos master node? Does one have to have mesos-master running on
>> slave nodes to do this? Am I smoking bad stuff?
>>
>> Advice appreciated.
>>
>> - Kent
>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Framework Scheduling on Slave Question
Posted by haosdent <ha...@gmail.com>.
Hi, @Kent.
> I’m trying to run a framework scheduler on a Mesos slave node.
> Does one have to have mesos-master running on slave nodes to do this?
If you mean run it manually, your could start your scheduler in any
machine, just make sure the network connection works between framework and
Mesos master.
> Does mesos-slave know how to pass scheduler requests back to a Mesos
master node?
Framework only could communicate with Mesos Master, could not send message
to Mesos Slave directly unless forward via Mesos Master.
On Wed, May 25, 2016 at 3:14 AM, Kent Harris <ks...@bvvine.com> wrote:
> Pardon me if this is a newbie question. I’m trying to port an existing
> simulation system that is comprised of many processes. The master process
> is executed by a user (on a Mesos master node) and it has a custom
> Framework that simply launches a process, call it the “root” process”, via
> a CommandInfo executor on a slave node.
>
> The root process represents a simulation system and it needs to further
> schedule many processes that represent various parts of the simulation
> (communicating via ZMQ primarily). To do this I have a second custom
> framework invoked by the root process that also uses CommandInfo executor
> semantics to launch a number of tasks I call agents. As an aside, all
> agents have a common base structure and a custom plugin structure where
> plugins represent different types of simulated hardware.
>
> Thus I’m trying to run a framework scheduler on a Mesos slave node. I
> that possible? Does mesos-slave know how to pass scheduler requests back
> to a Mesos master node? Does one have to have mesos-master running on
> slave nodes to do this? Am I smoking bad stuff?
>
> Advice appreciated.
>
> - Kent
--
Best Regards,
Haosdent Huang