You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by "Miller, Mark" <mm...@sdsc.edu> on 2016/10/25 15:05:29 UTC
mesos and moving jobs between clusters
Hi all,
I posed a question to Suresh (see below), and he asked me to put this question on the dev list.
So here it is. I will be grateful for any comments about the issues you all are facing, and what has come up in trying this, as
It seems likely that this is a much simpler problem in concept than it is in practice, but its solution has many benefits.
Here is my question:
A group of us have been discussing how we might simplify submitting jobs to different compute resources in our current implementation of CIPRES, and how cloud computing might facilitate this. But none of us are cloud experts. As I understand it, the mesos cluster that I have been seeing in the Airavata email threads is intended to make it possible to deploy jobs to multiple virtual clusters. I am (we are) wondering if Mesos manages submissions to identical virtual clusters on multiple machines, and if that works efficiently.
In our implementation, we have to change the rules to run efficiently on different machines, according to gpu availability, and cores per node. I am wondering how Mesos/ virtual clusters affect those considerations.
Can mesos create basically identical virtual clusters independent of machine?
Thanks for any advice.
Mark
Re: mesos and moving jobs between clusters
Posted by Suresh Marru <sm...@apache.org>.
> On Nov 1, 2016, at 9:35 AM, Madhusudhan Govindaraju <mg...@binghamton.edu> wrote:
>
>
> Hello Mangirish,
>
> Here is the text from Aurora's github page:
>
> ---
> When and when not to use Aurora
> Aurora can take over for most uses of software ... However, if you have very specific scheduling requirements, or are building a system that looks like a scheduler itself, you may want to explore developing your own framework.
> ---
> We believe Airavata will need a framework to customize scheduling policies for different communities, and so instead of making big changes in Aurora, we want to develop our own framework.
>
Hi Madhu, Pankaj,
New contributions and directions will be greatly welcome. Writing a new framework as a proof of concept or as an academic effort does not need much discussion, but if the goal is to deliver on a “production ready” scheduler then I think we need significant discussion and assessment of what the currents schedulers lack. Job management has lot of corner cases and it will take some collective substantial effort to work on the last 20%. I suggest the following steps to make sure every one in the community comes along and participates:
* Start with an architecture mailing list discussion on high level goals, shortcomings in current schedulers, why writing a new scheduler is justified over extending or contributing to existing ones. An example thread on a related topic - http://airavata.markmail.org/thread/f3ncoxyarateyn4y <http://airavata.markmail.org/thread/f3ncoxyarateyn4y> another on workflows - http://markmail.org/thread/tkpbj3sr4jhg6o6z <http://markmail.org/thread/tkpbj3sr4jhg6o6z>, an example on use of Zookeeper in Airavata - http://airavata.markmail.org/thread/sdidqqf4czprmpik <http://airavata.markmail.org/thread/sdidqqf4czprmpik>.
* Once we have a consensus on the architectural approaches, it will be great to do a design discussion on airavata dev list.
* Develop the scheduler from scratch on the mailing lists and constantly seek inputs and early users to try. The onus is on the contributor to some how interest from the community.
I can understand how laborious all of this sounds, but there are many dormant observers on dev and architecture lists and a good topic awakens them. Airavata strives on such volunteer intellectual contributions which is above and beyond the direct code contributions.
> Once you or Gourav-Shenoy have Airavata working with Aurora/Mesos, the idea is that Pankaj will work with you to use the same codebase/task-module in Airavata to launch jobs on Mesos using a custom framework.
>
The thrift client to Aurora is in a working state - https://github.com/apache/airavata/tree/develop/modules/cloud/aurora-client <https://github.com/apache/airavata/tree/develop/modules/cloud/aurora-client> the integration with Airavata is on two ends. Have the statuses pushed into registry (this should be ready by Thursday - I plan to demo it at the gateways workshop). There may be some hard wirings on Aurora end points and so on, which we need to integrate with App Catalog. This might have to wait until we gain better understand on Aurora.
Suresh
> -Madhu
>
>
> On 10/28/2016 12:46 PM, Mangirish Wagle wrote:
>> Hi Pankaj,
>>
>> I was curious to know what is your motivation to work on developing a custom framework and not use Aurora or any existing robust frameworks. It would be great if you could share some pointers on that.
>> I would also like to know what specific use cases you are targeting through your framework, as well as what are various stability concerns that you may have identified and how are you planning to handle them?
>>
>> Regards,
>> Mangirish
>>
>>
>>
>>
>>
>> On Tue, Oct 25, 2016 at 6:09 PM, Pankaj Saha <psaha4@binghamton.edu <ma...@binghamton.edu>> wrote:
>> Hi Mark,
>>
>> Mesos collects the resource information from all the nodes in the cluster (cores, memory, disk, and gpu) and presents a unified view, as if it is a single operating system. The Mesosphere, who a commercial entity for Mesos, has built an ecosystem around Mesos as the kernel called the "Data Center Operating System (DCOS)". Frameworks interact with Mesos to reserve resources and then use these resources to run jobs on the cluster. So, for example, if multiple frameworks such as Marathon, Apache Aurora, and a custom-MPI-framework are using Mesos, then there is a negotiation between Mesos and each framework on how many resources each framework gets. Once the framework, say Aurora, gets resources, it can decide how to use those resources. Some of the strengths of Mesos include fault tolerance at sca
>> l
>> e and the ability to co-schedule applications/frameworks on the cluster such that cluster utilization is high.
>>
>> Mesos off-the-shelf only works when the Mater and agent nodes have a line of communication to each other. We have worked on modifying the Mesos installation so that it even works when agents are behind firewalls on campus clusters. We are also working on getting the same setup to work on Jetstream and Chameleon where allocations are a mix of public IPs and internally accessible nodes. This will allow us to use Mesos to meta-schedule across clusters. We are also developing our own framework, to be able to customize scheduling and resource negotiations for science gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's team so that it works with Airavata.
>>
>> I will be presenting at the Gateways workshop in November, and then I will also be at SC along with my adviser (Madhu Govindaraju), if you would like to discuss any of these projects.
>>
>> We are working on packaging our work so that it can be shared with this community.
>>
>> Thanks
>> Pankaj
>>
>> On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle < <ma...@gmail.com>vaglomangirish@gmail.com <ma...@gmail.com>> wrote:
>> Hi Mark,
>>
>> Thanks for your question. So if I understand you correctly, you need kind of load balancing between identical clusters through a single Mesos master?
>>
>> With the current setup, from what I understand, we have a separate mesos masters for every cluster on separate clouds. However, its a good investigative topic if we can have single mesos master targeting multiple identical clusters. We have some work ongoing to use a virtual cluster setup with compute resources across clouds to install mesos, but not sure if that is what you are looking for.
>>
>> Regards,
>> Mangirish
>>
>>
>>
>>
>>
>> On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mmiller@sdsc.edu <ma...@sdsc.edu>> wrote:
>> Hi all,
>>
>>
>> I posed a question to Suresh (see below), and he asked me to put this question on the dev list.
>>
>> So here it is. I will be grateful for any comments about the issues you all are facing, and what has come up in trying this, as
>>
>> It seems likely that this is a much simpler problem in concept than it is in practice, but its solution has many benefits.
>>
>>
>> Here is my question:
>>
>> A group of us have been discussing how we might simplify submitting jobs to different compute resources in our current implementation of CIPRES, and how cloud computing might facilitate this. But none of us are cloud experts. As I understand it, the mesos cluster that I have been seeing in the Airavata email threads is intended to make it possible to deploy jobs to multiple virtual clusters. I am (we are) wondering if Mesos manages submissions to identical virtual clusters on multiple machines, and if that works efficiently.
>>
>>
>> In our implementation, we have to change the rules to run efficiently on different machines, according to gpu availability, and cores per node. I am wondering how Mesos/ virtual clusters affect those considerations.
>>
>> Can mesos create basically identical virtual clusters independent of machine?
>>
>>
>> Thanks for any advice.
>>
>>
>> Mark
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: mesos and moving jobs between clusters
Posted by Madhusudhan Govindaraju <mg...@binghamton.edu>.
Hello Mangirish,
Here is the text from Aurora's github page:
---
When and when not to use Aurora
Aurora can take over for most uses of software ... However, if you have
very specific scheduling requirements, or are building a system that
looks like a scheduler itself, you may want to explore developing your
ownframework.
---
We believe Airavata will need a framework to customize scheduling
policies for different communities, and so instead of making big changes
in Aurora, we want to develop our own framework. Once you or
Gourav-Shenoy have Airavata working with Aurora/Mesos, the idea is that
Pankaj will work with you to use the same codebase/task-module in
Airavata to launch jobs on Mesos using a custom framework.
-Madhu
On 10/28/2016 12:46 PM, Mangirish Wagle wrote:
> Hi Pankaj,
>
> I was curious to know what is your motivation to work on developing a
> custom framework and not use Aurora or any existing robust frameworks.
> It would be great if you could share some pointers on that.
> I would also like to know what specific use cases you are targeting
> through your framework, as well as what are various stability concerns
> that you may have identified and how are you planning to handle them?
>
> Regards,
> Mangirish
>
>
>
>
> On Tue, Oct 25, 2016 at 6:09 PM, Pankaj Saha <psaha4@binghamton.edu
> <ma...@binghamton.edu>> wrote:
>
> Hi Mark,
>
>
> Mesos collects the resource information from all the nodes in the
> cluster (cores, memory, disk, and gpu) and presents a unified
> view, as if it is a single operating system. The Mesosphere, who a
> commercial entity for Mesos, has built an ecosystem around Mesos
> as the kernel called the "Data Center Operating System (DCOS)".
> Frameworks interact with Mesos to reserve resources and then use
> these resources to run jobs on the cluster. So, for example, if
> multiple frameworks such as Marathon, Apache Aurora, and a
> custom-MPI-framework are using Mesos, then there is a negotiation
> between Mesos and each framework on how many resources each
> framework gets. Once the framework, say Aurora, gets resources, it
> can decide how to use those resources. Some of the strengths of
> Mesos include fault tolerance at scale and the ability to
> co-schedule applications/frameworks on the cluster such that
> cluster utilization is high.
>
>
> Mesos off-the-shelf only works when the Mater and agent nodes have
> a line of communication to each other. We have worked on modifying
> the Mesos installation so that it even works when agents are
> behind firewalls on campus clusters. We are also working on
> getting the same setup to work on Jetstream and Chameleon where
> allocations are a mix of public IPs and internally accessible
> nodes. This will allow us to use Mesos to meta-schedule across
> clusters. We are also developing our own framework, to be able to
> customize scheduling and resource negotiations for science
> gateways on Mesos clusters. Our plan is to work with Suresh and
> Marlon's team so that it works with Airavata.
>
>
> I will be presenting at the Gateways workshop in November, and
> then I will also be at SC along with my adviser (Madhu
> Govindaraju), if you would like to discuss any of these projects.
>
>
> We are working on packaging our work so that it can be shared with
> this community.
>
> Thanks
>
> Pankaj
>
>
> On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle
> <vaglomangirish@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Mark,
>
> Thanks for your question. So if I understand you correctly,
> you need kind of load balancing between identical clusters
> through a single Mesos master?
>
> With the current setup, from what I understand, we have a
> separate mesos masters for every cluster on separate clouds.
> However, its a good investigative topic if we can have single
> mesos master targeting multiple identical clusters. We have
> some work ongoing to use a virtual cluster setup with compute
> resources across clouds to install mesos, but not sure if that
> is what you are looking for.
>
> Regards,
> Mangirish
>
>
>
>
> On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark
> <mmiller@sdsc.edu <ma...@sdsc.edu>> wrote:
>
> Hi all,
>
> I posed a question to Suresh (see below), and he asked me
> to put this question on the dev list.
>
> So here it is. I will be grateful for any comments about
> the issues you all are facing, and what has come up in
> trying this, as
>
> It seems likely that this is a much simpler problem in
> concept than it is in practice, but its solution has many
> benefits.
>
> Here is my question:
>
> A group of us have been discussing how we might simplify
> submitting jobs to different compute resources in our
> current implementation of CIPRES, and how cloud computing
> might facilitate this. But none of us are cloud experts.
> As I understand it, the mesos cluster that I have been
> seeing in the Airavata email threads is intended to make
> it possible to deploy jobs to multiple virtual clusters. I
> am (we are) wondering if Mesos manages submissions to
> identical virtual clusters on multiple machines, and if
> that works efficiently.
>
> In our implementation, we have to change the rules to run
> efficiently on different machines, according to gpu
> availability, and cores per node. I am wondering how
> Mesos/ virtual clusters affect those considerations.
>
> Can mesos create basically identical virtual clusters
> independent of machine?
>
>
> Thanks for any advice.
>
> Mark
>
>
>
>
Re: mesos and moving jobs between clusters
Posted by Mangirish Wagle <va...@gmail.com>.
Hi Pankaj,
I was curious to know what is your motivation to work on developing a
custom framework and not use Aurora or any existing robust frameworks. It
would be great if you could share some pointers on that.
I would also like to know what specific use cases you are targeting through
your framework, as well as what are various stability concerns that you may
have identified and how are you planning to handle them?
Regards,
Mangirish
On Tue, Oct 25, 2016 at 6:09 PM, Pankaj Saha <ps...@binghamton.edu> wrote:
> Hi Mark,
>
> Mesos collects the resource information from all the nodes in the cluster
> (cores, memory, disk, and gpu) and presents a unified view, as if it is a
> single operating system. The Mesosphere, who a commercial entity for Mesos,
> has built an ecosystem around Mesos as the kernel called the "Data Center
> Operating System (DCOS)". Frameworks interact with Mesos to reserve
> resources and then use these resources to run jobs on the cluster. So, for
> example, if multiple frameworks such as Marathon, Apache Aurora, and a
> custom-MPI-framework are using Mesos, then there is a negotiation between
> Mesos and each framework on how many resources each framework gets. Once
> the framework, say Aurora, gets resources, it can decide how to use those
> resources. Some of the strengths of Mesos include fault tolerance at scale
> and the ability to co-schedule applications/frameworks on the cluster such
> that cluster utilization is high.
>
> Mesos off-the-shelf only works when the Mater and agent nodes have a line
> of communication to each other. We have worked on modifying the Mesos
> installation so that it even works when agents are behind firewalls on
> campus clusters. We are also working on getting the same setup to work on
> Jetstream and Chameleon where allocations are a mix of public IPs and
> internally accessible nodes. This will allow us to use Mesos to
> meta-schedule across clusters. We are also developing our own framework, to
> be able to customize scheduling and resource negotiations for science
> gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's
> team so that it works with Airavata.
>
> I will be presenting at the Gateways workshop in November, and then I will
> also be at SC along with my adviser (Madhu Govindaraju), if you would like
> to discuss any of these projects.
>
> We are working on packaging our work so that it can be shared with this
> community.
>
>
> Thanks
>
> Pankaj
>
> On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle <
> vaglomangirish@gmail.com> wrote:
>
>> Hi Mark,
>>
>> Thanks for your question. So if I understand you correctly, you need kind
>> of load balancing between identical clusters through a single Mesos master?
>>
>> With the current setup, from what I understand, we have a separate mesos
>> masters for every cluster on separate clouds. However, its a good
>> investigative topic if we can have single mesos master targeting multiple
>> identical clusters. We have some work ongoing to use a virtual cluster
>> setup with compute resources across clouds to install mesos, but not sure
>> if that is what you are looking for.
>>
>> Regards,
>> Mangirish
>>
>>
>>
>>
>> On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mm...@sdsc.edu> wrote:
>>
>>> Hi all,
>>>
>>>
>>>
>>> I posed a question to Suresh (see below), and he asked me to put this
>>> question on the dev list.
>>>
>>> So here it is. I will be grateful for any comments about the issues you
>>> all are facing, and what has come up in trying this, as
>>>
>>> It seems likely that this is a much simpler problem in concept than it
>>> is in practice, but its solution has many benefits.
>>>
>>>
>>>
>>> Here is my question:
>>>
>>> A group of us have been discussing how we might simplify submitting jobs
>>> to different compute resources in our current implementation of CIPRES, and
>>> how cloud computing might facilitate this. But none of us are cloud
>>> experts. As I understand it, the mesos cluster that I have been seeing in
>>> the Airavata email threads is intended to make it possible to deploy jobs
>>> to multiple virtual clusters. I am (we are) wondering if Mesos manages
>>> submissions to identical virtual clusters on multiple machines, and if that
>>> works efficiently.
>>>
>>>
>>>
>>> In our implementation, we have to change the rules to run efficiently on
>>> different machines, according to gpu availability, and cores per node. I am
>>> wondering how Mesos/ virtual clusters affect those considerations.
>>>
>>> Can mesos create basically identical virtual clusters independent of
>>> machine?
>>>
>>>
>>> Thanks for any advice.
>>>
>>>
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
Re: mesos and moving jobs between clusters
Posted by K Yoshimoto <ke...@sdsc.edu>.
If you don't have the ability to control policy on remote clusters,
it can be difficult to know for sure which remote cluster will run
the job soonest. This is the case where the remote workload exceeds
existing resources.
For those cases where there is no cost for job submission,
it can work well to submit to _all_ remote clusters, wait
for one to start, then cancel the rest. That's how MCP metascheduler
works. It does a resource suitability check, based on attributes
requested by job and attributes set for remote cluster, generates
job scripts for each suitable cluster, then submits to all.
This works well compared to more sophisticated predictive
metascheduling strategies:
http://dl.acm.org/citation.cfm?id=1838594
doi 10.1145/1838574.1838594
Not sure if this is relevant to clouds, where resources are
supposed to be elastic/infinite. In that situation, any job should
start immediately, so the selection criteria would be some balance
of lowest cost/best performance. Is that the situation Aurora
handles?
On Wed, Oct 26, 2016 at 04:23:57PM +0000, Shenoy, Gourav Ganesh wrote:
> Hi Mark,
>
> Sorry for responding late, but Pankaj & Mangirish have already summarized some very good options for the problem you mentioned. I am not sure if you already have, but I would recommend taking a look at Aurora \u2013 a job scheduler framework for Mesos.
>
> The scenario you mentioned, where there are compute resources, some with & without GPU and delegating jobs according to the resource requirements. Well, Aurora does this intelligently by detecting the availability of resources (cpu, gpu, ram, etc) on target slaves, based on the job needs; and runs the job on that resource. It also provides the ability to set resource quotas for specific users that submit jobs. Overall, they provide a rich set of features.
>
> Thanks and Regards,
> Gourav Shenoy
>
> From: "Miller, Mark" <mm...@sdsc.edu>
> Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
> Date: Wednesday, October 26, 2016 at 11:11 AM
> To: "dev@airavata.apache.org" <de...@airavata.apache.org>
> Subject: RE: mesos and moving jobs between clusters
>
> Hi Folks,
>
> Thanks for your kind answers.
> We are very specifically interested in how Mesos might allow us to submit to multiple machines without changing the \u201crules\u201d we impose that are machine dependent.
> For example, Gordon has 16 cores per node and no gpu
> Comet has 24 cores per node, and 4 gpu nodes.
> So we adjust our running rules depending on the resource.
>
> I can imagine one solution would be to find the least common denominator, and run all jobs
> On virtual clusters with 16 cores maximum, and no gpus.
> Or we might only submit jobs that can use gpus to resources that have them.
>
> Anyway, since we will all be in SD soon, it seems like we should chat a bit about this in person?.
> Maybe we can find a coffee break time, or set up a meeting during the meeting?
>
> Mark
>
>
>
>
> From: Pankaj Saha [mailto:psaha4@binghamton.edu]
> Sent: Tuesday, October 25, 2016 3:10 PM
> To: dev <de...@airavata.apache.org>
> Subject: Re: mesos and moving jobs between clusters
>
>
> Hi Mark,
>
>
> Mesos collects the resource information from all the nodes in the cluster (cores, memory, disk, and gpu) and presents a unified view, as if it is a single operating system. The Mesosphere, who a commercial entity for Mesos, has built an ecosystem around Mesos as the kernel called the "Data Center Operating System (DCOS)". Frameworks interact with Mesos to reserve resources and then use these resources to run jobs on the cluster. So, for example, if multiple frameworks such as Marathon, Apache Aurora, and a custom-MPI-framework are using Mesos, then there is a negotiation between Mesos and each framework on how many resources each framework gets. Once the framework, say Aurora, gets resources, it can decide how to use those resources. Some of the strengths of Mesos include fault tolerance at scale and the ability to co-schedule applications/frameworks on the cluster such that cluster utilization is high.
>
>
> Mesos off-the-shelf only works when the Mater and agent nodes have a line of communication to each other. We have worked on modifying the Mesos installation so that it even works when agents are behind firewalls on campus clusters. We are also working on getting the same setup to work on Jetstream and Chameleon where allocations are a mix of public IPs and internally accessible nodes. This will allow us to use Mesos to meta-schedule across clusters. We are also developing our own framework, to be able to customize scheduling and resource negotiations for science gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's team so that it works with Airavata.
>
>
> I will be presenting at the Gateways workshop in November, and then I will also be at SC along with my adviser (Madhu Govindaraju), if you would like to discuss any of these projects.
>
>
> We are working on packaging our work so that it can be shared with this community.
>
>
>
> Thanks
>
> Pankaj
>
> On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle <va...@gmail.com>> wrote:
> Hi Mark,
>
> Thanks for your question. So if I understand you correctly, you need kind of load balancing between identical clusters through a single Mesos master?
>
> With the current setup, from what I understand, we have a separate mesos masters for every cluster on separate clouds. However, its a good investigative topic if we can have single mesos master targeting multiple identical clusters. We have some work ongoing to use a virtual cluster setup with compute resources across clouds to install mesos, but not sure if that is what you are looking for.
>
> Regards,
> Mangirish
>
>
>
>
>
> On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mm...@sdsc.edu>> wrote:
> Hi all,
>
> I posed a question to Suresh (see below), and he asked me to put this question on the dev list.
> So here it is. I will be grateful for any comments about the issues you all are facing, and what has come up in trying this, as
> It seems likely that this is a much simpler problem in concept than it is in practice, but its solution has many benefits.
>
> Here is my question:
> A group of us have been discussing how we might simplify submitting jobs to different compute resources in our current implementation of CIPRES, and how cloud computing might facilitate this. But none of us are cloud experts. As I understand it, the mesos cluster that I have been seeing in the Airavata email threads is intended to make it possible to deploy jobs to multiple virtual clusters. I am (we are) wondering if Mesos manages submissions to identical virtual clusters on multiple machines, and if that works efficiently.
>
> In our implementation, we have to change the rules to run efficiently on different machines, according to gpu availability, and cores per node. I am wondering how Mesos/ virtual clusters affect those considerations.
> Can mesos create basically identical virtual clusters independent of machine?
>
> Thanks for any advice.
>
> Mark
>
>
>
>
>
>
Re: mesos and moving jobs between clusters
Posted by "Shenoy, Gourav Ganesh" <go...@indiana.edu>.
Hi Mark,
Sorry for responding late, but Pankaj & Mangirish have already summarized some very good options for the problem you mentioned. I am not sure if you already have, but I would recommend taking a look at Aurora – a job scheduler framework for Mesos.
The scenario you mentioned, where there are compute resources, some with & without GPU and delegating jobs according to the resource requirements. Well, Aurora does this intelligently by detecting the availability of resources (cpu, gpu, ram, etc) on target slaves, based on the job needs; and runs the job on that resource. It also provides the ability to set resource quotas for specific users that submit jobs. Overall, they provide a rich set of features.
Thanks and Regards,
Gourav Shenoy
From: "Miller, Mark" <mm...@sdsc.edu>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Wednesday, October 26, 2016 at 11:11 AM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: RE: mesos and moving jobs between clusters
Hi Folks,
Thanks for your kind answers.
We are very specifically interested in how Mesos might allow us to submit to multiple machines without changing the “rules” we impose that are machine dependent.
For example, Gordon has 16 cores per node and no gpu
Comet has 24 cores per node, and 4 gpu nodes.
So we adjust our running rules depending on the resource.
I can imagine one solution would be to find the least common denominator, and run all jobs
On virtual clusters with 16 cores maximum, and no gpus.
Or we might only submit jobs that can use gpus to resources that have them.
Anyway, since we will all be in SD soon, it seems like we should chat a bit about this in person?.
Maybe we can find a coffee break time, or set up a meeting during the meeting?
Mark
From: Pankaj Saha [mailto:psaha4@binghamton.edu]
Sent: Tuesday, October 25, 2016 3:10 PM
To: dev <de...@airavata.apache.org>
Subject: Re: mesos and moving jobs between clusters
Hi Mark,
Mesos collects the resource information from all the nodes in the cluster (cores, memory, disk, and gpu) and presents a unified view, as if it is a single operating system. The Mesosphere, who a commercial entity for Mesos, has built an ecosystem around Mesos as the kernel called the "Data Center Operating System (DCOS)". Frameworks interact with Mesos to reserve resources and then use these resources to run jobs on the cluster. So, for example, if multiple frameworks such as Marathon, Apache Aurora, and a custom-MPI-framework are using Mesos, then there is a negotiation between Mesos and each framework on how many resources each framework gets. Once the framework, say Aurora, gets resources, it can decide how to use those resources. Some of the strengths of Mesos include fault tolerance at scale and the ability to co-schedule applications/frameworks on the cluster such that cluster utilization is high.
Mesos off-the-shelf only works when the Mater and agent nodes have a line of communication to each other. We have worked on modifying the Mesos installation so that it even works when agents are behind firewalls on campus clusters. We are also working on getting the same setup to work on Jetstream and Chameleon where allocations are a mix of public IPs and internally accessible nodes. This will allow us to use Mesos to meta-schedule across clusters. We are also developing our own framework, to be able to customize scheduling and resource negotiations for science gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's team so that it works with Airavata.
I will be presenting at the Gateways workshop in November, and then I will also be at SC along with my adviser (Madhu Govindaraju), if you would like to discuss any of these projects.
We are working on packaging our work so that it can be shared with this community.
Thanks
Pankaj
On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle <va...@gmail.com>> wrote:
Hi Mark,
Thanks for your question. So if I understand you correctly, you need kind of load balancing between identical clusters through a single Mesos master?
With the current setup, from what I understand, we have a separate mesos masters for every cluster on separate clouds. However, its a good investigative topic if we can have single mesos master targeting multiple identical clusters. We have some work ongoing to use a virtual cluster setup with compute resources across clouds to install mesos, but not sure if that is what you are looking for.
Regards,
Mangirish
On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mm...@sdsc.edu>> wrote:
Hi all,
I posed a question to Suresh (see below), and he asked me to put this question on the dev list.
So here it is. I will be grateful for any comments about the issues you all are facing, and what has come up in trying this, as
It seems likely that this is a much simpler problem in concept than it is in practice, but its solution has many benefits.
Here is my question:
A group of us have been discussing how we might simplify submitting jobs to different compute resources in our current implementation of CIPRES, and how cloud computing might facilitate this. But none of us are cloud experts. As I understand it, the mesos cluster that I have been seeing in the Airavata email threads is intended to make it possible to deploy jobs to multiple virtual clusters. I am (we are) wondering if Mesos manages submissions to identical virtual clusters on multiple machines, and if that works efficiently.
In our implementation, we have to change the rules to run efficiently on different machines, according to gpu availability, and cores per node. I am wondering how Mesos/ virtual clusters affect those considerations.
Can mesos create basically identical virtual clusters independent of machine?
Thanks for any advice.
Mark
RE: mesos and moving jobs between clusters
Posted by "Miller, Mark" <mm...@sdsc.edu>.
Hi Folks,
Thanks for your kind answers.
We are very specifically interested in how Mesos might allow us to submit to multiple machines without changing the “rules” we impose that are machine dependent.
For example, Gordon has 16 cores per node and no gpu
Comet has 24 cores per node, and 4 gpu nodes.
So we adjust our running rules depending on the resource.
I can imagine one solution would be to find the least common denominator, and run all jobs
On virtual clusters with 16 cores maximum, and no gpus.
Or we might only submit jobs that can use gpus to resources that have them.
Anyway, since we will all be in SD soon, it seems like we should chat a bit about this in person?.
Maybe we can find a coffee break time, or set up a meeting during the meeting?
Mark
From: Pankaj Saha [mailto:psaha4@binghamton.edu]
Sent: Tuesday, October 25, 2016 3:10 PM
To: dev <de...@airavata.apache.org>
Subject: Re: mesos and moving jobs between clusters
Hi Mark,
Mesos collects the resource information from all the nodes in the cluster (cores, memory, disk, and gpu) and presents a unified view, as if it is a single operating system. The Mesosphere, who a commercial entity for Mesos, has built an ecosystem around Mesos as the kernel called the "Data Center Operating System (DCOS)". Frameworks interact with Mesos to reserve resources and then use these resources to run jobs on the cluster. So, for example, if multiple frameworks such as Marathon, Apache Aurora, and a custom-MPI-framework are using Mesos, then there is a negotiation between Mesos and each framework on how many resources each framework gets. Once the framework, say Aurora, gets resources, it can decide how to use those resources. Some of the strengths of Mesos include fault tolerance at scale and the ability to co-schedule applications/frameworks on the cluster such that cluster utilization is high.
Mesos off-the-shelf only works when the Mater and agent nodes have a line of communication to each other. We have worked on modifying the Mesos installation so that it even works when agents are behind firewalls on campus clusters. We are also working on getting the same setup to work on Jetstream and Chameleon where allocations are a mix of public IPs and internally accessible nodes. This will allow us to use Mesos to meta-schedule across clusters. We are also developing our own framework, to be able to customize scheduling and resource negotiations for science gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's team so that it works with Airavata.
I will be presenting at the Gateways workshop in November, and then I will also be at SC along with my adviser (Madhu Govindaraju), if you would like to discuss any of these projects.
We are working on packaging our work so that it can be shared with this community.
Thanks
Pankaj
On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle <va...@gmail.com>> wrote:
Hi Mark,
Thanks for your question. So if I understand you correctly, you need kind of load balancing between identical clusters through a single Mesos master?
With the current setup, from what I understand, we have a separate mesos masters for every cluster on separate clouds. However, its a good investigative topic if we can have single mesos master targeting multiple identical clusters. We have some work ongoing to use a virtual cluster setup with compute resources across clouds to install mesos, but not sure if that is what you are looking for.
Regards,
Mangirish
[https://mailtrack.io/trace/mail/8481e3ada5019bfcebef25b672239785318edb23.png?u=765734]
On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mm...@sdsc.edu>> wrote:
Hi all,
I posed a question to Suresh (see below), and he asked me to put this question on the dev list.
So here it is. I will be grateful for any comments about the issues you all are facing, and what has come up in trying this, as
It seems likely that this is a much simpler problem in concept than it is in practice, but its solution has many benefits.
Here is my question:
A group of us have been discussing how we might simplify submitting jobs to different compute resources in our current implementation of CIPRES, and how cloud computing might facilitate this. But none of us are cloud experts. As I understand it, the mesos cluster that I have been seeing in the Airavata email threads is intended to make it possible to deploy jobs to multiple virtual clusters. I am (we are) wondering if Mesos manages submissions to identical virtual clusters on multiple machines, and if that works efficiently.
In our implementation, we have to change the rules to run efficiently on different machines, according to gpu availability, and cores per node. I am wondering how Mesos/ virtual clusters affect those considerations.
Can mesos create basically identical virtual clusters independent of machine?
Thanks for any advice.
Mark
Re: mesos and moving jobs between clusters
Posted by Pankaj Saha <ps...@binghamton.edu>.
Hi Mark,
Mesos collects the resource information from all the nodes in the cluster
(cores, memory, disk, and gpu) and presents a unified view, as if it is a
single operating system. The Mesosphere, who a commercial entity for Mesos,
has built an ecosystem around Mesos as the kernel called the "Data Center
Operating System (DCOS)". Frameworks interact with Mesos to reserve
resources and then use these resources to run jobs on the cluster. So, for
example, if multiple frameworks such as Marathon, Apache Aurora, and a
custom-MPI-framework are using Mesos, then there is a negotiation between
Mesos and each framework on how many resources each framework gets. Once
the framework, say Aurora, gets resources, it can decide how to use those
resources. Some of the strengths of Mesos include fault tolerance at scale
and the ability to co-schedule applications/frameworks on the cluster such
that cluster utilization is high.
Mesos off-the-shelf only works when the Mater and agent nodes have a line
of communication to each other. We have worked on modifying the Mesos
installation so that it even works when agents are behind firewalls on
campus clusters. We are also working on getting the same setup to work on
Jetstream and Chameleon where allocations are a mix of public IPs and
internally accessible nodes. This will allow us to use Mesos to
meta-schedule across clusters. We are also developing our own framework, to
be able to customize scheduling and resource negotiations for science
gateways on Mesos clusters. Our plan is to work with Suresh and Marlon's
team so that it works with Airavata.
I will be presenting at the Gateways workshop in November, and then I will
also be at SC along with my adviser (Madhu Govindaraju), if you would like
to discuss any of these projects.
We are working on packaging our work so that it can be shared with this
community.
Thanks
Pankaj
On Tue, Oct 25, 2016 at 11:36 AM, Mangirish Wagle <va...@gmail.com>
wrote:
> Hi Mark,
>
> Thanks for your question. So if I understand you correctly, you need kind
> of load balancing between identical clusters through a single Mesos master?
>
> With the current setup, from what I understand, we have a separate mesos
> masters for every cluster on separate clouds. However, its a good
> investigative topic if we can have single mesos master targeting multiple
> identical clusters. We have some work ongoing to use a virtual cluster
> setup with compute resources across clouds to install mesos, but not sure
> if that is what you are looking for.
>
> Regards,
> Mangirish
>
>
>
>
> On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mm...@sdsc.edu> wrote:
>
>> Hi all,
>>
>>
>>
>> I posed a question to Suresh (see below), and he asked me to put this
>> question on the dev list.
>>
>> So here it is. I will be grateful for any comments about the issues you
>> all are facing, and what has come up in trying this, as
>>
>> It seems likely that this is a much simpler problem in concept than it is
>> in practice, but its solution has many benefits.
>>
>>
>>
>> Here is my question:
>>
>> A group of us have been discussing how we might simplify submitting jobs
>> to different compute resources in our current implementation of CIPRES, and
>> how cloud computing might facilitate this. But none of us are cloud
>> experts. As I understand it, the mesos cluster that I have been seeing in
>> the Airavata email threads is intended to make it possible to deploy jobs
>> to multiple virtual clusters. I am (we are) wondering if Mesos manages
>> submissions to identical virtual clusters on multiple machines, and if that
>> works efficiently.
>>
>>
>>
>> In our implementation, we have to change the rules to run efficiently on
>> different machines, according to gpu availability, and cores per node. I am
>> wondering how Mesos/ virtual clusters affect those considerations.
>>
>> Can mesos create basically identical virtual clusters independent of
>> machine?
>>
>>
>> Thanks for any advice.
>>
>>
>>
>> Mark
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: mesos and moving jobs between clusters
Posted by Mangirish Wagle <va...@gmail.com>.
Hi Mark,
Thanks for your question. So if I understand you correctly, you need kind
of load balancing between identical clusters through a single Mesos master?
With the current setup, from what I understand, we have a separate mesos
masters for every cluster on separate clouds. However, its a good
investigative topic if we can have single mesos master targeting multiple
identical clusters. We have some work ongoing to use a virtual cluster
setup with compute resources across clouds to install mesos, but not sure
if that is what you are looking for.
Regards,
Mangirish
On Tue, Oct 25, 2016 at 11:05 AM, Miller, Mark <mm...@sdsc.edu> wrote:
> Hi all,
>
>
>
> I posed a question to Suresh (see below), and he asked me to put this
> question on the dev list.
>
> So here it is. I will be grateful for any comments about the issues you
> all are facing, and what has come up in trying this, as
>
> It seems likely that this is a much simpler problem in concept than it is
> in practice, but its solution has many benefits.
>
>
>
> Here is my question:
>
> A group of us have been discussing how we might simplify submitting jobs
> to different compute resources in our current implementation of CIPRES, and
> how cloud computing might facilitate this. But none of us are cloud
> experts. As I understand it, the mesos cluster that I have been seeing in
> the Airavata email threads is intended to make it possible to deploy jobs
> to multiple virtual clusters. I am (we are) wondering if Mesos manages
> submissions to identical virtual clusters on multiple machines, and if that
> works efficiently.
>
>
>
> In our implementation, we have to change the rules to run efficiently on
> different machines, according to gpu availability, and cores per node. I am
> wondering how Mesos/ virtual clusters affect those considerations.
>
> Can mesos create basically identical virtual clusters independent of
> machine?
>
>
> Thanks for any advice.
>
>
>
> Mark
>
>
>
>
>
>
>
>
>