You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by DImuthu Upeksha <di...@gmail.com> on 2018/12/16 12:30:05 UTC
Metascheduler work
Hi Folks,
I found this [1] mail thread and the JIRA ticket [2] which have discussed
about coming up with an Airavata specific job scheduler. At the end of the
discussion, seems like an approach based on Mesos has been chosen to
tryout. Is there any other discussion/ documents regarding this topic? Has
anyone worked on this and if so, where are the code / design documents?
[1]
https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results
[2] https://issues.apache.org/jira/browse/AIRAVATA-1436
Thanks
Dimuthu
Re: Metascheduler work
Posted by Mangirish Wagle <va...@gmail.com>.
Hello Folks,
Sharing the report pdf of my investigation of running MPI jobs on Mesos,
just in case that would help in the evaluation process in any way. Please
find the same attached.
Thanks and Regards,
Mangirish Wagle
On Sun, Dec 16, 2018 at 4:39 AM DImuthu Upeksha <di...@gmail.com>
wrote:
> Hi Folks,
>
> I found this [1] mail thread and the JIRA ticket [2] which have discussed
> about coming up with an Airavata specific job scheduler. At the end of the
> discussion, seems like an approach based on Mesos has been chosen to
> tryout. Is there any other discussion/ documents regarding this topic? Has
> anyone worked on this and if so, where are the code / design documents?
>
> [1]
> https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results
> [2] https://issues.apache.org/jira/browse/AIRAVATA-1436
>
> Thanks
> Dimuthu
>
Re: Metascheduler work
Posted by Pankaj Saha <ps...@binghamton.edu>.
The solution I provided was to run MPI tasks with Mesos was with the help
of Docker Swarm. It required me to develop a Mesos Framework and an
Executor [here
<http://mesos.apache.org/documentation/latest/app-framework-development-guide/>].
Mesos containers could not communicate with each other and that was the
reason to bring Docker Swarm into the picture. However now, Mesos container
can communicate with help of Calico [here
<https://docs.projectcalico.org/v1.5/getting-started/mesos/>]. Calico
provides an overlay network and each Mesos container gets a unique IP
address.
Thanks
Pankaj
On Mon, Dec 17, 2018 at 12:13 PM Pamidighantam, Sudhakar V <
spamidig@illinois.edu> wrote:
> I thought Pankaj presented a way to implement MPI executions through mesos
> more recently.
>
> He may want to comment on this..
>
>
>
> Thanks,
>
> Sudakar.
>
>
>
> *From: *Marlon Pierce <ma...@iu.edu>
> *Reply-To: *<de...@airavata.apache.org>
> *Date: *Monday, December 17, 2018 at 8:52 AM
> *To: *"dev@airavata.apache.org" <de...@airavata.apache.org>
> *Subject: *Re: Metascheduler work
>
>
>
> Hi Dimuthu,
>
>
>
> This is something we should re-evaluate. Mangirish Wangle looked at Mesos
> integration with Airavata back in 2016, but he ultimately ran into many
> difficulties, including getting MPI jobs to work, if I recall correctly.
>
>
>
> Marlon
>
>
>
>
>
> *From: *"dimuthu.upeksha2@gmail.com" <di...@gmail.com>
> *Reply-To: *dev <de...@airavata.apache.org>
> *Date: *Sunday, December 16, 2018 at 7:30 AM
> *To: *dev <de...@airavata.apache.org>
> *Subject: *Metascheduler work
>
>
>
> Hi Folks,
>
>
>
> I found this [1] mail thread and the JIRA ticket [2] which have discussed
> about coming up with an Airavata specific job scheduler. At the end of the
> discussion, seems like an approach based on Mesos has been chosen to
> tryout. Is there any other discussion/ documents regarding this topic? Has
> anyone worked on this and if so, where are the code / design documents?
>
>
>
> [1]
> https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results
>
> [2] https://issues.apache.org/jira/browse/AIRAVATA-1436
>
>
>
> Thanks
>
> Dimuthu
>
Re: Metascheduler work
Posted by "Pamidighantam, Sudhakar V" <sp...@illinois.edu>.
I thought Pankaj presented a way to implement MPI executions through mesos more recently.
He may want to comment on this..
Thanks,
Sudakar.
From: Marlon Pierce <ma...@iu.edu>
Reply-To: <de...@airavata.apache.org>
Date: Monday, December 17, 2018 at 8:52 AM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Re: Metascheduler work
Hi Dimuthu,
This is something we should re-evaluate. Mangirish Wangle looked at Mesos integration with Airavata back in 2016, but he ultimately ran into many difficulties, including getting MPI jobs to work, if I recall correctly.
Marlon
From: "dimuthu.upeksha2@gmail.com" <di...@gmail.com>
Reply-To: dev <de...@airavata.apache.org>
Date: Sunday, December 16, 2018 at 7:30 AM
To: dev <de...@airavata.apache.org>
Subject: Metascheduler work
Hi Folks,
I found this [1] mail thread and the JIRA ticket [2] which have discussed about coming up with an Airavata specific job scheduler. At the end of the discussion, seems like an approach based on Mesos has been chosen to tryout. Is there any other discussion/ documents regarding this topic? Has anyone worked on this and if so, where are the code / design documents?
[1] https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results
[2] https://issues.apache.org/jira/browse/AIRAVATA-1436
Thanks
Dimuthu
Re: Metascheduler work
Posted by "Christie, Marcus Aaron" <ma...@iu.edu>.
Hi Dimuthu,
> On Dec 17, 2018, at 9:55 AM, DImuthu Upeksha <di...@gmail.com> wrote:
>
> Metascheduling Usecase 3: Users within a gateway need to fairly use a community account. Computational resources like XSEDE enforce fair-share across users, but since gateway job submissions are funneled through a single community account, different users within a gateway are impacted. Airavata will need to implement fair-share scheduling among these users to ensure fair use of allocations as well as allowable queue limits and work with resources policies.
>
> Comments
> 1. Is there any existing work regarding this usecase? I can remember that two students created a UI to enforce these limits and save in a database.
>
I'm guessing that the fair share regarding metascheduling is a little different from the allocation manager work done by IU students in the spring. Here fair share would mean fairly scheduling jobs so that no one user gets a greater share of running jobs. For example, if a queue only allows 50 jobs per user and all jobs from a gateway are submitted as a single community account user, and you have gateway user A submit 50 jobs and gateway users B and C submit 1 job each, then you would want to avoid the situation where all 50 of user A's jobs get submitted and user B and C's jobs are waiting on them to complete before their jobs get submitted to the queue. You would rather want to have user B and user C's jobs get submitted and 48 of user A's jobs.
Re: Metascheduler work
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi Marlon,
Thanks for the clarification. Yes I agree that we need to re-evaluate both
the use case and the approach we are going to implement this.
Hi All,
Considering current requirements of gateways, we somehow need a better job
throttling/ scheduling proxy before the actual schedulers. I will simply
copy and paste the existing use cases mentioned in the wiki and please
share your views on the validity of these use cases in current gateways.
Once we have come up with a concrete set of requirements, then let's think
about the technical side of the solution. I have added my point of view on
each usecase in comments section.
*Metascheduling Usecase 1: Users/Gateways submitting a series of jobs to a
resource. Whereas a resource enforces a per user job limit within a queue
and ensure fair use of the clusters ((example: stampede allows 50 jobs per
user in the normal queue). Airavata will need to implement queues and
throttle jobs respecting the max-job-per-queue limits of a underlying
resource queue. *
Comments:
1. What will happen to the pending jobs in the Airavata queue? Will they
stay in the queue forever until the queues in the clusters become available
or is there an expiry?
2. Are we assuming that Airavata is the only entry point for the queues in
the clusters or can there be some other gateways/ independent users
submitting jobs to the same queue?
*Metascheduling Usecase 2: Users/Gateways delegate job scheduling across
available computational resources to Airavata. Airavata will need to
implement schedulers which become aware of existing loads on the clusters
and spread jobs efficiently. The scheduler should also be able to get
access to heuristics on previous executions and current requirements which
includes job size (number of nodes/cores), memory requirements, wall time
estimates and so forth. *
Comments:
1. Even though the first part makes sense for me, second part looks more
like a machine learning problem and a nice to have a feature. Is this
something like user comes into portal and launch an application with a set
of inputs and Airavata decides in which machine / cluster should this job
run?
*Metascheduling Usecase 3: Users within a gateway need to fairly use a
community account. Computational resources like XSEDE enforce fair-share
across users, but since gateway job submissions are funneled through a
single community account, different users within a gateway are impacted.
Airavata will need to implement fair-share scheduling among these users to
ensure fair use of allocations as well as allowable queue limits and work
with resources policies.*
Comments
1. Is there any existing work regarding this usecase? I can remember that
two students created a UI to enforce these limits and save in a database.
Thanks
Dimuthu
On Mon, Dec 17, 2018 at 7:22 PM Pierce, Marlon <ma...@iu.edu> wrote:
> Hi Dimuthu,
>
>
>
> This is something we should re-evaluate. Mangirish Wangle looked at Mesos
> integration with Airavata back in 2016, but he ultimately ran into many
> difficulties, including getting MPI jobs to work, if I recall correctly.
>
>
>
> Marlon
>
>
>
>
>
> *From: *"dimuthu.upeksha2@gmail.com" <di...@gmail.com>
> *Reply-To: *dev <de...@airavata.apache.org>
> *Date: *Sunday, December 16, 2018 at 7:30 AM
> *To: *dev <de...@airavata.apache.org>
> *Subject: *Metascheduler work
>
>
>
> Hi Folks,
>
>
>
> I found this [1] mail thread and the JIRA ticket [2] which have discussed
> about coming up with an Airavata specific job scheduler. At the end of the
> discussion, seems like an approach based on Mesos has been chosen to
> tryout. Is there any other discussion/ documents regarding this topic? Has
> anyone worked on this and if so, where are the code / design documents?
>
>
>
> [1]
> https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results
>
> [2] https://issues.apache.org/jira/browse/AIRAVATA-1436
>
>
>
> Thanks
>
> Dimuthu
>
Re: Metascheduler work
Posted by "Pierce, Marlon" <ma...@iu.edu>.
Hi Dimuthu,
This is something we should re-evaluate. Mangirish Wangle looked at Mesos integration with Airavata back in 2016, but he ultimately ran into many difficulties, including getting MPI jobs to work, if I recall correctly.
Marlon
From: "dimuthu.upeksha2@gmail.com" <di...@gmail.com>
Reply-To: dev <de...@airavata.apache.org>
Date: Sunday, December 16, 2018 at 7:30 AM
To: dev <de...@airavata.apache.org>
Subject: Metascheduler work
Hi Folks,
I found this [1] mail thread and the JIRA ticket [2] which have discussed about coming up with an Airavata specific job scheduler. At the end of the discussion, seems like an approach based on Mesos has been chosen to tryout. Is there any other discussion/ documents regarding this topic? Has anyone worked on this and if so, where are the code / design documents?
[1] https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results
[2] https://issues.apache.org/jira/browse/AIRAVATA-1436
Thanks
Dimuthu