You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Gourav Rattihalli <gr...@binghamton.edu> on 2016/04/27 19:01:00 UTC
Design of Mesos/Aurora integration with Airavata on Jetstream
Hi Dev's,
I have been working on the integration of Apache Aurora and Mesos with
Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
created using on Jetstream. I want to understand how Airavata is likely to
use the VMs for applications. Will airavata re-use existing VMs for
successive applications for a given community, or will new VMs be created
for each application? This will decide how we design the automated creation
of a Mesos cluster using the new VMs that are created. I understand that we
will have a Mesos master VM that will run all the time on Jetstream.
Here is the Job submission flow that I am assuming:
Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)
-->My-Apache-Aurora-module-->(A script will create .aurora configuration
file that will be used to launch the job)-->Aurora/Mesos-->VMs
So, I will design a module that will work with Orchestrator-->GFAC to
generate the appropriate Apache Aurora script to be submitted to the Aurora
master.
Please let me know if there are any comments, suggestions on this plan.
--
Regards,
Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Gourav Rattihalli <gr...@binghamton.edu>.
Hi Marlon,
The worker VMs need to have the Mesos libraries installed and the Mesos
master needs to be configured with the IPs of the worker VMs. So, if new
VMs are created each time, we will develop the playbook for the creation of
the Mesos cluster on the fly, and then launch the application on it.
If the VMs are assumed to be already available, due to the reason you
describe, then this is an easier case as the Mesos/Aurora setup is a one
time task, which will be well tested before the applications are launched.
On Wed, Apr 27, 2016 at 1:06 PM, Pierce, Marlon <ma...@iu.edu> wrote:
> Thanks, Gourav, this is great. How will the design differ in the two
> cases?
>
> Since installing the applications is not a simple task and there aren’t
> good docker libraries (for example) for many of them, I would assume that
> we would have a collection of VMs that have the codes already installed.
>
> Marlon
>
>
> From: Gourav Rattihalli <gr...@binghamton.edu>
> Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
> Date: Wednesday, April 27, 2016 at 1:01 PM
> To: "dev@airavata.apache.org" <de...@airavata.apache.org>
> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>
> Hi Dev's,
>
> I have been working on the integration of Apache Aurora and Mesos with
> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
> created using on Jetstream. I want to understand how Airavata is likely to
> use the VMs for applications. Will airavata re-use existing VMs for
> successive applications for a given community, or will new VMs be created
> for each application? This will decide how we design the automated creation
> of a Mesos cluster using the new VMs that are created. I understand that we
> will have a Mesos master VM that will run all the time on Jetstream.
>
> Here is the Job submission flow that I am assuming:
>
> Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)
> -->My-Apache-Aurora-module-->(A script will create .aurora configuration
> file that will be used to launch the job)-->Aurora/Mesos-->VMs
>
> So, I will design a module that will work with Orchestrator-->GFAC to
> generate the appropriate Apache Aurora script to be submitted to the Aurora
> master.
>
> Please let me know if there are any comments, suggestions on this plan.
>
> --
> Regards,
> Gourav Rattihalli
>
--
Regards,
Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Emre Brookes <em...@biochem.uthscsa.edu>.
Hi Marlon,
Yes, I have thought about it and it should be a nice addition.
Wasn't sure about Docker vs alternatives such as LXD or Rocket...
But I don't expect to get to this until 3 or 4th quarter.
-Emre
Pierce, Marlon wrote:
> Hi Emre, are you looking at containerizing any of your applications?
>
>
>
> On 4/27/16, 1:21 PM, "Emre Brookes" <em...@biochem.uthscsa.edu> wrote:
>
>> Pierce, Marlon wrote:
>>> Thanks, Gourav, this is great. How will the design differ in the two
>>> cases?
>>>
>>> Since installing the applications is not a simple task and there
>>> aren’t good docker libraries (for example) for many of them, I would
>>> assume that we would have a collection of VMs that have the codes
>>> already installed.
>> FWIW, that model would work well with GenApp usage also. Although, I
>> can imagine a scenario where a part of job initialization script
>> (running under the already "code installed" VM) checks for updates to
>> the code and installs them, but that can be independent of
>> Airavata/Aurora/Mesos's knowledge.
>>
>> -Emre
>>
>>> Marlon
>>>
>>>
>>> From: Gourav Rattihalli <grattih1@binghamton.edu
>>> <ma...@binghamton.edu>>
>>> Reply-To: "dev@airavata.apache.org <ma...@airavata.apache.org>"
>>> <dev@airavata.apache.org <ma...@airavata.apache.org>>
>>> Date: Wednesday, April 27, 2016 at 1:01 PM
>>> To: "dev@airavata.apache.org <ma...@airavata.apache.org>"
>>> <dev@airavata.apache.org <ma...@airavata.apache.org>>
>>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>>>
>>> Hi Dev's,
>>>
>>> I have been working on the integration of Apache Aurora and Mesos with
>>> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
>>> created using on Jetstream. I want to understand how Airavata is
>>> likely to use the VMs for applications. Will airavata re-use existing
>>> VMs for successive applications for a given community, or will new VMs
>>> be created for each application? This will decide how we design the
>>> automated creation of a Mesos cluster using the new VMs that are
>>> created. I understand that we will have a Mesos master VM that will
>>> run all the time on Jetstream.
>>>
>>> Here is the Job submission flow that I am assuming:
>>>
>>> Airavata Client--> Airavata
>>> Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script
>>> will create .aurora configuration file that will be used to launch the
>>> job)-->Aurora/Mesos-->VMs
>>>
>>> So, I will design a module that will work with Orchestrator-->GFAC to
>>> generate the appropriate Apache Aurora script to be submitted to the
>>> Aurora master.
>>>
>>> Please let me know if there are any comments, suggestions on this plan.
>>>
>>> --
>>> Regards,
>>> Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by "Pierce, Marlon" <ma...@iu.edu>.
Hi Emre, are you looking at containerizing any of your applications?
On 4/27/16, 1:21 PM, "Emre Brookes" <em...@biochem.uthscsa.edu> wrote:
>Pierce, Marlon wrote:
>> Thanks, Gourav, this is great. How will the design differ in the two
>> cases?
>>
>> Since installing the applications is not a simple task and there
>> aren’t good docker libraries (for example) for many of them, I would
>> assume that we would have a collection of VMs that have the codes
>> already installed.
>FWIW, that model would work well with GenApp usage also. Although, I
>can imagine a scenario where a part of job initialization script
>(running under the already "code installed" VM) checks for updates to
>the code and installs them, but that can be independent of
>Airavata/Aurora/Mesos's knowledge.
>
>-Emre
>
>>
>> Marlon
>>
>>
>> From: Gourav Rattihalli <grattih1@binghamton.edu
>> <ma...@binghamton.edu>>
>> Reply-To: "dev@airavata.apache.org <ma...@airavata.apache.org>"
>> <dev@airavata.apache.org <ma...@airavata.apache.org>>
>> Date: Wednesday, April 27, 2016 at 1:01 PM
>> To: "dev@airavata.apache.org <ma...@airavata.apache.org>"
>> <dev@airavata.apache.org <ma...@airavata.apache.org>>
>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>>
>> Hi Dev's,
>>
>> I have been working on the integration of Apache Aurora and Mesos with
>> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
>> created using on Jetstream. I want to understand how Airavata is
>> likely to use the VMs for applications. Will airavata re-use existing
>> VMs for successive applications for a given community, or will new VMs
>> be created for each application? This will decide how we design the
>> automated creation of a Mesos cluster using the new VMs that are
>> created. I understand that we will have a Mesos master VM that will
>> run all the time on Jetstream.
>>
>> Here is the Job submission flow that I am assuming:
>>
>> Airavata Client--> Airavata
>> Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script
>> will create .aurora configuration file that will be used to launch the
>> job)-->Aurora/Mesos-->VMs
>>
>> So, I will design a module that will work with Orchestrator-->GFAC to
>> generate the appropriate Apache Aurora script to be submitted to the
>> Aurora master.
>>
>> Please let me know if there are any comments, suggestions on this plan.
>>
>> --
>> Regards,
>> Gourav Rattihalli
>
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Suresh Marru <sm...@apache.org>.
On Apr 27, 2016, at 3:55 PM, Gourav Rattihalli <gr...@binghamton.edu> wrote:
>
> Hi Suresh,
>
> Yes, I will use the develop branch on Github.
>
> Elaborating on the second option.
>
> If the cluster already exists with VM's that have the required tools to run the application, then the task is much easier as we will not have to create a VM and move it to the cluster. The cluster would already be created and the application launches-->executes-->dies on the VM, but the VM stays.
Thanks, this is helpful. I agree the additional time in provision and deprovisioning, but I think the complexity of correlating which VM has which all applications (added to the library conflicts) might outweigh the simplicity of one vm to one application. More over each application will have different resources needs, so the instance sizes might not align well.
Suresh
>
> On Wed, Apr 27, 2016 at 3:28 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
> On Apr 27, 2016, at 2:57 PM, Gourav Rattihalli <grattih1@binghamton.edu <ma...@binghamton.edu>> wrote:
>>
>>
>> Hi Suresh,
>>
>> First of all I would like to commend you for starting this discussion. This is exactly kind of the mailing list engagement we would like to see as part of your GSOC. You already drew Emre’s attention, often discussions done in isolation undermine this impact. So good job on a start and we will expect over the summer you will engage lot more.
>>
>>
>> Thanks. Yes, I plan to actively use this mailing list to keep my design focused on real use cases, and also to get regular feedback.
>>
>> Application dependency libraries, compiler versions can quickly get tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod <https://github.com/TACC/Lmod>) within a VM to alleviate this but that will be an over kill. Eventually containers will hopefully alleviate this problem.
>>
>>> The worker VMs need to have the Mesos libraries installed and the Mesos master needs to be configured with the IPs of the worker VMs. So, if new VMs are created each time, we will develop the playbook for the creation of the Mesos cluster on the fly, and then launch the application on it.
>>
>>
>> Can the slave not register itself dynamically and report its IP Address? With either multiple applications per VM or one per VM, you will need to look into the capability of a persistent master and short lived elastic slave instances right? May be I am mis-understating your questions. Can you help clarify this more?
>>
>>
>> Yes, this is possible, and was the first of the two options I had emailed.
>
> Can you elaborate on the second option, may be I missed what you tried to convey? Lets put aside how many applications per vm for this discussion since thats orthogonal.
>
>> To support this kind of elasticity, we will work on developing a script/playbook to install the Mesos libraries on the new slaves/agents and connect them with the master.
>
> Can you please develop them in Airavata repo itself? I mean lets not make perfection be the enemy here and start sending pull requests as you bake these playbooks. You can put the scripts here - https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks <https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks> (note that it is develop branch).
>
> You can read about contributions here - http://airavata.staging.apache.org/community/how-to-contribute-code.html <http://airavata.staging.apache.org/community/how-to-contribute-code.html> (we need to move these instructions to the new website).
>
> Suresh
>
>>
>> Thanks,
>> Gourav
>>
>>> On Apr 27, 2016, at 1:21 PM, Emre Brookes <emre@biochem.uthscsa.edu <ma...@biochem.uthscsa.edu>> wrote:
>>>
>>> Pierce, Marlon wrote:
>>>> Thanks, Gourav, this is great. How will the design differ in the two cases?
>>>>
>>>> Since installing the applications is not a simple task and there aren’t good docker libraries (for example) for many of them, I would assume that we would have a collection of VMs that have the codes already installed.
>>> FWIW, that model would work well with GenApp usage also. Although, I can imagine a scenario where a part of job initialization script (running under the already "code installed" VM) checks for updates to the code and installs them, but that can be independent of Airavata/Aurora/Mesos's knowledge.
>>>
>>> -Emre
>>>
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Gourav Rattihalli <grattih1@binghamton.edu <ma...@binghamton.edu> <mailto:grattih1@binghamton.edu <ma...@binghamton.edu>>>
>>>> Reply-To: "dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>" <dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>>
>>>> Date: Wednesday, April 27, 2016 at 1:01 PM
>>>> To: "dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>" <dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>>
>>>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>>>>
>>>> Hi Dev's,
>>>>
>>>> I have been working on the integration of Apache Aurora and Mesos with Airavata and Jetstream. Using Mangirish's latest code, VMs can now be created using on Jetstream. I want to understand how Airavata is likely to use the VMs for applications. Will airavata re-use existing VMs for successive applications for a given community, or will new VMs be created for each application? This will decide how we design the automated creation of a Mesos cluster using the new VMs that are created. I understand that we will have a Mesos master VM that will run all the time on Jetstream.
>>>>
>>>> Here is the Job submission flow that I am assuming:
>>>>
>>>> Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will create .aurora configuration file that will be used to launch the job)-->Aurora/Mesos-->VMs
>>>>
>>>> So, I will design a module that will work with Orchestrator-->GFAC to generate the appropriate Apache Aurora script to be submitted to the Aurora master.
>>>>
>>>> Please let me know if there are any comments, suggestions on this plan.
>>>>
>>>> --
>>>> Regards,
>>>> Gourav Rattihalli
>>>
>>
>>
>>
>>
>> --
>> Regards,
>> Gourav Rattihalli
>
>
>
>
> --
> Regards,
> Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Gourav Rattihalli <gr...@binghamton.edu>.
Hi Suresh,
Yes, I will use the develop branch on Github.
Elaborating on the second option.
If the cluster already exists with VM's that have the required tools to run
the application, then the task is much easier as we will not have to create
a VM and move it to the cluster. The cluster would already be created and
the application launches-->executes-->dies on the VM, but the VM stays.
On Wed, Apr 27, 2016 at 3:28 PM, Suresh Marru <sm...@apache.org> wrote:
> On Apr 27, 2016, at 2:57 PM, Gourav Rattihalli <gr...@binghamton.edu>
> wrote:
>
>
>
> Hi Suresh,
>
> First of all I would like to commend you for starting this discussion.
>> This is exactly kind of the mailing list engagement we would like to see as
>> part of your GSOC. You already drew Emre’s attention, often discussions
>> done in isolation undermine this impact. So good job on a start and we will
>> expect over the summer you will engage lot more.
>>
>>
> Thanks. Yes, I plan to actively use this mailing list to keep my design
> focused on real use cases, and also to get regular feedback.
>
>
>> Application dependency libraries, compiler versions can quickly get
>> tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod)
>> within a VM to alleviate this but that will be an over kill. Eventually
>> containers will hopefully alleviate this problem.
>>
>> The worker VMs need to have the Mesos libraries installed and the Mesos
>> master needs to be configured with the IPs of the worker VMs. So, if new
>> VMs are created each time, we will develop the playbook for the creation of
>> the Mesos cluster on the fly, and then launch the application on it.
>>
>>
>> Can the slave not register itself dynamically and report its IP Address?
>> With either multiple applications per VM or one per VM, you will need to
>> look into the capability of a persistent master and short lived elastic
>> slave instances right? May be I am mis-understating your questions. Can you
>> help clarify this more?
>>
>>
> Yes, this is possible, and was the first of the two options I had emailed.
>
>
> Can you elaborate on the second option, may be I missed what you tried to
> convey? Lets put aside how many applications per vm for this discussion
> since thats orthogonal.
>
> To support this kind of elasticity, we will work on developing a
> script/playbook to install the Mesos libraries on the new slaves/agents and
> connect them with the master.
>
>
> Can you please develop them in Airavata repo itself? I mean lets not make
> perfection be the enemy here and start sending pull requests as you bake
> these playbooks. You can put the scripts here -
> https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks (note
> that it is develop branch).
>
> You can read about contributions here -
> http://airavata.staging.apache.org/community/how-to-contribute-code.html (we
> need to move these instructions to the new website).
>
> Suresh
>
>
> Thanks,
> Gourav
>
> On Apr 27, 2016, at 1:21 PM, Emre Brookes <em...@biochem.uthscsa.edu>
> wrote:
>
> Pierce, Marlon wrote:
>
> Thanks, Gourav, this is great. How will the design differ in the two cases?
>
> Since installing the applications is not a simple task and there aren’t
> good docker libraries (for example) for many of them, I would assume that
> we would have a collection of VMs that have the codes already installed.
>
> FWIW, that model would work well with GenApp usage also. Although, I can
> imagine a scenario where a part of job initialization script (running under
> the already "code installed" VM) checks for updates to the code and
> installs them, but that can be independent of Airavata/Aurora/Mesos's
> knowledge.
>
> -Emre
>
>
> Marlon
>
>
> From: Gourav Rattihalli <grattih1@binghamton.edu <
> mailto:grattih1@binghamton.edu <gr...@binghamton.edu>>>
> Reply-To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org
> <de...@airavata.apache.org>>" <dev@airavata.apache.org <
> mailto:dev@airavata.apache.org <de...@airavata.apache.org>>>
> Date: Wednesday, April 27, 2016 at 1:01 PM
> To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org
> <de...@airavata.apache.org>>" <dev@airavata.apache.org <
> mailto:dev@airavata.apache.org <de...@airavata.apache.org>>>
> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>
> Hi Dev's,
>
> I have been working on the integration of Apache Aurora and Mesos with
> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
> created using on Jetstream. I want to understand how Airavata is likely to
> use the VMs for applications. Will airavata re-use existing VMs for
> successive applications for a given community, or will new VMs be created
> for each application? This will decide how we design the automated creation
> of a Mesos cluster using the new VMs that are created. I understand that we
> will have a Mesos master VM that will run all the time on Jetstream.
>
> Here is the Job submission flow that I am assuming:
>
> Airavata Client--> Airavata
> Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will
> create .aurora configuration file that will be used to launch the
> job)-->Aurora/Mesos-->VMs
>
> So, I will design a module that will work with Orchestrator-->GFAC to
> generate the appropriate Apache Aurora script to be submitted to the Aurora
> master.
>
> Please let me know if there are any comments, suggestions on this plan.
>
> --
> Regards,
> Gourav Rattihalli
>
>
>
>
>
>
> --
> Regards,
> Gourav Rattihalli
>
>
>
--
Regards,
Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Suresh Marru <sm...@apache.org>.
On Apr 27, 2016, at 2:57 PM, Gourav Rattihalli <gr...@binghamton.edu> wrote:
>
>
> Hi Suresh,
>
> First of all I would like to commend you for starting this discussion. This is exactly kind of the mailing list engagement we would like to see as part of your GSOC. You already drew Emre’s attention, often discussions done in isolation undermine this impact. So good job on a start and we will expect over the summer you will engage lot more.
>
>
> Thanks. Yes, I plan to actively use this mailing list to keep my design focused on real use cases, and also to get regular feedback.
>
> Application dependency libraries, compiler versions can quickly get tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod <https://github.com/TACC/Lmod>) within a VM to alleviate this but that will be an over kill. Eventually containers will hopefully alleviate this problem.
>
>> The worker VMs need to have the Mesos libraries installed and the Mesos master needs to be configured with the IPs of the worker VMs. So, if new VMs are created each time, we will develop the playbook for the creation of the Mesos cluster on the fly, and then launch the application on it.
>
>
> Can the slave not register itself dynamically and report its IP Address? With either multiple applications per VM or one per VM, you will need to look into the capability of a persistent master and short lived elastic slave instances right? May be I am mis-understating your questions. Can you help clarify this more?
>
>
> Yes, this is possible, and was the first of the two options I had emailed.
Can you elaborate on the second option, may be I missed what you tried to convey? Lets put aside how many applications per vm for this discussion since thats orthogonal.
> To support this kind of elasticity, we will work on developing a script/playbook to install the Mesos libraries on the new slaves/agents and connect them with the master.
Can you please develop them in Airavata repo itself? I mean lets not make perfection be the enemy here and start sending pull requests as you bake these playbooks. You can put the scripts here - https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks <https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks> (note that it is develop branch).
You can read about contributions here - http://airavata.staging.apache.org/community/how-to-contribute-code.html <http://airavata.staging.apache.org/community/how-to-contribute-code.html> (we need to move these instructions to the new website).
Suresh
>
> Thanks,
> Gourav
>
>> On Apr 27, 2016, at 1:21 PM, Emre Brookes <emre@biochem.uthscsa.edu <ma...@biochem.uthscsa.edu>> wrote:
>>
>> Pierce, Marlon wrote:
>>> Thanks, Gourav, this is great. How will the design differ in the two cases?
>>>
>>> Since installing the applications is not a simple task and there aren’t good docker libraries (for example) for many of them, I would assume that we would have a collection of VMs that have the codes already installed.
>> FWIW, that model would work well with GenApp usage also. Although, I can imagine a scenario where a part of job initialization script (running under the already "code installed" VM) checks for updates to the code and installs them, but that can be independent of Airavata/Aurora/Mesos's knowledge.
>>
>> -Emre
>>
>>>
>>> Marlon
>>>
>>>
>>> From: Gourav Rattihalli <grattih1@binghamton.edu <ma...@binghamton.edu> <mailto:grattih1@binghamton.edu <ma...@binghamton.edu>>>
>>> Reply-To: "dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>" <dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>>
>>> Date: Wednesday, April 27, 2016 at 1:01 PM
>>> To: "dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>" <dev@airavata.apache.org <ma...@airavata.apache.org> <mailto:dev@airavata.apache.org <ma...@airavata.apache.org>>>
>>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>>>
>>> Hi Dev's,
>>>
>>> I have been working on the integration of Apache Aurora and Mesos with Airavata and Jetstream. Using Mangirish's latest code, VMs can now be created using on Jetstream. I want to understand how Airavata is likely to use the VMs for applications. Will airavata re-use existing VMs for successive applications for a given community, or will new VMs be created for each application? This will decide how we design the automated creation of a Mesos cluster using the new VMs that are created. I understand that we will have a Mesos master VM that will run all the time on Jetstream.
>>>
>>> Here is the Job submission flow that I am assuming:
>>>
>>> Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will create .aurora configuration file that will be used to launch the job)-->Aurora/Mesos-->VMs
>>>
>>> So, I will design a module that will work with Orchestrator-->GFAC to generate the appropriate Apache Aurora script to be submitted to the Aurora master.
>>>
>>> Please let me know if there are any comments, suggestions on this plan.
>>>
>>> --
>>> Regards,
>>> Gourav Rattihalli
>>
>
>
>
>
> --
> Regards,
> Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Gourav Rattihalli <gr...@binghamton.edu>.
Hi Suresh,
First of all I would like to commend you for starting this discussion. This
> is exactly kind of the mailing list engagement we would like to see as part
> of your GSOC. You already drew Emre’s attention, often discussions done in
> isolation undermine this impact. So good job on a start and we will expect
> over the summer you will engage lot more.
>
>
Thanks. Yes, I plan to actively use this mailing list to keep my design
focused on real use cases, and also to get regular feedback.
> Application dependency libraries, compiler versions can quickly get
> tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod)
> within a VM to alleviate this but that will be an over kill. Eventually
> containers will hopefully alleviate this problem.
>
> The worker VMs need to have the Mesos libraries installed and the Mesos
> master needs to be configured with the IPs of the worker VMs. So, if new
> VMs are created each time, we will develop the playbook for the creation of
> the Mesos cluster on the fly, and then launch the application on it.
>
>
> Can the slave not register itself dynamically and report its IP Address?
> With either multiple applications per VM or one per VM, you will need to
> look into the capability of a persistent master and short lived elastic
> slave instances right? May be I am mis-understating your questions. Can you
> help clarify this more?
>
>
Yes, this is possible, and was the first of the two options I had emailed.
To support this kind of elasticity, we will work on developing a
script/playbook to install the Mesos libraries on the new slaves/agents and
connect them with the master.
Thanks,
Gourav
On Apr 27, 2016, at 1:21 PM, Emre Brookes <em...@biochem.uthscsa.edu> wrote:
Pierce, Marlon wrote:
Thanks, Gourav, this is great. How will the design differ in the two cases?
Since installing the applications is not a simple task and there aren’t
good docker libraries (for example) for many of them, I would assume that
we would have a collection of VMs that have the codes already installed.
FWIW, that model would work well with GenApp usage also. Although, I can
imagine a scenario where a part of job initialization script (running under
the already "code installed" VM) checks for updates to the code and
installs them, but that can be independent of Airavata/Aurora/Mesos's
knowledge.
-Emre
Marlon
From: Gourav Rattihalli <grattih1@binghamton.edu <
mailto:grattih1@binghamton.edu <gr...@binghamton.edu>>>
Reply-To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org
<de...@airavata.apache.org>>" <dev@airavata.apache.org <
mailto:dev@airavata.apache.org <de...@airavata.apache.org>>>
Date: Wednesday, April 27, 2016 at 1:01 PM
To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org
<de...@airavata.apache.org>>" <dev@airavata.apache.org <
mailto:dev@airavata.apache.org <de...@airavata.apache.org>>>
Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
Hi Dev's,
I have been working on the integration of Apache Aurora and Mesos with
Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
created using on Jetstream. I want to understand how Airavata is likely to
use the VMs for applications. Will airavata re-use existing VMs for
successive applications for a given community, or will new VMs be created
for each application? This will decide how we design the automated creation
of a Mesos cluster using the new VMs that are created. I understand that we
will have a Mesos master VM that will run all the time on Jetstream.
Here is the Job submission flow that I am assuming:
Airavata Client--> Airavata
Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will
create .aurora configuration file that will be used to launch the
job)-->Aurora/Mesos-->VMs
So, I will design a module that will work with Orchestrator-->GFAC to
generate the appropriate Apache Aurora script to be submitted to the Aurora
master.
Please let me know if there are any comments, suggestions on this plan.
--
Regards,
Gourav Rattihalli
--
Regards,
Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Suresh Marru <sm...@apache.org>.
Hi Gourav,
First of all I would like to commend you for starting this discussion. This is exactly kind of the mailing list engagement we would like to see as part of your GSOC. You already drew Emre’s attention, often discussions done in isolation undermine this impact. So good job on a start and we will expect over the summer you will engage lot more.
Application dependency libraries, compiler versions can quickly get tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod <https://github.com/TACC/Lmod>) within a VM to alleviate this but that will be an over kill. Eventually containers will hopefully alleviate this problem.
> The worker VMs need to have the Mesos libraries installed and the Mesos master needs to be configured with the IPs of the worker VMs. So, if new VMs are created each time, we will develop the playbook for the creation of the Mesos cluster on the fly, and then launch the application on it.
Can the slave not register itself dynamically and report its IP Address? With either multiple applications per VM or one per VM, you will need to look into the capability of a persistent master and short lived elastic slave instances right? May be I am mis-understating your questions. Can you help clarify this more?
Suresh
> On Apr 27, 2016, at 1:21 PM, Emre Brookes <em...@biochem.uthscsa.edu> wrote:
>
> Pierce, Marlon wrote:
>> Thanks, Gourav, this is great. How will the design differ in the two cases?
>>
>> Since installing the applications is not a simple task and there aren’t good docker libraries (for example) for many of them, I would assume that we would have a collection of VMs that have the codes already installed.
> FWIW, that model would work well with GenApp usage also. Although, I can imagine a scenario where a part of job initialization script (running under the already "code installed" VM) checks for updates to the code and installs them, but that can be independent of Airavata/Aurora/Mesos's knowledge.
>
> -Emre
>
>>
>> Marlon
>>
>>
>> From: Gourav Rattihalli <grattih1@binghamton.edu <ma...@binghamton.edu>>
>> Reply-To: "dev@airavata.apache.org <ma...@airavata.apache.org>" <dev@airavata.apache.org <ma...@airavata.apache.org>>
>> Date: Wednesday, April 27, 2016 at 1:01 PM
>> To: "dev@airavata.apache.org <ma...@airavata.apache.org>" <dev@airavata.apache.org <ma...@airavata.apache.org>>
>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>>
>> Hi Dev's,
>>
>> I have been working on the integration of Apache Aurora and Mesos with Airavata and Jetstream. Using Mangirish's latest code, VMs can now be created using on Jetstream. I want to understand how Airavata is likely to use the VMs for applications. Will airavata re-use existing VMs for successive applications for a given community, or will new VMs be created for each application? This will decide how we design the automated creation of a Mesos cluster using the new VMs that are created. I understand that we will have a Mesos master VM that will run all the time on Jetstream.
>>
>> Here is the Job submission flow that I am assuming:
>>
>> Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will create .aurora configuration file that will be used to launch the job)-->Aurora/Mesos-->VMs
>>
>> So, I will design a module that will work with Orchestrator-->GFAC to generate the appropriate Apache Aurora script to be submitted to the Aurora master.
>>
>> Please let me know if there are any comments, suggestions on this plan.
>>
>> --
>> Regards,
>> Gourav Rattihalli
>
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by Emre Brookes <em...@biochem.uthscsa.edu>.
Pierce, Marlon wrote:
> Thanks, Gourav, this is great. How will the design differ in the two
> cases?
>
> Since installing the applications is not a simple task and there
> aren’t good docker libraries (for example) for many of them, I would
> assume that we would have a collection of VMs that have the codes
> already installed.
FWIW, that model would work well with GenApp usage also. Although, I
can imagine a scenario where a part of job initialization script
(running under the already "code installed" VM) checks for updates to
the code and installs them, but that can be independent of
Airavata/Aurora/Mesos's knowledge.
-Emre
>
> Marlon
>
>
> From: Gourav Rattihalli <grattih1@binghamton.edu
> <ma...@binghamton.edu>>
> Reply-To: "dev@airavata.apache.org <ma...@airavata.apache.org>"
> <dev@airavata.apache.org <ma...@airavata.apache.org>>
> Date: Wednesday, April 27, 2016 at 1:01 PM
> To: "dev@airavata.apache.org <ma...@airavata.apache.org>"
> <dev@airavata.apache.org <ma...@airavata.apache.org>>
> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
>
> Hi Dev's,
>
> I have been working on the integration of Apache Aurora and Mesos with
> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
> created using on Jetstream. I want to understand how Airavata is
> likely to use the VMs for applications. Will airavata re-use existing
> VMs for successive applications for a given community, or will new VMs
> be created for each application? This will decide how we design the
> automated creation of a Mesos cluster using the new VMs that are
> created. I understand that we will have a Mesos master VM that will
> run all the time on Jetstream.
>
> Here is the Job submission flow that I am assuming:
>
> Airavata Client--> Airavata
> Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script
> will create .aurora configuration file that will be used to launch the
> job)-->Aurora/Mesos-->VMs
>
> So, I will design a module that will work with Orchestrator-->GFAC to
> generate the appropriate Apache Aurora script to be submitted to the
> Aurora master.
>
> Please let me know if there are any comments, suggestions on this plan.
>
> --
> Regards,
> Gourav Rattihalli
Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Posted by "Pierce, Marlon" <ma...@iu.edu>.
Thanks, Gourav, this is great. How will the design differ in the two cases?
Since installing the applications is not a simple task and there aren’t good docker libraries (for example) for many of them, I would assume that we would have a collection of VMs that have the codes already installed.
Marlon
From: Gourav Rattihalli <gr...@binghamton.edu>>
Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Date: Wednesday, April 27, 2016 at 1:01 PM
To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Subject: Design of Mesos/Aurora integration with Airavata on Jetstream
Hi Dev's,
I have been working on the integration of Apache Aurora and Mesos with Airavata and Jetstream. Using Mangirish's latest code, VMs can now be created using on Jetstream. I want to understand how Airavata is likely to use the VMs for applications. Will airavata re-use existing VMs for successive applications for a given community, or will new VMs be created for each application? This will decide how we design the automated creation of a Mesos cluster using the new VMs that are created. I understand that we will have a Mesos master VM that will run all the time on Jetstream.
Here is the Job submission flow that I am assuming:
Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will create .aurora configuration file that will be used to launch the job)-->Aurora/Mesos-->VMs
So, I will design a module that will work with Orchestrator-->GFAC to generate the appropriate Apache Aurora script to be submitted to the Aurora master.
Please let me know if there are any comments, suggestions on this plan.
--
Regards,
Gourav Rattihalli