You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Pankaj Saha <ps...@binghamton.edu> on 2015/11/14 20:06:36 UTC
Re: launching a job through Airavata to Mesos cluster
Hi Suresh and Marlon
I am trying to launch a hello world sample application from my local docker
Airavata PGA. Can I get a demo admin role for few days so that I can create
an application and test some sample experiments? It looks like without
admin role I can not create applications.
Thanks
Pankaj
On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <ma...@iu.edu> wrote:
> But you could answer my implied question below with zero knowledge of
> Airavata.
>
> From: Pankaj Saha <ps...@binghamton.edu>
> Reply-To: Airavata Dev <de...@airavata.apache.org>
> Date: Wednesday, October 28, 2015 at 1:48 PM
>
> To: Marlon Pierce <ma...@iu.edu>
> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
> dev@airavata.apache.org>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> As Suresh said I think this is the time when I have to understand under
> the hood details of Airavata Architecture, which I could not gain much
> during GSoC.
>
> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>
>> The “hello world” for us is usually to just run some linux command on the
>> remote queuing system. So that is something you could easily wrap via
>> docker and submit via Mesos (I hope it is easy…). Related to this is
>> determining that the job has gone through various states (queued,
>> executing, completed) and communicating that back to Airavata.
>>
>>
>> Marlon
>>
>>
>> From: Pankaj Saha <ps...@binghamton.edu>
>> Date: Wednesday, October 28, 2015 at 12:54 PM
>> To: Marlon Pierce <ma...@iu.edu>
>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>> dev@airavata.apache.org>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Marlon,
>> "we more typically use Airavata to run scientific applications on
>> clusters" - Our plan is to run dockerized scientific application through
>> Airavata.
>> Definitely our target is not to run a Tomcat webserver, I just said that
>> as an example.
>>
>> Give me some time, after I discuss with Prof. Madhu and Renan I will try
>> to reply your queries in details.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>>
>>> Hi Pankaj—
>>>
>>> I may not have the full background on what you want to do, but we more
>>> typically use Airavata to run scientific applications on clusters. This
>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM). If
>>> you want to do this, you would first need to say how you would submit a
>>> dockerized application using Mesos as your queueing system. You could do
>>> this for a Tomcat server, but that is a little different from the jobs we
>>> normally run.
>>>
>>> We don’t use Airavata to bootstrap itself, although that may be an
>>> interesting way to think of things. There it is interesting to use Mesos +
>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>> that would be done today outside Airavata.
>>>
>>> Marlon
>>>
>>>
>>> From: Suresh Marru <sm...@apache.org>
>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>> To: Airavata Dev <de...@airavata.apache.org>
>>> Cc: Marlon Pierce <ma...@iu.edu>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Pankaj,
>>>
>>> You cannot do this out of the box now, but with some guidance you can
>>> easily write a task to run Dockerized application on a Mesos cluster,
>>> please use 0.16 for this. As Shameera explained this is a improvised
>>> architecture and any contributions you will make will carry forward. Note:
>>> This will require you to understand under the hood details of Airavata
>>> Architecture but I am sure you will motivate others to work with you to
>>> speed it up.
>>>
>>> Suresh
>>>
>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <ps...@binghamton.edu> wrote:
>>>
>>> “launch a dockerized job” means submitting a docker container for any
>>> specific job (say a java webserver docker container) through Airavata.
>>> What I remember Airavata php gateway can not submit any docker
>>> container(correct me if I am wrong). What I want to figure out initially is
>>> whether its feasible to submit a docker container with help of Airavata
>>> into a Mesos cluster.
>>>
>>> Thanks
>>> Pankaj
>>>
>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <ma...@iu.edu>
>>> wrote:
>>>
>>>> I’ll add: if submitting a job to (for example) a SLURM queuing system,
>>>> we need to create the correct SLURM submission script and submit it by
>>>> executing the correct command line operation (sbatch).
>>>>
>>>> From: Marlon Pierce <ma...@iu.edu>
>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>> To: Airavata Dev <de...@airavata.apache.org>, Suresh Marru <
>>>> smarru@apache.org>, Pankaj Saha <ps...@binghamton.edu>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>>>> job”?
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>> To: Suresh Marru <sm...@apache.org>
>>>> Cc: Airavata Dev <de...@airavata.apache.org>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Suresh,
>>>>
>>>> My initial understanding is, I have to launch a dockerized job through
>>>> Airavata which will be run in the Mesos cluster. I was looking for the
>>>> code which submits jobs and wanted to make changes such a way that it can
>>>> submit docker containers to Mesos/Marathon cluster.
>>>>
>>>> I can use 0.15 branch and I have no idea about data transfer protocol
>>>> and job submission protocols that Shameera has mentioned. I may want to
>>>> submit jobs by submitting a JSON through command line or any other way that
>>>> you guys feel is more appropriate.
>>>>
>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Pankaj can you clarify the following:
>>>>>
>>>>> Do you want an Airavata instance to run some dockerized applications
>>>>> scheduled by Mesos? Or do you just need a client which will connect to
>>>>> Airavata hosted and managed by Mesos/Marathon?
>>>>>
>>>>> Suresh
>>>>>
>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>> shameerainfo@gmail.com> wrote:
>>>>>
>>>>> Hi Pankaj,
>>>>>
>>>>> Wich version of Airavata you are working on? what is the data
>>>>> transfer protocol? What is the job submission protocol?
>>>>>
>>>>> Short answer: if you are using Airavata 0.15 then you need to write
>>>>> new Provider implementation to submit the request to Mesos/Marathon
>>>>> cluster. But if you are using Airavata 16.0 which is current master, then
>>>>> you need to write JobSubmissionTask implementation. Either case you can go
>>>>> through the existing implementations, for Provider implementation see
>>>>> SSHProvider and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>
>>>>> If I get the answers to my questions then i can provide exactly what
>>>>> you need to do. BTW we have cleaned our internal architecture in Airavata
>>>>> 16.0, as a developer you would find it easy to work with Airavata 16.0 that
>>>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>>>
>>>>> Regards,
>>>>> Shameera.
>>>>>
>>>>>
>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <ps...@binghamton.edu>
>>>>> wrote:
>>>>>
>>>>>> Hello Shameera,
>>>>>> I am working on jet-stream project, where I have to find out a way to
>>>>>> submit a job in mesos/marathon cluster through Airavata client. I don't
>>>>>> have much idea from where to start looking into. Can you please give some
>>>>>> clue so that I can start working and making changes to java code for the
>>>>>> same.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>> --
>>>>> Shameera Rathnayaka
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: launching a job through Airavata to Mesos cluster
Posted by Supun Nakandala <su...@gmail.com>.
yes Pankaj. Even though the roles information are in IS the capabilities
for different roles are defined in the PGA. Every user by default has the
role "Internal/everyone". So when you PGA admin-role to that you will gain
admin capabilities in your PGA.
On Sat, Nov 14, 2015 at 3:05 PM, Pankaj Saha <ps...@binghamton.edu> wrote:
> Hi Supun,
> I need a small clarification here. I am using local docker for PGA and
> servers but I am using the sam credential service that you guys are using.
> So do you think making local changes in the pga_config will do the work?
>
> Thanks
> Pankaj
>
> On Sat, Nov 14, 2015 at 3:01 PM, Supun Nakandala <
> supun.nakandala@gmail.com> wrote:
>
>> Hi Pankaj,
>>
>> You can set to admin role to Internal/everyone in pga_config.php (in
>> <PGA_ROOT>/app/config/pga_config.php) and you will admin permissions.
>>
>> On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <ps...@binghamton.edu>
>> wrote:
>>
>>> Hi Suresh and Marlon
>>> I am trying to launch a hello world sample application from my local
>>> docker Airavata PGA. Can I get a demo admin role for few days so that I can
>>> create an application and test some sample experiments? It looks like
>>> without admin role I can not create applications.
>>>
>>> Thanks
>>> Pankaj
>>>
>>> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>>>
>>>> But you could answer my implied question below with zero knowledge of
>>>> Airavata.
>>>>
>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>>>
>>>> To: Marlon Pierce <ma...@iu.edu>
>>>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>>>> dev@airavata.apache.org>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> As Suresh said I think this is the time when I have to understand
>>>> under the hood details of Airavata Architecture, which I could not gain
>>>> much during GSoC.
>>>>
>>>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <ma...@iu.edu>
>>>> wrote:
>>>>
>>>>> The “hello world” for us is usually to just run some linux command on
>>>>> the remote queuing system. So that is something you could easily wrap via
>>>>> docker and submit via Mesos (I hope it is easy…). Related to this is
>>>>> determining that the job has gone through various states (queued,
>>>>> executing, completed) and communicating that back to Airavata.
>>>>>
>>>>>
>>>>> Marlon
>>>>>
>>>>>
>>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>>>> To: Marlon Pierce <ma...@iu.edu>
>>>>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>>>>> dev@airavata.apache.org>
>>>>>
>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>
>>>>> Hi Marlon,
>>>>> "we more typically use Airavata to run scientific applications on
>>>>> clusters" - Our plan is to run dockerized scientific application
>>>>> through Airavata.
>>>>> Definitely our target is not to run a Tomcat webserver, I just said
>>>>> that as an example.
>>>>>
>>>>> Give me some time, after I discuss with Prof. Madhu and Renan I will
>>>>> try to reply your queries in details.
>>>>>
>>>>> Thanks
>>>>> Pankaj
>>>>>
>>>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <ma...@iu.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Pankaj—
>>>>>>
>>>>>> I may not have the full background on what you want to do, but we
>>>>>> more typically use Airavata to run scientific applications on clusters.
>>>>>> This goes through a scheduling and queuing system (Torque/MOAB or SLURM).
>>>>>> If you want to do this, you would first need to say how you would submit a
>>>>>> dockerized application using Mesos as your queueing system. You could do
>>>>>> this for a Tomcat server, but that is a little different from the jobs we
>>>>>> normally run.
>>>>>>
>>>>>> We don’t use Airavata to bootstrap itself, although that may be an
>>>>>> interesting way to think of things. There it is interesting to use Mesos +
>>>>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>>>>> that would be done today outside Airavata.
>>>>>>
>>>>>> Marlon
>>>>>>
>>>>>>
>>>>>> From: Suresh Marru <sm...@apache.org>
>>>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>>>> To: Airavata Dev <de...@airavata.apache.org>
>>>>>> Cc: Marlon Pierce <ma...@iu.edu>
>>>>>>
>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>
>>>>>> Hi Pankaj,
>>>>>>
>>>>>> You cannot do this out of the box now, but with some guidance you can
>>>>>> easily write a task to run Dockerized application on a Mesos cluster,
>>>>>> please use 0.16 for this. As Shameera explained this is a improvised
>>>>>> architecture and any contributions you will make will carry forward. Note:
>>>>>> This will require you to understand under the hood details of Airavata
>>>>>> Architecture but I am sure you will motivate others to work with you to
>>>>>> speed it up.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <ps...@binghamton.edu>
>>>>>> wrote:
>>>>>>
>>>>>> “launch a dockerized job” means submitting a docker container for
>>>>>> any specific job (say a java webserver docker container) through Airavata.
>>>>>> What I remember Airavata php gateway can not submit any docker
>>>>>> container(correct me if I am wrong). What I want to figure out initially is
>>>>>> whether its feasible to submit a docker container with help of Airavata
>>>>>> into a Mesos cluster.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <ma...@iu.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> I’ll add: if submitting a job to (for example) a SLURM queuing
>>>>>>> system, we need to create the correct SLURM submission script and submit it
>>>>>>> by executing the correct command line operation (sbatch).
>>>>>>>
>>>>>>> From: Marlon Pierce <ma...@iu.edu>
>>>>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>>>>> To: Airavata Dev <de...@airavata.apache.org>, Suresh Marru <
>>>>>>> smarru@apache.org>, Pankaj Saha <ps...@binghamton.edu>
>>>>>>>
>>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>>
>>>>>>> Hi Pankaj, can you say more about what you mean by “launch a
>>>>>>> dockerized job”?
>>>>>>>
>>>>>>> Marlon
>>>>>>>
>>>>>>>
>>>>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>>>>> To: Suresh Marru <sm...@apache.org>
>>>>>>> Cc: Airavata Dev <de...@airavata.apache.org>
>>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>>
>>>>>>> Hi Suresh,
>>>>>>>
>>>>>>> My initial understanding is, I have to launch a dockerized job
>>>>>>> through Airavata which will be run in the Mesos cluster. I was looking for
>>>>>>> the code which submits jobs and wanted to make changes such a way that it
>>>>>>> can submit docker containers to Mesos/Marathon cluster.
>>>>>>>
>>>>>>> I can use 0.15 branch and I have no idea about data transfer
>>>>>>> protocol and job submission protocols that Shameera has mentioned. I may
>>>>>>> want to submit jobs by submitting a JSON through command line or any other
>>>>>>> way that you guys feel is more appropriate.
>>>>>>>
>>>>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Pankaj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sm...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Pankaj can you clarify the following:
>>>>>>>>
>>>>>>>> Do you want an Airavata instance to run some dockerized
>>>>>>>> applications scheduled by Mesos? Or do you just need a client which will
>>>>>>>> connect to Airavata hosted and managed by Mesos/Marathon?
>>>>>>>>
>>>>>>>> Suresh
>>>>>>>>
>>>>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Pankaj,
>>>>>>>>
>>>>>>>> Wich version of Airavata you are working on? what is the data
>>>>>>>> transfer protocol? What is the job submission protocol?
>>>>>>>>
>>>>>>>> Short answer: if you are using Airavata 0.15 then you need to
>>>>>>>> write new Provider implementation to submit the request to Mesos/Marathon
>>>>>>>> cluster. But if you are using Airavata 16.0 which is current master, then
>>>>>>>> you need to write JobSubmissionTask implementation. Either case you can go
>>>>>>>> through the existing implementations, for Provider implementation see
>>>>>>>> SSHProvider and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>>>>
>>>>>>>> If I get the answers to my questions then i can provide exactly
>>>>>>>> what you need to do. BTW we have cleaned our internal architecture in
>>>>>>>> Airavata 16.0, as a developer you would find it easy to work with Airavata
>>>>>>>> 16.0 that Airavata 15.0. But notice master is not yet stable as Airavata
>>>>>>>> 15.0.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Shameera.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <ps...@binghamton.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello Shameera,
>>>>>>>>> I am working on jet-stream project, where I have to find out a way
>>>>>>>>> to submit a job in mesos/marathon cluster through Airavata client. I don't
>>>>>>>>> have much idea from where to start looking into. Can you please give some
>>>>>>>>> clue so that I can start working and making changes to java code for the
>>>>>>>>> same.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Pankaj
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Shameera Rathnayaka
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Thank you
>> Supun Nakandala
>> Dept. Computer Science and Engineering
>> University of Moratuwa
>>
>
>
--
Thank you
Supun Nakandala
Dept. Computer Science and Engineering
University of Moratuwa
Re: launching a job through Airavata to Mesos cluster
Posted by Pankaj Saha <ps...@binghamton.edu>.
Hi Supun,
I need a small clarification here. I am using local docker for PGA and
servers but I am using the sam credential service that you guys are using.
So do you think making local changes in the pga_config will do the work?
Thanks
Pankaj
On Sat, Nov 14, 2015 at 3:01 PM, Supun Nakandala <su...@gmail.com>
wrote:
> Hi Pankaj,
>
> You can set to admin role to Internal/everyone in pga_config.php (in
> <PGA_ROOT>/app/config/pga_config.php) and you will admin permissions.
>
> On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <ps...@binghamton.edu>
> wrote:
>
>> Hi Suresh and Marlon
>> I am trying to launch a hello world sample application from my local
>> docker Airavata PGA. Can I get a demo admin role for few days so that I can
>> create an application and test some sample experiments? It looks like
>> without admin role I can not create applications.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>>
>>> But you could answer my implied question below with zero knowledge of
>>> Airavata.
>>>
>>> From: Pankaj Saha <ps...@binghamton.edu>
>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>>
>>> To: Marlon Pierce <ma...@iu.edu>
>>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>>> dev@airavata.apache.org>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> As Suresh said I think this is the time when I have to understand
>>> under the hood details of Airavata Architecture, which I could not gain
>>> much during GSoC.
>>>
>>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>>>
>>>> The “hello world” for us is usually to just run some linux command on
>>>> the remote queuing system. So that is something you could easily wrap via
>>>> docker and submit via Mesos (I hope it is easy…). Related to this is
>>>> determining that the job has gone through various states (queued,
>>>> executing, completed) and communicating that back to Airavata.
>>>>
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>>> To: Marlon Pierce <ma...@iu.edu>
>>>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>>>> dev@airavata.apache.org>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Marlon,
>>>> "we more typically use Airavata to run scientific applications on
>>>> clusters" - Our plan is to run dockerized scientific application
>>>> through Airavata.
>>>> Definitely our target is not to run a Tomcat webserver, I just said
>>>> that as an example.
>>>>
>>>> Give me some time, after I discuss with Prof. Madhu and Renan I will
>>>> try to reply your queries in details.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <ma...@iu.edu>
>>>> wrote:
>>>>
>>>>> Hi Pankaj—
>>>>>
>>>>> I may not have the full background on what you want to do, but we more
>>>>> typically use Airavata to run scientific applications on clusters. This
>>>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM). If
>>>>> you want to do this, you would first need to say how you would submit a
>>>>> dockerized application using Mesos as your queueing system. You could do
>>>>> this for a Tomcat server, but that is a little different from the jobs we
>>>>> normally run.
>>>>>
>>>>> We don’t use Airavata to bootstrap itself, although that may be an
>>>>> interesting way to think of things. There it is interesting to use Mesos +
>>>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>>>> that would be done today outside Airavata.
>>>>>
>>>>> Marlon
>>>>>
>>>>>
>>>>> From: Suresh Marru <sm...@apache.org>
>>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>>> To: Airavata Dev <de...@airavata.apache.org>
>>>>> Cc: Marlon Pierce <ma...@iu.edu>
>>>>>
>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>
>>>>> Hi Pankaj,
>>>>>
>>>>> You cannot do this out of the box now, but with some guidance you can
>>>>> easily write a task to run Dockerized application on a Mesos cluster,
>>>>> please use 0.16 for this. As Shameera explained this is a improvised
>>>>> architecture and any contributions you will make will carry forward. Note:
>>>>> This will require you to understand under the hood details of Airavata
>>>>> Architecture but I am sure you will motivate others to work with you to
>>>>> speed it up.
>>>>>
>>>>> Suresh
>>>>>
>>>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <ps...@binghamton.edu>
>>>>> wrote:
>>>>>
>>>>> “launch a dockerized job” means submitting a docker container for any
>>>>> specific job (say a java webserver docker container) through Airavata.
>>>>> What I remember Airavata php gateway can not submit any docker
>>>>> container(correct me if I am wrong). What I want to figure out initially is
>>>>> whether its feasible to submit a docker container with help of Airavata
>>>>> into a Mesos cluster.
>>>>>
>>>>> Thanks
>>>>> Pankaj
>>>>>
>>>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <ma...@iu.edu>
>>>>> wrote:
>>>>>
>>>>>> I’ll add: if submitting a job to (for example) a SLURM queuing
>>>>>> system, we need to create the correct SLURM submission script and submit it
>>>>>> by executing the correct command line operation (sbatch).
>>>>>>
>>>>>> From: Marlon Pierce <ma...@iu.edu>
>>>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>>>> To: Airavata Dev <de...@airavata.apache.org>, Suresh Marru <
>>>>>> smarru@apache.org>, Pankaj Saha <ps...@binghamton.edu>
>>>>>>
>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>
>>>>>> Hi Pankaj, can you say more about what you mean by “launch a
>>>>>> dockerized job”?
>>>>>>
>>>>>> Marlon
>>>>>>
>>>>>>
>>>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>>>> To: Suresh Marru <sm...@apache.org>
>>>>>> Cc: Airavata Dev <de...@airavata.apache.org>
>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>
>>>>>> Hi Suresh,
>>>>>>
>>>>>> My initial understanding is, I have to launch a dockerized job
>>>>>> through Airavata which will be run in the Mesos cluster. I was looking for
>>>>>> the code which submits jobs and wanted to make changes such a way that it
>>>>>> can submit docker containers to Mesos/Marathon cluster.
>>>>>>
>>>>>> I can use 0.15 branch and I have no idea about data transfer protocol
>>>>>> and job submission protocols that Shameera has mentioned. I may want to
>>>>>> submit jobs by submitting a JSON through command line or any other way that
>>>>>> you guys feel is more appropriate.
>>>>>>
>>>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sm...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Pankaj can you clarify the following:
>>>>>>>
>>>>>>> Do you want an Airavata instance to run some dockerized applications
>>>>>>> scheduled by Mesos? Or do you just need a client which will connect to
>>>>>>> Airavata hosted and managed by Mesos/Marathon?
>>>>>>>
>>>>>>> Suresh
>>>>>>>
>>>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Pankaj,
>>>>>>>
>>>>>>> Wich version of Airavata you are working on? what is the data
>>>>>>> transfer protocol? What is the job submission protocol?
>>>>>>>
>>>>>>> Short answer: if you are using Airavata 0.15 then you need to write
>>>>>>> new Provider implementation to submit the request to Mesos/Marathon
>>>>>>> cluster. But if you are using Airavata 16.0 which is current master, then
>>>>>>> you need to write JobSubmissionTask implementation. Either case you can go
>>>>>>> through the existing implementations, for Provider implementation see
>>>>>>> SSHProvider and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>>>
>>>>>>> If I get the answers to my questions then i can provide exactly what
>>>>>>> you need to do. BTW we have cleaned our internal architecture in Airavata
>>>>>>> 16.0, as a developer you would find it easy to work with Airavata 16.0 that
>>>>>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Shameera.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <ps...@binghamton.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello Shameera,
>>>>>>>> I am working on jet-stream project, where I have to find out a way
>>>>>>>> to submit a job in mesos/marathon cluster through Airavata client. I don't
>>>>>>>> have much idea from where to start looking into. Can you please give some
>>>>>>>> clue so that I can start working and making changes to java code for the
>>>>>>>> same.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Pankaj
>>>>>>>>
>>>>>>>> --
>>>>>>> Shameera Rathnayaka
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Thank you
> Supun Nakandala
> Dept. Computer Science and Engineering
> University of Moratuwa
>
Re: launching a job through Airavata to Mesos cluster
Posted by Supun Nakandala <su...@gmail.com>.
Hi Pankaj,
You can set to admin role to Internal/everyone in pga_config.php (in
<PGA_ROOT>/app/config/pga_config.php) and you will admin permissions.
On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <ps...@binghamton.edu> wrote:
> Hi Suresh and Marlon
> I am trying to launch a hello world sample application from my local
> docker Airavata PGA. Can I get a demo admin role for few days so that I can
> create an application and test some sample experiments? It looks like
> without admin role I can not create applications.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>
>> But you could answer my implied question below with zero knowledge of
>> Airavata.
>>
>> From: Pankaj Saha <ps...@binghamton.edu>
>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>
>> To: Marlon Pierce <ma...@iu.edu>
>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>> dev@airavata.apache.org>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> As Suresh said I think this is the time when I have to understand under
>> the hood details of Airavata Architecture, which I could not gain much
>> during GSoC.
>>
>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <ma...@iu.edu> wrote:
>>
>>> The “hello world” for us is usually to just run some linux command on
>>> the remote queuing system. So that is something you could easily wrap via
>>> docker and submit via Mesos (I hope it is easy…). Related to this is
>>> determining that the job has gone through various states (queued,
>>> executing, completed) and communicating that back to Airavata.
>>>
>>>
>>> Marlon
>>>
>>>
>>> From: Pankaj Saha <ps...@binghamton.edu>
>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>> To: Marlon Pierce <ma...@iu.edu>
>>> Cc: Suresh Marru <sm...@apache.org>, Airavata Dev <
>>> dev@airavata.apache.org>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Marlon,
>>> "we more typically use Airavata to run scientific applications on
>>> clusters" - Our plan is to run dockerized scientific application
>>> through Airavata.
>>> Definitely our target is not to run a Tomcat webserver, I just said that
>>> as an example.
>>>
>>> Give me some time, after I discuss with Prof. Madhu and Renan I will try
>>> to reply your queries in details.
>>>
>>> Thanks
>>> Pankaj
>>>
>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <ma...@iu.edu>
>>> wrote:
>>>
>>>> Hi Pankaj—
>>>>
>>>> I may not have the full background on what you want to do, but we more
>>>> typically use Airavata to run scientific applications on clusters. This
>>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM). If
>>>> you want to do this, you would first need to say how you would submit a
>>>> dockerized application using Mesos as your queueing system. You could do
>>>> this for a Tomcat server, but that is a little different from the jobs we
>>>> normally run.
>>>>
>>>> We don’t use Airavata to bootstrap itself, although that may be an
>>>> interesting way to think of things. There it is interesting to use Mesos +
>>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>>> that would be done today outside Airavata.
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Suresh Marru <sm...@apache.org>
>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>> To: Airavata Dev <de...@airavata.apache.org>
>>>> Cc: Marlon Pierce <ma...@iu.edu>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Pankaj,
>>>>
>>>> You cannot do this out of the box now, but with some guidance you can
>>>> easily write a task to run Dockerized application on a Mesos cluster,
>>>> please use 0.16 for this. As Shameera explained this is a improvised
>>>> architecture and any contributions you will make will carry forward. Note:
>>>> This will require you to understand under the hood details of Airavata
>>>> Architecture but I am sure you will motivate others to work with you to
>>>> speed it up.
>>>>
>>>> Suresh
>>>>
>>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <ps...@binghamton.edu>
>>>> wrote:
>>>>
>>>> “launch a dockerized job” means submitting a docker container for any
>>>> specific job (say a java webserver docker container) through Airavata.
>>>> What I remember Airavata php gateway can not submit any docker
>>>> container(correct me if I am wrong). What I want to figure out initially is
>>>> whether its feasible to submit a docker container with help of Airavata
>>>> into a Mesos cluster.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <ma...@iu.edu>
>>>> wrote:
>>>>
>>>>> I’ll add: if submitting a job to (for example) a SLURM queuing system,
>>>>> we need to create the correct SLURM submission script and submit it by
>>>>> executing the correct command line operation (sbatch).
>>>>>
>>>>> From: Marlon Pierce <ma...@iu.edu>
>>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>>> To: Airavata Dev <de...@airavata.apache.org>, Suresh Marru <
>>>>> smarru@apache.org>, Pankaj Saha <ps...@binghamton.edu>
>>>>>
>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>
>>>>> Hi Pankaj, can you say more about what you mean by “launch a
>>>>> dockerized job”?
>>>>>
>>>>> Marlon
>>>>>
>>>>>
>>>>> From: Pankaj Saha <ps...@binghamton.edu>
>>>>> Reply-To: Airavata Dev <de...@airavata.apache.org>
>>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>>> To: Suresh Marru <sm...@apache.org>
>>>>> Cc: Airavata Dev <de...@airavata.apache.org>
>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>
>>>>> Hi Suresh,
>>>>>
>>>>> My initial understanding is, I have to launch a dockerized job through
>>>>> Airavata which will be run in the Mesos cluster. I was looking for the
>>>>> code which submits jobs and wanted to make changes such a way that it can
>>>>> submit docker containers to Mesos/Marathon cluster.
>>>>>
>>>>> I can use 0.15 branch and I have no idea about data transfer protocol
>>>>> and job submission protocols that Shameera has mentioned. I may want to
>>>>> submit jobs by submitting a JSON through command line or any other way that
>>>>> you guys feel is more appropriate.
>>>>>
>>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>>
>>>>> Thanks
>>>>> Pankaj
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Pankaj can you clarify the following:
>>>>>>
>>>>>> Do you want an Airavata instance to run some dockerized applications
>>>>>> scheduled by Mesos? Or do you just need a client which will connect to
>>>>>> Airavata hosted and managed by Mesos/Marathon?
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>
>>>>>> Hi Pankaj,
>>>>>>
>>>>>> Wich version of Airavata you are working on? what is the data
>>>>>> transfer protocol? What is the job submission protocol?
>>>>>>
>>>>>> Short answer: if you are using Airavata 0.15 then you need to write
>>>>>> new Provider implementation to submit the request to Mesos/Marathon
>>>>>> cluster. But if you are using Airavata 16.0 which is current master, then
>>>>>> you need to write JobSubmissionTask implementation. Either case you can go
>>>>>> through the existing implementations, for Provider implementation see
>>>>>> SSHProvider and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>>
>>>>>> If I get the answers to my questions then i can provide exactly what
>>>>>> you need to do. BTW we have cleaned our internal architecture in Airavata
>>>>>> 16.0, as a developer you would find it easy to work with Airavata 16.0 that
>>>>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>>>>
>>>>>> Regards,
>>>>>> Shameera.
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <ps...@binghamton.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Shameera,
>>>>>>> I am working on jet-stream project, where I have to find out a way
>>>>>>> to submit a job in mesos/marathon cluster through Airavata client. I don't
>>>>>>> have much idea from where to start looking into. Can you please give some
>>>>>>> clue so that I can start working and making changes to java code for the
>>>>>>> same.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Pankaj
>>>>>>>
>>>>>>> --
>>>>>> Shameera Rathnayaka
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
--
Thank you
Supun Nakandala
Dept. Computer Science and Engineering
University of Moratuwa