Clarification about Airavata GFac

[Nipun Udara <y....@gmail.com>]

Hi all,

I was trying to understand GFac module and found different providers which
use to execute jobs like BESProvider and GRAMProvider which extends
AbstractProvider class like to know the use of those.

And when search abouth EC2 instances there are three different kind of
instances such as reserved instances ,spot instances. i would like to know
out of those which kind is using in GFac-ec2 module.And is it need to
configure to pass user-data scripts that runs the job each time launch a
job. I would be grateful if somebody can clarify regarding these.

Regards
Nipun Udara

*Undergraduate*
*Department of Computer Science & Engineering*
*University of Moratuwa*
*Sri Lanka*

Re: Clarification about Airavata GFac

[Saminda Wijeratne <sa...@gmail.com>]

GFac Providers are for different computing resources (UNICORE/GRAM/SSH/EC2)
which have different ways of handing application execution (more accurate
word is job submission). But details of knowing, selecting and configuring
a provider is not the scope of a general user. Thus GFac hides provider
details through the GFacAPI (soon to be re-introduced in a different form
as the GFacCPI) by giving the application an abstract interface (which
consists of the application name, its inputs and outputs). A general user
will request job submissions through GFac using this abstract interface.
GFac will look through the registry to retrieve application deployment
details that correspond to the abstract interface and based on deployment,
it will select the correct GFac Provider and pass on the request to that
provider.


On Tue, Mar 4, 2014 at 1:32 PM, Nipun Udara <y....@gmail.com> wrote:

> Hi all,
>
> I was trying to understand GFac module and found different providers which
> use to execute jobs like BESProvider and GRAMProvider which extends
> AbstractProvider class like to know the use of those.
>
> And when search abouth EC2 instances there are three different kind of
> instances such as reserved instances ,spot instances. i would like to know
> out of those which kind is using in GFac-ec2 module.And is it need to
> configure to pass user-data scripts that runs the job each time launch a
> job. I would be grateful if somebody can clarify regarding these.
>
> Regards
> Nipun Udara
>
> *Undergraduate*
> *Department of Computer Science & Engineering*
> *University of Moratuwa*
> *Sri Lanka*
>

Re: Clarification about Airavata GFac

[Nipun Udara <y....@gmail.com>]

Hi saminda and Heshan,

Thank you very much for the detailed explanation.


On Wed, Mar 5, 2014 at 11:04 PM, Heshan Suriyaarachchi <
heshan.suriyaarachchi@gmail.com> wrote:

> Hello Nipun,
>
> Please see my inline comments.
>
>
> On Tue, Mar 4, 2014 at 10:32 AM, Nipun Udara <y....@gmail.com>wrote:
>
>> Hi all,
>>
>> I was trying to understand GFac module and found different providers
>> which use to execute jobs like BESProvider and GRAMProvider which extends
>> AbstractProvider class like to know the use of those.
>>
>> And when search abouth EC2 instances there are three different kind of
>> instances such as reserved instances ,spot instances. i would like to know
>> out of those which kind is using in GFac-ec2 module.
>>
>
> As you might know, Amazon provides a java-sdk to communicate with it's
> AWS. EC2Provdier uses it to interact with AMIs and instances. When testing
> out or running a workflow, I usually used a general purpose instance as I
> did not have any requirement for running a workflow in compute optimized
> instance family. Having said that, you will be able to run a workflow in
> any kind of instance that you want. You might have to tweak the UI a bit to
> support this (ie. like adding an option to a drop down, etc and modifying
> the backend a bit).
>
> As for the sample workouts that you might be planning to run, you don't
> need to run them in compute optimized instances as your workflow might not
> be using the full potential of the resources. Keep in mind you are paying
> for the instance type as well. As you reserve an instance with higher
> compute power, you might be paying more (You might want to check the
> pricing of instances before running sample workflows in compute optimized
> instances).
>
> So, the short answer to your question might be, you can use which ever
> instance type you want as long as the AWS java-sdk api is exposing the
> capability of interacting with those instances (which they do).
>
> And is it need to configure to pass user-data scripts that runs the job
>> each time launch a job. I would be grateful if somebody can clarify
>> regarding these.
>>
>
> Let me tell you how the EC2 job submissions work in Airavata.
>
> First you should have a base image which has your application installed.
> This application can be a jar/shell script or other kind of a script. Try
> and wrap your application with a shell script and make it available for
> shell execution.
>
> The EC2 Provider will get the user information (secret key, access key,
> AMI id) and start up an instance from the base image (which has your
> application installed.) Once the instance is running, the EC2 provider
> establishes a SSH connection to the running instance and then runs the
> shell script (which wraps your application) with the user given input
> params. Provider will poll the standard out from the script and extract the
> result out. Then it will be returned to the provider. Then this will passed
> on to the workflow.
>
> AFAICR I used J2ssh for the ssh connections. Since you are getting the
> shell access to the instance, you might be able to run any script type that
> you want but please remember that if you are using any script that is
> dependent on the environment/version, you might run into some issues (eg.
> Running a python 3.0 script on a Python 2.5 system). These things are also
> minor issues and can be easily overcome. The reason why I told you this is
> that, then you would not get side tracked because of any environment
> issues.
>
> I didn't check Airavata mailing list for a long time. Therefore, didn't
> see this. :) Lahiru, pointed me to this thread. If you have any
> questions/doubts, please don't hesitate to ask.
>
> Thanks,
> Heshan.
>
>>
>> Regards
>> Nipun Udara
>>
>> *Undergraduate*
>> *Department of Computer Science & Engineering*
>> *University of Moratuwa*
>> *Sri Lanka*
>>
>
>
>
> --
> Regards,
> Heshan Suriyaarachchi
>
> http://heshans.blogspot.com/
> http://www.linkedin.com/in/heshan
>



-- 
Nipun Udara

*Undergraduate*
*Department of Computer Science & Engineering*
*University of Moratuwa*
*Sri Lanka*

Re: Clarification about Airavata GFac

[Heshan Suriyaarachchi <he...@gmail.com>]

Hello Nipun,

Please see my inline comments.


On Tue, Mar 4, 2014 at 10:32 AM, Nipun Udara <y....@gmail.com> wrote:

> Hi all,
>
> I was trying to understand GFac module and found different providers which
> use to execute jobs like BESProvider and GRAMProvider which extends
> AbstractProvider class like to know the use of those.
>
> And when search abouth EC2 instances there are three different kind of
> instances such as reserved instances ,spot instances. i would like to know
> out of those which kind is using in GFac-ec2 module.
>

As you might know, Amazon provides a java-sdk to communicate with it's AWS.
EC2Provdier uses it to interact with AMIs and instances. When testing out
or running a workflow, I usually used a general purpose instance as I did
not have any requirement for running a workflow in compute optimized
instance family. Having said that, you will be able to run a workflow in
any kind of instance that you want. You might have to tweak the UI a bit to
support this (ie. like adding an option to a drop down, etc and modifying
the backend a bit).

As for the sample workouts that you might be planning to run, you don't
need to run them in compute optimized instances as your workflow might not
be using the full potential of the resources. Keep in mind you are paying
for the instance type as well. As you reserve an instance with higher
compute power, you might be paying more (You might want to check the
pricing of instances before running sample workflows in compute optimized
instances).

So, the short answer to your question might be, you can use which ever
instance type you want as long as the AWS java-sdk api is exposing the
capability of interacting with those instances (which they do).

And is it need to configure to pass user-data scripts that runs the job
> each time launch a job. I would be grateful if somebody can clarify
> regarding these.
>

Let me tell you how the EC2 job submissions work in Airavata.

First you should have a base image which has your application installed.
This application can be a jar/shell script or other kind of a script. Try
and wrap your application with a shell script and make it available for
shell execution.

The EC2 Provider will get the user information (secret key, access key, AMI
id) and start up an instance from the base image (which has your
application installed.) Once the instance is running, the EC2 provider
establishes a SSH connection to the running instance and then runs the
shell script (which wraps your application) with the user given input
params. Provider will poll the standard out from the script and extract the
result out. Then it will be returned to the provider. Then this will passed
on to the workflow.

AFAICR I used J2ssh for the ssh connections. Since you are getting the
shell access to the instance, you might be able to run any script type that
you want but please remember that if you are using any script that is
dependent on the environment/version, you might run into some issues (eg.
Running a python 3.0 script on a Python 2.5 system). These things are also
minor issues and can be easily overcome. The reason why I told you this is
that, then you would not get side tracked because of any environment
issues.

I didn't check Airavata mailing list for a long time. Therefore, didn't see
this. :) Lahiru, pointed me to this thread. If you have any
questions/doubts, please don't hesitate to ask.

Thanks,
Heshan.

>
> Regards
> Nipun Udara
>
> *Undergraduate*
> *Department of Computer Science & Engineering*
> *University of Moratuwa*
> *Sri Lanka*
>



-- 
Regards,
Heshan Suriyaarachchi

http://heshans.blogspot.com/
http://www.linkedin.com/in/heshan