Support high-throughput computing and development/integration strategies

[Yan Liu <ya...@cybergis.org>]

High-throughput computing (HTC) resources are available from national 
cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC 
resources are suitable for running serial or embarrassingly parallel 
user jobs. Unlike high-performance computing (HPC) resources, HTC 
environment is more distributed, loosely coupled, and managed 
individually by various resource contributors. For HTC users, a HTC 
environment presents a virtual resource pool with dynamically aggregated 
resources and can be accessed through a unified client software (e.g., 
OSG/VDT/Condor).

Integrating HTC capabilities in Airavata is important for users to 
access HTC resources seamlessly as they access other kinds of computing 
environments supported by Airavata. At first glance, the integration may 
be straightforward by adding middleware support (e.g., Condor or BOSCO) 
into Airavata. However, I am proposing a user-oriented approach to the 
integration in order to fully leverage HTC client software's capabilities.

An Airavata user does not care the underlying middleware when she/he 
composes a job, ideally. What the user cares is the computational 
capability provided by the underlying resources. A HTC environment, with 
the support from the Condor middleware, is desirable for running:
- large batch jobs
- parameter-sweeping jobs
- stochastic jobs with the same configuration but requiring a large 
number of repeated runs in order to obtain statistically confident results
- workflow jobs that can be represented as DAG (directed acyclic graph)

Therefore, instead of presenting a raw Condor interface to Airavata 
users, tailored interfaces to aforementioned user job types will be more 
useful. Technically, Condor submmit script syntax supports all of the 
described jobs through job macros and DAG support. If Airavata can 
bridge user job requirements and the composition of the technical Condor 
submission script, HTC resources can be more effectively represented for 
and used by Airavata community.

The development roadmap is upon Airavata team's design, I'm willing to 
contribute a disease mapping application for the testing and evaluation 
of the new components and capabilities developed in Airavata for this 
purpose.

Thanks,
Yan

Re: Support high-throughput computing and development/integration strategies

[Suresh Marru <sm...@apache.org>]

Hi Yan,

Welcome to the Airavata Community.

Thank for you these great thoughts. My initial reaction was how is it different than what Airavata currently provides, in the sense of adding a provider and then working at a higher level through workflow representation and specifying the interaction through API. But as I read through your email few times, I kind of understand the perspective. I think if we flip the way we think and focus in the features like these, it might lead to fundamental way we look at Airavata design. 

It will be great if you can review Airavata design from a user-oriented perspective and provide constructive feedback. We can have google hangout and explain the current design. I am sure it it will be a bit tough o understand from website/wiki  but complain please - constructive complaints are the number 1 contributions to a apache project). 

Cheers,
Suresh

On Mar 2, 2013, at 5:43 PM, Yan Liu <ya...@cybergis.org> wrote:

> High-throughput computing (HTC) resources are available from national cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC resources are suitable for running serial or embarrassingly parallel user jobs. Unlike high-performance computing (HPC) resources, HTC environment is more distributed, loosely coupled, and managed individually by various resource contributors. For HTC users, a HTC environment presents a virtual resource pool with dynamically aggregated resources and can be accessed through a unified client software (e.g., OSG/VDT/Condor).
> 
> Integrating HTC capabilities in Airavata is important for users to access HTC resources seamlessly as they access other kinds of computing environments supported by Airavata. At first glance, the integration may be straightforward by adding middleware support (e.g., Condor or BOSCO) into Airavata. However, I am proposing a user-oriented approach to the integration in order to fully leverage HTC client software's capabilities.
> 
> An Airavata user does not care the underlying middleware when she/he composes a job, ideally. What the user cares is the computational capability provided by the underlying resources. A HTC environment, with the support from the Condor middleware, is desirable for running:
> - large batch jobs
> - parameter-sweeping jobs
> - stochastic jobs with the same configuration but requiring a large number of repeated runs in order to obtain statistically confident results
> - workflow jobs that can be represented as DAG (directed acyclic graph)
> 
> Therefore, instead of presenting a raw Condor interface to Airavata users, tailored interfaces to aforementioned user job types will be more useful. Technically, Condor submmit script syntax supports all of the described jobs through job macros and DAG support. If Airavata can bridge user job requirements and the composition of the technical Condor submission script, HTC resources can be more effectively represented for and used by Airavata community.
> 
> The development roadmap is upon Airavata team's design, I'm willing to contribute a disease mapping application for the testing and evaluation of the new components and capabilities developed in Airavata for this purpose.
> 
> Thanks,
> Yan

Re: Support high-throughput computing and development/integration strategies

[Danushka Menikkumbura <da...@gmail.com>]

As I understand, Airavata should already be able to invoke jobs in an HTC
environment (GramProvider). If not it should be a little variation of the
current the current provider. We should also be able to have support for
different types of job in XBaya and provision them in the providers
(new/existing).

What I am not sure about is whether different HTC providers use a unified
API. For example API of Condor may be completely different from them of OSG
and in which case Airvata would end up having different extensions for
different HTC providers.

Monitoring (provenance) HTC jobs is another concern.

Thanks,
Danushka


On Sun, Mar 3, 2013 at 9:01 AM, Raminder Singh <ra...@gmail.com>wrote:

> Amila, Condor is a middleware to provision HTC resources. Following slides
> can provide basic information.
>
> http://research.cs.wisc.edu/htcondor/tutorials/condor-g-dagman-talk.ppt
>
> Yan, If i understood it right, collecting batch job and parameter-sweep
> information to construct a DAG is a good idea and its a good use-case for
> airavata.  Its going to be interesting for managing large number of jobs
> running in parallel.
>
> Thanks
> Raminder
>
> On Mar 2, 2013, at 9:56 PM, Amila Jayasekara wrote:
>
> > Hi Yan,
> >
> > This is a great idea. Please some inline comments below.
> >
> > Thanks
> > Amila
> >
> > On Sat, Mar 2, 2013 at 5:43 PM, Yan Liu <ya...@cybergis.org> wrote:
> >> High-throughput computing (HTC) resources are available from national
> >> cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC
> >> resources are suitable for running serial or embarrassingly parallel
> user
> >> jobs. Unlike high-performance computing (HPC) resources, HTC
> environment is
> >> more distributed, loosely coupled, and managed individually by various
> >> resource contributors. For HTC users, a HTC environment presents a
> virtual
> >> resource pool with dynamically aggregated resources and can be accessed
> >> through a unified client software (e.g., OSG/VDT/Condor).
> >>
> >> Integrating HTC capabilities in Airavata is important for users to
> access
> >> HTC resources seamlessly as they access other kinds of computing
> >> environments supported by Airavata. At first glance, the integration
> may be
> >> straightforward by adding middleware support (e.g., Condor or BOSCO)
> into
> >> Airavata. However, I am proposing a user-oriented approach to the
> >> integration in order to fully leverage HTC client software's
> capabilities.
> >>
> >> An Airavata user does not care the underlying middleware when she/he
> >> composes a job, ideally. What the user cares is the computational
> capability
> >> provided by the underlying resources. A HTC environment, with the
> support
> >> from the Condor middleware, is desirable for running:
> >> - large batch jobs
> >> - parameter-sweeping jobs
> >> - stochastic jobs with the same configuration but requiring a large
> number
> >> of repeated runs in order to obtain statistically confident results
> >> - workflow jobs that can be represented as DAG (directed acyclic graph)
> >
> > I am +1 for this idea. So how do you precisely define a HTC
> > environment ? (In other words what parameters does user needs to
> > specify when configuring a HTC environment ?)
> > I was kind of thinking along the same path. I agree with you that user
> > does not need to care about underlying middleware where his/her job
> > should run. So Airavata should be able to figure out an appropriate
> > place (machine) to run the job based on provided configurations.  I
> > think that configuration should be what you proposed as "HTC
> > environment".
> >
> > I am also curious to know whether you have thought about how
> > middleware related security (authentication, proxy certificates etc
> > ...) should be handle in this scenario. (My knowledge about Condor is
> > limited)
> >
> > Further I didnt quite understand how Condor middleware enables you to
> > run "workflow jobs that can be represented as DAG". Appreciate your
> > explanation on this.
> >
> > Also can Condor work with providers such as EC2 ?
> >
> >>
> >> Therefore, instead of presenting a raw Condor interface to Airavata
> users,
> >> tailored interfaces to aforementioned user job types will be more
> useful.
> >> Technically, Condor submmit script syntax supports all of the described
> jobs
> >> through job macros and DAG support. If Airavata can bridge user job
> >> requirements and the composition of the technical Condor submission
> script,
> >> HTC resources can be more effectively represented for and used by
> Airavata
> >> community.
> >>
> >> The development roadmap is upon Airavata team's design, I'm willing to
> >> contribute a disease mapping application for the testing and evaluation
> of
> >> the new components and capabilities developed in Airavata for this
> purpose.
> >>
> >> Thanks,
> >> Yan
>
>

Re: Support high-throughput computing and development/integration strategies

[Raminder Singh <ra...@gmail.com>]

Amila, Condor is a middleware to provision HTC resources. Following slides can provide basic information. 

http://research.cs.wisc.edu/htcondor/tutorials/condor-g-dagman-talk.ppt

Yan, If i understood it right, collecting batch job and parameter-sweep information to construct a DAG is a good idea and its a good use-case for airavata.  Its going to be interesting for managing large number of jobs running in parallel. 

Thanks
Raminder
 
On Mar 2, 2013, at 9:56 PM, Amila Jayasekara wrote:

> Hi Yan,
> 
> This is a great idea. Please some inline comments below.
> 
> Thanks
> Amila
> 
> On Sat, Mar 2, 2013 at 5:43 PM, Yan Liu <ya...@cybergis.org> wrote:
>> High-throughput computing (HTC) resources are available from national
>> cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC
>> resources are suitable for running serial or embarrassingly parallel user
>> jobs. Unlike high-performance computing (HPC) resources, HTC environment is
>> more distributed, loosely coupled, and managed individually by various
>> resource contributors. For HTC users, a HTC environment presents a virtual
>> resource pool with dynamically aggregated resources and can be accessed
>> through a unified client software (e.g., OSG/VDT/Condor).
>> 
>> Integrating HTC capabilities in Airavata is important for users to access
>> HTC resources seamlessly as they access other kinds of computing
>> environments supported by Airavata. At first glance, the integration may be
>> straightforward by adding middleware support (e.g., Condor or BOSCO) into
>> Airavata. However, I am proposing a user-oriented approach to the
>> integration in order to fully leverage HTC client software's capabilities.
>> 
>> An Airavata user does not care the underlying middleware when she/he
>> composes a job, ideally. What the user cares is the computational capability
>> provided by the underlying resources. A HTC environment, with the support
>> from the Condor middleware, is desirable for running:
>> - large batch jobs
>> - parameter-sweeping jobs
>> - stochastic jobs with the same configuration but requiring a large number
>> of repeated runs in order to obtain statistically confident results
>> - workflow jobs that can be represented as DAG (directed acyclic graph)
> 
> I am +1 for this idea. So how do you precisely define a HTC
> environment ? (In other words what parameters does user needs to
> specify when configuring a HTC environment ?)
> I was kind of thinking along the same path. I agree with you that user
> does not need to care about underlying middleware where his/her job
> should run. So Airavata should be able to figure out an appropriate
> place (machine) to run the job based on provided configurations.  I
> think that configuration should be what you proposed as "HTC
> environment".
> 
> I am also curious to know whether you have thought about how
> middleware related security (authentication, proxy certificates etc
> ...) should be handle in this scenario. (My knowledge about Condor is
> limited)
> 
> Further I didnt quite understand how Condor middleware enables you to
> run "workflow jobs that can be represented as DAG". Appreciate your
> explanation on this.
> 
> Also can Condor work with providers such as EC2 ?
> 
>> 
>> Therefore, instead of presenting a raw Condor interface to Airavata users,
>> tailored interfaces to aforementioned user job types will be more useful.
>> Technically, Condor submmit script syntax supports all of the described jobs
>> through job macros and DAG support. If Airavata can bridge user job
>> requirements and the composition of the technical Condor submission script,
>> HTC resources can be more effectively represented for and used by Airavata
>> community.
>> 
>> The development roadmap is upon Airavata team's design, I'm willing to
>> contribute a disease mapping application for the testing and evaluation of
>> the new components and capabilities developed in Airavata for this purpose.
>> 
>> Thanks,
>> Yan

Re: Support high-throughput computing and development/integration strategies

[Amila Jayasekara <th...@gmail.com>]

Hi Yan,

This is a great idea. Please some inline comments below.

Thanks
Amila

On Sat, Mar 2, 2013 at 5:43 PM, Yan Liu <ya...@cybergis.org> wrote:
> High-throughput computing (HTC) resources are available from national
> cyberinfrastructures such as the Open Science Grid and NSF XSEDE. HTC
> resources are suitable for running serial or embarrassingly parallel user
> jobs. Unlike high-performance computing (HPC) resources, HTC environment is
> more distributed, loosely coupled, and managed individually by various
> resource contributors. For HTC users, a HTC environment presents a virtual
> resource pool with dynamically aggregated resources and can be accessed
> through a unified client software (e.g., OSG/VDT/Condor).
>
> Integrating HTC capabilities in Airavata is important for users to access
> HTC resources seamlessly as they access other kinds of computing
> environments supported by Airavata. At first glance, the integration may be
> straightforward by adding middleware support (e.g., Condor or BOSCO) into
> Airavata. However, I am proposing a user-oriented approach to the
> integration in order to fully leverage HTC client software's capabilities.
>
> An Airavata user does not care the underlying middleware when she/he
> composes a job, ideally. What the user cares is the computational capability
> provided by the underlying resources. A HTC environment, with the support
> from the Condor middleware, is desirable for running:
> - large batch jobs
> - parameter-sweeping jobs
> - stochastic jobs with the same configuration but requiring a large number
> of repeated runs in order to obtain statistically confident results
> - workflow jobs that can be represented as DAG (directed acyclic graph)

I am +1 for this idea. So how do you precisely define a HTC
environment ? (In other words what parameters does user needs to
specify when configuring a HTC environment ?)
I was kind of thinking along the same path. I agree with you that user
does not need to care about underlying middleware where his/her job
should run. So Airavata should be able to figure out an appropriate
place (machine) to run the job based on provided configurations.  I
think that configuration should be what you proposed as "HTC
environment".

I am also curious to know whether you have thought about how
middleware related security (authentication, proxy certificates etc
...) should be handle in this scenario. (My knowledge about Condor is
limited)

Further I didnt quite understand how Condor middleware enables you to
run "workflow jobs that can be represented as DAG". Appreciate your
explanation on this.

Also can Condor work with providers such as EC2 ?

>
> Therefore, instead of presenting a raw Condor interface to Airavata users,
> tailored interfaces to aforementioned user job types will be more useful.
> Technically, Condor submmit script syntax supports all of the described jobs
> through job macros and DAG support. If Airavata can bridge user job
> requirements and the composition of the technical Condor submission script,
> HTC resources can be more effectively represented for and used by Airavata
> community.
>
> The development roadmap is upon Airavata team's design, I'm willing to
> contribute a disease mapping application for the testing and evaluation of
> the new components and capabilities developed in Airavata for this purpose.
>
> Thanks,
> Yan