You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Sanjaya Medonsa <sa...@gmail.com> on 2013/02/17 19:22:25 UTC
Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Hi Dev Team,
As I have posted previously, I am working on a Apache Airavata + Apache
OODT integration as my MSc project. Following is one of the possible
integration to leverage Apache OODT file management capability into Apache
Airavata. Please review the proposal and let me know your feedback.
Proposal To Use Apache OODT products as input to Apache Airavata Workflows
and staging product files into node where execution happens
========================================================================================================
1. Introduce "Apache OODT Product" as a new GFacParameterType. New "Apache
OODT Product" input type can sepcify "Product ID" or "File Path to Product"
as an input to Apache Airavata workflows.
2. Introduce new PreExecuteChain to retrieve Apache OODT Products from File
Manager Server managed using Apache OODT.
1. Using Apache OODT File Manager componenet transfer Product from Server
to input directory of the application as configured using XBaya-GUI under
advanced configuration. (Here the assumption is that Products are accesible
through Apache OODT File Manager server)
2. Finally reset the input value to local file path. I think we can remove
the OODT Product parameter from invocation context and add new file
parameter with value set to 'local path of the transferred product'. I am
not quite sure what are the implications of changing input parameter type
during the execution.
Similar approach has been implemented for GridFTP and HTTP.
Best Regards,
Sanjaya
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Shabhaz,
Yes, I am about to do that, I just have one more task to finish this work,
once I am done, will remove the old code from trunk.
Regards
Lahiru
On Tue, Feb 19, 2013 at 6:28 AM, Shahbaz Memon <m....@fz-juelich.de>wrote:
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
>
> IMHO the unused or outdated code should be moved to any temporary
> branch, so that people checking-out trunk would get the actual/latest
> source of the architecture realization. In that sense it will not be a
> problem for devs if later the outdated code is reused.
>
> Cheers,
>
> Shahbaz
>
> On Mon, Feb 18, 2013 at 8:59 PM, Lahiru Gunathilake <gl...@gmail.com>
> wrote:
> > Hi Sanjaya,
> >
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
> >
> > Best place to start with is referring is from test classes
> > (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> > to GFacAPI class and see how to execution is flawing.
> >
> > if you have further questions please post on the list and more than happy
> > to help. I will be doing some documentation about the architecture, once
> I
> > am done, will post in to the list. And we will be having an architecture
> > review this week, so please watch the mailing list, if possible please
> try
> > to join us.
> >
> > Regards
> > Lahiru
> >
> > On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >wrote:
> >
> >> Thanks Suresh and Chris! It seems I am moving on the correct path. I
> have
> >> followed the email thread on improved GFac architecture. Though I am not
> >> entirely clear on the improved GFac architecture, proposed integration
> with
> >> OODT is primarily based on the GFac extension, PreExecustionChain, which
> >> has not been modified with the Architecture improvements (As per one of
> the
> >> replies from Lahiru, output extension is supported with new
> Architecture. I
> >> assume input extension is also supported).
> >>
> >> I have looked into provenance manager and related implementation. Still
> I
> >> am unclear how Airavata support provenance aware work flow processing.
> >>
> >> Best Regards,
> >> Sanjaya
> >>
> >> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> wrote:
> >>
> >> > Hi Sanjaya,
> >> >
> >> > This sounds very exciting. Both Airavata and OODT projects have good
> >> > synergies and have been long looking for volunteers who can bridge
> them
> >> > both. Please do not hesitate to ask any questions to either or both
> the
> >> dev
> >> > lists. The more engaged you are, you will find use cases and feedback
> >> which
> >> > should help your MSc project.
> >> >
> >> > Your plan sounds good. If you are following dev list, you may have
> >> > noticed, the GFac architecture has been improved to properly support
> this
> >> > kind of handler architecture.
> >> >
> >> > You may also want to look at Airavata Registry API which has
> organically
> >> > emerged with the need, but can greatly benefit for OODT's provenance
> >> > capabilities.
> >> >
> >> > Suresh
> >> >
> >> > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >
> >> > > +1, sounds like a great idea Sanjaya!
> >> > >
> >> > > I'm copying dev@oodt so they can be in the conversation too.
> >> > >
> >> > > Cheers,
> >> > > Chris
> >> > >
> >> > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> wrote:
> >> > >
> >> > >> Hi Dev Team,
> >> > >> As I have posted previously, I am working on a Apache Airavata +
> >> Apache
> >> > >> OODT integration as my MSc project. Following is one of the
> possible
> >> > >> integration to leverage Apache OODT file management capability into
> >> > Apache
> >> > >> Airavata. Please review the proposal and let me know your feedback.
> >> > >>
> >> > >> Proposal To Use Apache OODT products as input to Apache Airavata
> >> > Workflows
> >> > >> and staging product files into node where execution happens
> >> > >>
> >> >
> >>
> ==========================================================================
> >> > >> ==============================
> >> > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> >> > "Apache
> >> > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> >> > >> Product"
> >> > >> as an input to Apache Airavata workflows.
> >> > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> from
> >> > >> File
> >> > >> Manager Server managed using Apache OODT.
> >> > >> 1. Using Apache OODT File Manager componenet transfer Product from
> >> > Server
> >> > >> to input directory of the application as configured using XBaya-GUI
> >> > under
> >> > >> advanced configuration. (Here the assumption is that Products are
> >> > >> accesible
> >> > >> through Apache OODT File Manager server)
> >> > >> 2. Finally reset the input value to local file path. I think we can
> >> > remove
> >> > >> the OODT Product parameter from invocation context and add new file
> >> > >> parameter with value set to 'local path of the transferred
> product'. I
> >> > am
> >> > >> not quite sure what are the implications of changing input
> parameter
> >> > type
> >> > >> during the execution.
> >> > >>
> >> > >> Similar approach has been implemented for GridFTP and HTTP.
> >> > >>
> >> > >> Best Regards,
> >> > >> Sanjaya
> >> > >
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > System Analyst Programmer
> > PTI Lab
> > Indiana University
>
>
>
> ------------------------------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
> ------------------------------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------------------------
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Shabhaz,
Yes, I am about to do that, I just have one more task to finish this work,
once I am done, will remove the old code from trunk.
Regards
Lahiru
On Tue, Feb 19, 2013 at 6:28 AM, Shahbaz Memon <m....@fz-juelich.de>wrote:
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
>
> IMHO the unused or outdated code should be moved to any temporary
> branch, so that people checking-out trunk would get the actual/latest
> source of the architecture realization. In that sense it will not be a
> problem for devs if later the outdated code is reused.
>
> Cheers,
>
> Shahbaz
>
> On Mon, Feb 18, 2013 at 8:59 PM, Lahiru Gunathilake <gl...@gmail.com>
> wrote:
> > Hi Sanjaya,
> >
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
> >
> > Best place to start with is referring is from test classes
> > (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> > to GFacAPI class and see how to execution is flawing.
> >
> > if you have further questions please post on the list and more than happy
> > to help. I will be doing some documentation about the architecture, once
> I
> > am done, will post in to the list. And we will be having an architecture
> > review this week, so please watch the mailing list, if possible please
> try
> > to join us.
> >
> > Regards
> > Lahiru
> >
> > On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >wrote:
> >
> >> Thanks Suresh and Chris! It seems I am moving on the correct path. I
> have
> >> followed the email thread on improved GFac architecture. Though I am not
> >> entirely clear on the improved GFac architecture, proposed integration
> with
> >> OODT is primarily based on the GFac extension, PreExecustionChain, which
> >> has not been modified with the Architecture improvements (As per one of
> the
> >> replies from Lahiru, output extension is supported with new
> Architecture. I
> >> assume input extension is also supported).
> >>
> >> I have looked into provenance manager and related implementation. Still
> I
> >> am unclear how Airavata support provenance aware work flow processing.
> >>
> >> Best Regards,
> >> Sanjaya
> >>
> >> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> wrote:
> >>
> >> > Hi Sanjaya,
> >> >
> >> > This sounds very exciting. Both Airavata and OODT projects have good
> >> > synergies and have been long looking for volunteers who can bridge
> them
> >> > both. Please do not hesitate to ask any questions to either or both
> the
> >> dev
> >> > lists. The more engaged you are, you will find use cases and feedback
> >> which
> >> > should help your MSc project.
> >> >
> >> > Your plan sounds good. If you are following dev list, you may have
> >> > noticed, the GFac architecture has been improved to properly support
> this
> >> > kind of handler architecture.
> >> >
> >> > You may also want to look at Airavata Registry API which has
> organically
> >> > emerged with the need, but can greatly benefit for OODT's provenance
> >> > capabilities.
> >> >
> >> > Suresh
> >> >
> >> > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >
> >> > > +1, sounds like a great idea Sanjaya!
> >> > >
> >> > > I'm copying dev@oodt so they can be in the conversation too.
> >> > >
> >> > > Cheers,
> >> > > Chris
> >> > >
> >> > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> wrote:
> >> > >
> >> > >> Hi Dev Team,
> >> > >> As I have posted previously, I am working on a Apache Airavata +
> >> Apache
> >> > >> OODT integration as my MSc project. Following is one of the
> possible
> >> > >> integration to leverage Apache OODT file management capability into
> >> > Apache
> >> > >> Airavata. Please review the proposal and let me know your feedback.
> >> > >>
> >> > >> Proposal To Use Apache OODT products as input to Apache Airavata
> >> > Workflows
> >> > >> and staging product files into node where execution happens
> >> > >>
> >> >
> >>
> ==========================================================================
> >> > >> ==============================
> >> > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> >> > "Apache
> >> > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> >> > >> Product"
> >> > >> as an input to Apache Airavata workflows.
> >> > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> from
> >> > >> File
> >> > >> Manager Server managed using Apache OODT.
> >> > >> 1. Using Apache OODT File Manager componenet transfer Product from
> >> > Server
> >> > >> to input directory of the application as configured using XBaya-GUI
> >> > under
> >> > >> advanced configuration. (Here the assumption is that Products are
> >> > >> accesible
> >> > >> through Apache OODT File Manager server)
> >> > >> 2. Finally reset the input value to local file path. I think we can
> >> > remove
> >> > >> the OODT Product parameter from invocation context and add new file
> >> > >> parameter with value set to 'local path of the transferred
> product'. I
> >> > am
> >> > >> not quite sure what are the implications of changing input
> parameter
> >> > type
> >> > >> during the execution.
> >> > >>
> >> > >> Similar approach has been implemented for GridFTP and HTTP.
> >> > >>
> >> > >> Best Regards,
> >> > >> Sanjaya
> >> > >
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > System Analyst Programmer
> > PTI Lab
> > Indiana University
>
>
>
> ------------------------------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
> ------------------------------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------------------------
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Shahbaz Memon <m....@fz-juelich.de>.
> If you want to understand the new architecture by looking in to the code,
> please just refer the package org.apache.airavata.gfac, do not refer any of
> the classes in org.apache.airavata.core.gfac.
IMHO the unused or outdated code should be moved to any temporary
branch, so that people checking-out trunk would get the actual/latest
source of the architecture realization. In that sense it will not be a
problem for devs if later the outdated code is reused.
Cheers,
Shahbaz
On Mon, Feb 18, 2013 at 8:59 PM, Lahiru Gunathilake <gl...@gmail.com> wrote:
> Hi Sanjaya,
>
> If you want to understand the new architecture by looking in to the code,
> please just refer the package org.apache.airavata.gfac, do not refer any of
> the classes in org.apache.airavata.core.gfac.
>
> Best place to start with is referring is from test classes
> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> to GFacAPI class and see how to execution is flawing.
>
> if you have further questions please post on the list and more than happy
> to help. I will be doing some documentation about the architecture, once I
> am done, will post in to the list. And we will be having an architecture
> review this week, so please watch the mailing list, if possible please try
> to join us.
>
> Regards
> Lahiru
>
> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
>
>> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
>> followed the email thread on improved GFac architecture. Though I am not
>> entirely clear on the improved GFac architecture, proposed integration with
>> OODT is primarily based on the GFac extension, PreExecustionChain, which
>> has not been modified with the Architecture improvements (As per one of the
>> replies from Lahiru, output extension is supported with new Architecture. I
>> assume input extension is also supported).
>>
>> I have looked into provenance manager and related implementation. Still I
>> am unclear how Airavata support provenance aware work flow processing.
>>
>> Best Regards,
>> Sanjaya
>>
>> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>> > Hi Sanjaya,
>> >
>> > This sounds very exciting. Both Airavata and OODT projects have good
>> > synergies and have been long looking for volunteers who can bridge them
>> > both. Please do not hesitate to ask any questions to either or both the
>> dev
>> > lists. The more engaged you are, you will find use cases and feedback
>> which
>> > should help your MSc project.
>> >
>> > Your plan sounds good. If you are following dev list, you may have
>> > noticed, the GFac architecture has been improved to properly support this
>> > kind of handler architecture.
>> >
>> > You may also want to look at Airavata Registry API which has organically
>> > emerged with the need, but can greatly benefit for OODT's provenance
>> > capabilities.
>> >
>> > Suresh
>> >
>> > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > > +1, sounds like a great idea Sanjaya!
>> > >
>> > > I'm copying dev@oodt so they can be in the conversation too.
>> > >
>> > > Cheers,
>> > > Chris
>> > >
>> > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>> > >
>> > >> Hi Dev Team,
>> > >> As I have posted previously, I am working on a Apache Airavata +
>> Apache
>> > >> OODT integration as my MSc project. Following is one of the possible
>> > >> integration to leverage Apache OODT file management capability into
>> > Apache
>> > >> Airavata. Please review the proposal and let me know your feedback.
>> > >>
>> > >> Proposal To Use Apache OODT products as input to Apache Airavata
>> > Workflows
>> > >> and staging product files into node where execution happens
>> > >>
>> >
>> ==========================================================================
>> > >> ==============================
>> > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
>> > "Apache
>> > >> OODT Product" input type can sepcify "Product ID" or "File Path to
>> > >> Product"
>> > >> as an input to Apache Airavata workflows.
>> > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>> > >> File
>> > >> Manager Server managed using Apache OODT.
>> > >> 1. Using Apache OODT File Manager componenet transfer Product from
>> > Server
>> > >> to input directory of the application as configured using XBaya-GUI
>> > under
>> > >> advanced configuration. (Here the assumption is that Products are
>> > >> accesible
>> > >> through Apache OODT File Manager server)
>> > >> 2. Finally reset the input value to local file path. I think we can
>> > remove
>> > >> the OODT Product parameter from invocation context and add new file
>> > >> parameter with value set to 'local path of the transferred product'. I
>> > am
>> > >> not quite sure what are the implications of changing input parameter
>> > type
>> > >> during the execution.
>> > >>
>> > >> Similar approach has been implemented for GridFTP and HTTP.
>> > >>
>> > >> Best Regards,
>> > >> Sanjaya
>> > >
>> >
>> >
>>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris! As you suggested best way to reuse the PgeTaskInstance is to
sub-class, as it contains several protected methods which are very useful
for integrating Apache OODT with workflow execution.
Best Regards,
Sanjaya
On Fri, Feb 22, 2013 at 9:22 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> Great. You may simply want to extend (sub-class) PgeTaskInstance, and
> create one for Airavata?
>
> Cheers,
> Chris
>
> On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
> >plan is to reuse the same code. I have looked at PGETaskInstance where
> >most
> >of these pre/post task execution implementation resides. I believe
> >FileManageFileStager class should be able to reuse easily. First I'll
> >focus on the input fileStaging and then I am planning to focus on
> >ingesting
> >products and updating metadata as post execution task.
> >
> >Best Regards,
> >Sanjaya
> >
> >On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hi Sanjaya,
> >>
> >> I would seriously recommend looking at OODT CAS-PGE, which already does
> >> file staging, and connects to the file manager using queries. You could
> >> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
> >>of
> >> the existing FM to WM to RM and crawl infrastructure from OODT.
> >>
> >> Cheers,
> >> Chris
> >>
> >> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> >>
> >> >Thanks Lahiru! I have gone through the test classes and the classes in
> >> >package org.apache.airavata.gfac. It was really helpful to understand
> >>the
> >> >new architecture. I have listed down my approach based on new
> >>architecture
> >> >to use Apache OODT products as an input to Airavata.
> >> > 1. Introduce new Data Type to represent Apache OODT Product
> >>as an
> >> >DataType. Basically new DataType is added into the
> >>GFacParameterTypes.xsd.
> >> > 2. With new Architecture In Handlers and Out Handlers replaces
> >> >the
> >> >Pre/Post execution chains in old architecture. For the moment I am
> >> >focusing
> >> >on using Apache OODT product ID or file path as an input and stage the
> >> >file
> >> >(product) into host where actual execution happens. File staging
> >>requires
> >> >to retrieve product from a File Manager server to the host where
> >>execution
> >> >occurs. File staging can be implemented as an* In Handler* and needs
> >>to be
> >> >configured as a new item in the list of configured In Handlers.
> >> > 3. Handler should first verify the input parameter types
> >>listed
> >> >in
> >> >Service Description of the Application context of the
> >> >*JobExecutionContext*.
> >> >If input type matches the new parameter type, in handler stage file
> >>into
> >> >host machine using Apache OODT file manager component. Corresponding
> >>input
> >> >value can be retrieved from *In MessageContex*t. If a parameter type in
> >> >MessageContext matches the new input type, then corresponding value is
> >>the
> >> >id or file path to product managed by Apache OODT File Manager server.
> >> >
> >> >Best Regards,
> >> >Sanjaya
> >> >
> >> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
> >> ><gl...@gmail.com>wrote:
> >> >
> >> >> Hi Sanjaya,
> >> >>
> >> >> If you want to understand the new architecture by looking in to the
> >> >>code,
> >> >> please just refer the package org.apache.airavata.gfac, do not refer
> >> >>any of
> >> >> the classes in org.apache.airavata.core.gfac.
> >> >>
> >> >> Best place to start with is referring is from test classes
> >> >> (LocalProviderTest,GramProviderTest), from tehre you can start
> >>looking
> >> >>in
> >> >> to GFacAPI class and see how to execution is flawing.
> >> >>
> >> >> if you have further questions please post on the list and more than
> >> >>happy
> >> >> to help. I will be doing some documentation about the architecture,
> >> >>once I
> >> >> am done, will post in to the list. And we will be having an
> >>architecture
> >> >> review this week, so please watch the mailing list, if possible
> >>please
> >> >>try
> >> >> to join us.
> >> >>
> >> >> Regards
> >> >> Lahiru
> >> >>
> >> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
> >><sanjayamrt@gmail.com
> >> >> >wrote:
> >> >>
> >> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
> >>I
> >> >>have
> >> >> > followed the email thread on improved GFac architecture. Though I
> >>am
> >> >>not
> >> >> > entirely clear on the improved GFac architecture, proposed
> >>integration
> >> >> with
> >> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
> >> >>which
> >> >> > has not been modified with the Architecture improvements (As per
> >>one
> >> >>of
> >> >> the
> >> >> > replies from Lahiru, output extension is supported with new
> >> >> Architecture. I
> >> >> > assume input extension is also supported).
> >> >> >
> >> >> > I have looked into provenance manager and related implementation.
> >> >>Still I
> >> >> > am unclear how Airavata support provenance aware work flow
> >>processing.
> >> >> >
> >> >> > Best Regards,
> >> >> > Sanjaya
> >> >> >
> >> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> >> >>wrote:
> >> >> >
> >> >> > > Hi Sanjaya,
> >> >> > >
> >> >> > > This sounds very exciting. Both Airavata and OODT projects have
> >>good
> >> >> > > synergies and have been long looking for volunteers who can
> >>bridge
> >> >>them
> >> >> > > both. Please do not hesitate to ask any questions to either or
> >>both
> >> >>the
> >> >> > dev
> >> >> > > lists. The more engaged you are, you will find use cases and
> >> >>feedback
> >> >> > which
> >> >> > > should help your MSc project.
> >> >> > >
> >> >> > > Your plan sounds good. If you are following dev list, you may
> >>have
> >> >> > > noticed, the GFac architecture has been improved to properly
> >>support
> >> >> this
> >> >> > > kind of handler architecture.
> >> >> > >
> >> >> > > You may also want to look at Airavata Registry API which has
> >> >> organically
> >> >> > > emerged with the need, but can greatly benefit for OODT's
> >>provenance
> >> >> > > capabilities.
> >> >> > >
> >> >> > > Suresh
> >> >> > >
> >> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >> > >
> >> >> > > > +1, sounds like a great idea Sanjaya!
> >> >> > > >
> >> >> > > > I'm copying dev@oodt so they can be in the conversation too.
> >> >> > > >
> >> >> > > > Cheers,
> >> >> > > > Chris
> >> >> > > >
> >> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> >> >>wrote:
> >> >> > > >
> >> >> > > >> Hi Dev Team,
> >> >> > > >> As I have posted previously, I am working on a Apache
> >>Airavata +
> >> >> > Apache
> >> >> > > >> OODT integration as my MSc project. Following is one of the
> >> >>possible
> >> >> > > >> integration to leverage Apache OODT file management capability
> >> >>into
> >> >> > > Apache
> >> >> > > >> Airavata. Please review the proposal and let me know your
> >> >>feedback.
> >> >> > > >>
> >> >> > > >> Proposal To Use Apache OODT products as input to Apache
> >>Airavata
> >> >> > > Workflows
> >> >> > > >> and staging product files into node where execution happens
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
> >>>>=======================================================================
> >>>>==
> >> >>=
> >> >> > > >> ==============================
> >> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
> >> >>New
> >> >> > > "Apache
> >> >> > > >> OODT Product" input type can sepcify "Product ID" or "File
> >>Path
> >> >>to
> >> >> > > >> Product"
> >> >> > > >> as an input to Apache Airavata workflows.
> >> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT
> >>Products
> >> >> from
> >> >> > > >> File
> >> >> > > >> Manager Server managed using Apache OODT.
> >> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
> >> >>from
> >> >> > > Server
> >> >> > > >> to input directory of the application as configured using
> >> >>XBaya-GUI
> >> >> > > under
> >> >> > > >> advanced configuration. (Here the assumption is that Products
> >>are
> >> >> > > >> accesible
> >> >> > > >> through Apache OODT File Manager server)
> >> >> > > >> 2. Finally reset the input value to local file path. I think
> >>we
> >> >>can
> >> >> > > remove
> >> >> > > >> the OODT Product parameter from invocation context and add new
> >> >>file
> >> >> > > >> parameter with value set to 'local path of the transferred
> >> >> product'. I
> >> >> > > am
> >> >> > > >> not quite sure what are the implications of changing input
> >> >>parameter
> >> >> > > type
> >> >> > > >> during the execution.
> >> >> > > >>
> >> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
> >> >> > > >>
> >> >> > > >> Best Regards,
> >> >> > > >> Sanjaya
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> System Analyst Programmer
> >> >> PTI Lab
> >> >> Indiana University
> >> >>
> >>
> >>
>
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris! As you suggested best way to reuse the PgeTaskInstance is to
sub-class, as it contains several protected methods which are very useful
for integrating Apache OODT with workflow execution.
Best Regards,
Sanjaya
On Fri, Feb 22, 2013 at 9:22 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> Great. You may simply want to extend (sub-class) PgeTaskInstance, and
> create one for Airavata?
>
> Cheers,
> Chris
>
> On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
> >plan is to reuse the same code. I have looked at PGETaskInstance where
> >most
> >of these pre/post task execution implementation resides. I believe
> >FileManageFileStager class should be able to reuse easily. First I'll
> >focus on the input fileStaging and then I am planning to focus on
> >ingesting
> >products and updating metadata as post execution task.
> >
> >Best Regards,
> >Sanjaya
> >
> >On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hi Sanjaya,
> >>
> >> I would seriously recommend looking at OODT CAS-PGE, which already does
> >> file staging, and connects to the file manager using queries. You could
> >> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
> >>of
> >> the existing FM to WM to RM and crawl infrastructure from OODT.
> >>
> >> Cheers,
> >> Chris
> >>
> >> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> >>
> >> >Thanks Lahiru! I have gone through the test classes and the classes in
> >> >package org.apache.airavata.gfac. It was really helpful to understand
> >>the
> >> >new architecture. I have listed down my approach based on new
> >>architecture
> >> >to use Apache OODT products as an input to Airavata.
> >> > 1. Introduce new Data Type to represent Apache OODT Product
> >>as an
> >> >DataType. Basically new DataType is added into the
> >>GFacParameterTypes.xsd.
> >> > 2. With new Architecture In Handlers and Out Handlers replaces
> >> >the
> >> >Pre/Post execution chains in old architecture. For the moment I am
> >> >focusing
> >> >on using Apache OODT product ID or file path as an input and stage the
> >> >file
> >> >(product) into host where actual execution happens. File staging
> >>requires
> >> >to retrieve product from a File Manager server to the host where
> >>execution
> >> >occurs. File staging can be implemented as an* In Handler* and needs
> >>to be
> >> >configured as a new item in the list of configured In Handlers.
> >> > 3. Handler should first verify the input parameter types
> >>listed
> >> >in
> >> >Service Description of the Application context of the
> >> >*JobExecutionContext*.
> >> >If input type matches the new parameter type, in handler stage file
> >>into
> >> >host machine using Apache OODT file manager component. Corresponding
> >>input
> >> >value can be retrieved from *In MessageContex*t. If a parameter type in
> >> >MessageContext matches the new input type, then corresponding value is
> >>the
> >> >id or file path to product managed by Apache OODT File Manager server.
> >> >
> >> >Best Regards,
> >> >Sanjaya
> >> >
> >> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
> >> ><gl...@gmail.com>wrote:
> >> >
> >> >> Hi Sanjaya,
> >> >>
> >> >> If you want to understand the new architecture by looking in to the
> >> >>code,
> >> >> please just refer the package org.apache.airavata.gfac, do not refer
> >> >>any of
> >> >> the classes in org.apache.airavata.core.gfac.
> >> >>
> >> >> Best place to start with is referring is from test classes
> >> >> (LocalProviderTest,GramProviderTest), from tehre you can start
> >>looking
> >> >>in
> >> >> to GFacAPI class and see how to execution is flawing.
> >> >>
> >> >> if you have further questions please post on the list and more than
> >> >>happy
> >> >> to help. I will be doing some documentation about the architecture,
> >> >>once I
> >> >> am done, will post in to the list. And we will be having an
> >>architecture
> >> >> review this week, so please watch the mailing list, if possible
> >>please
> >> >>try
> >> >> to join us.
> >> >>
> >> >> Regards
> >> >> Lahiru
> >> >>
> >> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
> >><sanjayamrt@gmail.com
> >> >> >wrote:
> >> >>
> >> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
> >>I
> >> >>have
> >> >> > followed the email thread on improved GFac architecture. Though I
> >>am
> >> >>not
> >> >> > entirely clear on the improved GFac architecture, proposed
> >>integration
> >> >> with
> >> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
> >> >>which
> >> >> > has not been modified with the Architecture improvements (As per
> >>one
> >> >>of
> >> >> the
> >> >> > replies from Lahiru, output extension is supported with new
> >> >> Architecture. I
> >> >> > assume input extension is also supported).
> >> >> >
> >> >> > I have looked into provenance manager and related implementation.
> >> >>Still I
> >> >> > am unclear how Airavata support provenance aware work flow
> >>processing.
> >> >> >
> >> >> > Best Regards,
> >> >> > Sanjaya
> >> >> >
> >> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> >> >>wrote:
> >> >> >
> >> >> > > Hi Sanjaya,
> >> >> > >
> >> >> > > This sounds very exciting. Both Airavata and OODT projects have
> >>good
> >> >> > > synergies and have been long looking for volunteers who can
> >>bridge
> >> >>them
> >> >> > > both. Please do not hesitate to ask any questions to either or
> >>both
> >> >>the
> >> >> > dev
> >> >> > > lists. The more engaged you are, you will find use cases and
> >> >>feedback
> >> >> > which
> >> >> > > should help your MSc project.
> >> >> > >
> >> >> > > Your plan sounds good. If you are following dev list, you may
> >>have
> >> >> > > noticed, the GFac architecture has been improved to properly
> >>support
> >> >> this
> >> >> > > kind of handler architecture.
> >> >> > >
> >> >> > > You may also want to look at Airavata Registry API which has
> >> >> organically
> >> >> > > emerged with the need, but can greatly benefit for OODT's
> >>provenance
> >> >> > > capabilities.
> >> >> > >
> >> >> > > Suresh
> >> >> > >
> >> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >> > >
> >> >> > > > +1, sounds like a great idea Sanjaya!
> >> >> > > >
> >> >> > > > I'm copying dev@oodt so they can be in the conversation too.
> >> >> > > >
> >> >> > > > Cheers,
> >> >> > > > Chris
> >> >> > > >
> >> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> >> >>wrote:
> >> >> > > >
> >> >> > > >> Hi Dev Team,
> >> >> > > >> As I have posted previously, I am working on a Apache
> >>Airavata +
> >> >> > Apache
> >> >> > > >> OODT integration as my MSc project. Following is one of the
> >> >>possible
> >> >> > > >> integration to leverage Apache OODT file management capability
> >> >>into
> >> >> > > Apache
> >> >> > > >> Airavata. Please review the proposal and let me know your
> >> >>feedback.
> >> >> > > >>
> >> >> > > >> Proposal To Use Apache OODT products as input to Apache
> >>Airavata
> >> >> > > Workflows
> >> >> > > >> and staging product files into node where execution happens
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
> >>>>=======================================================================
> >>>>==
> >> >>=
> >> >> > > >> ==============================
> >> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
> >> >>New
> >> >> > > "Apache
> >> >> > > >> OODT Product" input type can sepcify "Product ID" or "File
> >>Path
> >> >>to
> >> >> > > >> Product"
> >> >> > > >> as an input to Apache Airavata workflows.
> >> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT
> >>Products
> >> >> from
> >> >> > > >> File
> >> >> > > >> Manager Server managed using Apache OODT.
> >> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
> >> >>from
> >> >> > > Server
> >> >> > > >> to input directory of the application as configured using
> >> >>XBaya-GUI
> >> >> > > under
> >> >> > > >> advanced configuration. (Here the assumption is that Products
> >>are
> >> >> > > >> accesible
> >> >> > > >> through Apache OODT File Manager server)
> >> >> > > >> 2. Finally reset the input value to local file path. I think
> >>we
> >> >>can
> >> >> > > remove
> >> >> > > >> the OODT Product parameter from invocation context and add new
> >> >>file
> >> >> > > >> parameter with value set to 'local path of the transferred
> >> >> product'. I
> >> >> > > am
> >> >> > > >> not quite sure what are the implications of changing input
> >> >>parameter
> >> >> > > type
> >> >> > > >> during the execution.
> >> >> > > >>
> >> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
> >> >> > > >>
> >> >> > > >> Best Regards,
> >> >> > > >> Sanjaya
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> System Analyst Programmer
> >> >> PTI Lab
> >> >> Indiana University
> >> >>
> >>
> >>
>
>
Re: Proposal To Use Apache OODT products as input to Apache
Airavata Workflows and staging product files into node where execution
happens
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
Great. You may simply want to extend (sub-class) PgeTaskInstance, and
create one for Airavata?
Cheers,
Chris
On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
>plan is to reuse the same code. I have looked at PGETaskInstance where
>most
>of these pre/post task execution implementation resides. I believe
>FileManageFileStager class should be able to reuse easily. First I'll
>focus on the input fileStaging and then I am planning to focus on
>ingesting
>products and updating metadata as post execution task.
>
>Best Regards,
>Sanjaya
>
>On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> I would seriously recommend looking at OODT CAS-PGE, which already does
>> file staging, and connects to the file manager using queries. You could
>> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
>>of
>> the existing FM to WM to RM and crawl infrastructure from OODT.
>>
>> Cheers,
>> Chris
>>
>> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>
>> >Thanks Lahiru! I have gone through the test classes and the classes in
>> >package org.apache.airavata.gfac. It was really helpful to understand
>>the
>> >new architecture. I have listed down my approach based on new
>>architecture
>> >to use Apache OODT products as an input to Airavata.
>> > 1. Introduce new Data Type to represent Apache OODT Product
>>as an
>> >DataType. Basically new DataType is added into the
>>GFacParameterTypes.xsd.
>> > 2. With new Architecture In Handlers and Out Handlers replaces
>> >the
>> >Pre/Post execution chains in old architecture. For the moment I am
>> >focusing
>> >on using Apache OODT product ID or file path as an input and stage the
>> >file
>> >(product) into host where actual execution happens. File staging
>>requires
>> >to retrieve product from a File Manager server to the host where
>>execution
>> >occurs. File staging can be implemented as an* In Handler* and needs
>>to be
>> >configured as a new item in the list of configured In Handlers.
>> > 3. Handler should first verify the input parameter types
>>listed
>> >in
>> >Service Description of the Application context of the
>> >*JobExecutionContext*.
>> >If input type matches the new parameter type, in handler stage file
>>into
>> >host machine using Apache OODT file manager component. Corresponding
>>input
>> >value can be retrieved from *In MessageContex*t. If a parameter type in
>> >MessageContext matches the new input type, then corresponding value is
>>the
>> >id or file path to product managed by Apache OODT File Manager server.
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
>> ><gl...@gmail.com>wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> If you want to understand the new architecture by looking in to the
>> >>code,
>> >> please just refer the package org.apache.airavata.gfac, do not refer
>> >>any of
>> >> the classes in org.apache.airavata.core.gfac.
>> >>
>> >> Best place to start with is referring is from test classes
>> >> (LocalProviderTest,GramProviderTest), from tehre you can start
>>looking
>> >>in
>> >> to GFacAPI class and see how to execution is flawing.
>> >>
>> >> if you have further questions please post on the list and more than
>> >>happy
>> >> to help. I will be doing some documentation about the architecture,
>> >>once I
>> >> am done, will post in to the list. And we will be having an
>>architecture
>> >> review this week, so please watch the mailing list, if possible
>>please
>> >>try
>> >> to join us.
>> >>
>> >> Regards
>> >> Lahiru
>> >>
>> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
>><sanjayamrt@gmail.com
>> >> >wrote:
>> >>
>> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
>>I
>> >>have
>> >> > followed the email thread on improved GFac architecture. Though I
>>am
>> >>not
>> >> > entirely clear on the improved GFac architecture, proposed
>>integration
>> >> with
>> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
>> >>which
>> >> > has not been modified with the Architecture improvements (As per
>>one
>> >>of
>> >> the
>> >> > replies from Lahiru, output extension is supported with new
>> >> Architecture. I
>> >> > assume input extension is also supported).
>> >> >
>> >> > I have looked into provenance manager and related implementation.
>> >>Still I
>> >> > am unclear how Airavata support provenance aware work flow
>>processing.
>> >> >
>> >> > Best Regards,
>> >> > Sanjaya
>> >> >
>> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
>> >>wrote:
>> >> >
>> >> > > Hi Sanjaya,
>> >> > >
>> >> > > This sounds very exciting. Both Airavata and OODT projects have
>>good
>> >> > > synergies and have been long looking for volunteers who can
>>bridge
>> >>them
>> >> > > both. Please do not hesitate to ask any questions to either or
>>both
>> >>the
>> >> > dev
>> >> > > lists. The more engaged you are, you will find use cases and
>> >>feedback
>> >> > which
>> >> > > should help your MSc project.
>> >> > >
>> >> > > Your plan sounds good. If you are following dev list, you may
>>have
>> >> > > noticed, the GFac architecture has been improved to properly
>>support
>> >> this
>> >> > > kind of handler architecture.
>> >> > >
>> >> > > You may also want to look at Airavata Registry API which has
>> >> organically
>> >> > > emerged with the need, but can greatly benefit for OODT's
>>provenance
>> >> > > capabilities.
>> >> > >
>> >> > > Suresh
>> >> > >
>> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> > >
>> >> > > > +1, sounds like a great idea Sanjaya!
>> >> > > >
>> >> > > > I'm copying dev@oodt so they can be in the conversation too.
>> >> > > >
>> >> > > > Cheers,
>> >> > > > Chris
>> >> > > >
>> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
>> >>wrote:
>> >> > > >
>> >> > > >> Hi Dev Team,
>> >> > > >> As I have posted previously, I am working on a Apache
>>Airavata +
>> >> > Apache
>> >> > > >> OODT integration as my MSc project. Following is one of the
>> >>possible
>> >> > > >> integration to leverage Apache OODT file management capability
>> >>into
>> >> > > Apache
>> >> > > >> Airavata. Please review the proposal and let me know your
>> >>feedback.
>> >> > > >>
>> >> > > >> Proposal To Use Apache OODT products as input to Apache
>>Airavata
>> >> > > Workflows
>> >> > > >> and staging product files into node where execution happens
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>
>>>>=======================================================================
>>>>==
>> >>=
>> >> > > >> ==============================
>> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
>> >>New
>> >> > > "Apache
>> >> > > >> OODT Product" input type can sepcify "Product ID" or "File
>>Path
>> >>to
>> >> > > >> Product"
>> >> > > >> as an input to Apache Airavata workflows.
>> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT
>>Products
>> >> from
>> >> > > >> File
>> >> > > >> Manager Server managed using Apache OODT.
>> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
>> >>from
>> >> > > Server
>> >> > > >> to input directory of the application as configured using
>> >>XBaya-GUI
>> >> > > under
>> >> > > >> advanced configuration. (Here the assumption is that Products
>>are
>> >> > > >> accesible
>> >> > > >> through Apache OODT File Manager server)
>> >> > > >> 2. Finally reset the input value to local file path. I think
>>we
>> >>can
>> >> > > remove
>> >> > > >> the OODT Product parameter from invocation context and add new
>> >>file
>> >> > > >> parameter with value set to 'local path of the transferred
>> >> product'. I
>> >> > > am
>> >> > > >> not quite sure what are the implications of changing input
>> >>parameter
>> >> > > type
>> >> > > >> during the execution.
>> >> > > >>
>> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
>> >> > > >>
>> >> > > >> Best Regards,
>> >> > > >> Sanjaya
>> >> > > >
>> >> > >
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> System Analyst Programmer
>> >> PTI Lab
>> >> Indiana University
>> >>
>>
>>
Re: Proposal To Use Apache OODT products as input to Apache
Airavata Workflows and staging product files into node where execution
happens
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
Great. You may simply want to extend (sub-class) PgeTaskInstance, and
create one for Airavata?
Cheers,
Chris
On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
>plan is to reuse the same code. I have looked at PGETaskInstance where
>most
>of these pre/post task execution implementation resides. I believe
>FileManageFileStager class should be able to reuse easily. First I'll
>focus on the input fileStaging and then I am planning to focus on
>ingesting
>products and updating metadata as post execution task.
>
>Best Regards,
>Sanjaya
>
>On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> I would seriously recommend looking at OODT CAS-PGE, which already does
>> file staging, and connects to the file manager using queries. You could
>> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
>>of
>> the existing FM to WM to RM and crawl infrastructure from OODT.
>>
>> Cheers,
>> Chris
>>
>> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>
>> >Thanks Lahiru! I have gone through the test classes and the classes in
>> >package org.apache.airavata.gfac. It was really helpful to understand
>>the
>> >new architecture. I have listed down my approach based on new
>>architecture
>> >to use Apache OODT products as an input to Airavata.
>> > 1. Introduce new Data Type to represent Apache OODT Product
>>as an
>> >DataType. Basically new DataType is added into the
>>GFacParameterTypes.xsd.
>> > 2. With new Architecture In Handlers and Out Handlers replaces
>> >the
>> >Pre/Post execution chains in old architecture. For the moment I am
>> >focusing
>> >on using Apache OODT product ID or file path as an input and stage the
>> >file
>> >(product) into host where actual execution happens. File staging
>>requires
>> >to retrieve product from a File Manager server to the host where
>>execution
>> >occurs. File staging can be implemented as an* In Handler* and needs
>>to be
>> >configured as a new item in the list of configured In Handlers.
>> > 3. Handler should first verify the input parameter types
>>listed
>> >in
>> >Service Description of the Application context of the
>> >*JobExecutionContext*.
>> >If input type matches the new parameter type, in handler stage file
>>into
>> >host machine using Apache OODT file manager component. Corresponding
>>input
>> >value can be retrieved from *In MessageContex*t. If a parameter type in
>> >MessageContext matches the new input type, then corresponding value is
>>the
>> >id or file path to product managed by Apache OODT File Manager server.
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
>> ><gl...@gmail.com>wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> If you want to understand the new architecture by looking in to the
>> >>code,
>> >> please just refer the package org.apache.airavata.gfac, do not refer
>> >>any of
>> >> the classes in org.apache.airavata.core.gfac.
>> >>
>> >> Best place to start with is referring is from test classes
>> >> (LocalProviderTest,GramProviderTest), from tehre you can start
>>looking
>> >>in
>> >> to GFacAPI class and see how to execution is flawing.
>> >>
>> >> if you have further questions please post on the list and more than
>> >>happy
>> >> to help. I will be doing some documentation about the architecture,
>> >>once I
>> >> am done, will post in to the list. And we will be having an
>>architecture
>> >> review this week, so please watch the mailing list, if possible
>>please
>> >>try
>> >> to join us.
>> >>
>> >> Regards
>> >> Lahiru
>> >>
>> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
>><sanjayamrt@gmail.com
>> >> >wrote:
>> >>
>> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
>>I
>> >>have
>> >> > followed the email thread on improved GFac architecture. Though I
>>am
>> >>not
>> >> > entirely clear on the improved GFac architecture, proposed
>>integration
>> >> with
>> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
>> >>which
>> >> > has not been modified with the Architecture improvements (As per
>>one
>> >>of
>> >> the
>> >> > replies from Lahiru, output extension is supported with new
>> >> Architecture. I
>> >> > assume input extension is also supported).
>> >> >
>> >> > I have looked into provenance manager and related implementation.
>> >>Still I
>> >> > am unclear how Airavata support provenance aware work flow
>>processing.
>> >> >
>> >> > Best Regards,
>> >> > Sanjaya
>> >> >
>> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
>> >>wrote:
>> >> >
>> >> > > Hi Sanjaya,
>> >> > >
>> >> > > This sounds very exciting. Both Airavata and OODT projects have
>>good
>> >> > > synergies and have been long looking for volunteers who can
>>bridge
>> >>them
>> >> > > both. Please do not hesitate to ask any questions to either or
>>both
>> >>the
>> >> > dev
>> >> > > lists. The more engaged you are, you will find use cases and
>> >>feedback
>> >> > which
>> >> > > should help your MSc project.
>> >> > >
>> >> > > Your plan sounds good. If you are following dev list, you may
>>have
>> >> > > noticed, the GFac architecture has been improved to properly
>>support
>> >> this
>> >> > > kind of handler architecture.
>> >> > >
>> >> > > You may also want to look at Airavata Registry API which has
>> >> organically
>> >> > > emerged with the need, but can greatly benefit for OODT's
>>provenance
>> >> > > capabilities.
>> >> > >
>> >> > > Suresh
>> >> > >
>> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> > >
>> >> > > > +1, sounds like a great idea Sanjaya!
>> >> > > >
>> >> > > > I'm copying dev@oodt so they can be in the conversation too.
>> >> > > >
>> >> > > > Cheers,
>> >> > > > Chris
>> >> > > >
>> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
>> >>wrote:
>> >> > > >
>> >> > > >> Hi Dev Team,
>> >> > > >> As I have posted previously, I am working on a Apache
>>Airavata +
>> >> > Apache
>> >> > > >> OODT integration as my MSc project. Following is one of the
>> >>possible
>> >> > > >> integration to leverage Apache OODT file management capability
>> >>into
>> >> > > Apache
>> >> > > >> Airavata. Please review the proposal and let me know your
>> >>feedback.
>> >> > > >>
>> >> > > >> Proposal To Use Apache OODT products as input to Apache
>>Airavata
>> >> > > Workflows
>> >> > > >> and staging product files into node where execution happens
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>
>>>>=======================================================================
>>>>==
>> >>=
>> >> > > >> ==============================
>> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
>> >>New
>> >> > > "Apache
>> >> > > >> OODT Product" input type can sepcify "Product ID" or "File
>>Path
>> >>to
>> >> > > >> Product"
>> >> > > >> as an input to Apache Airavata workflows.
>> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT
>>Products
>> >> from
>> >> > > >> File
>> >> > > >> Manager Server managed using Apache OODT.
>> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
>> >>from
>> >> > > Server
>> >> > > >> to input directory of the application as configured using
>> >>XBaya-GUI
>> >> > > under
>> >> > > >> advanced configuration. (Here the assumption is that Products
>>are
>> >> > > >> accesible
>> >> > > >> through Apache OODT File Manager server)
>> >> > > >> 2. Finally reset the input value to local file path. I think
>>we
>> >>can
>> >> > > remove
>> >> > > >> the OODT Product parameter from invocation context and add new
>> >>file
>> >> > > >> parameter with value set to 'local path of the transferred
>> >> product'. I
>> >> > > am
>> >> > > >> not quite sure what are the implications of changing input
>> >>parameter
>> >> > > type
>> >> > > >> during the execution.
>> >> > > >>
>> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
>> >> > > >>
>> >> > > >> Best Regards,
>> >> > > >> Sanjaya
>> >> > > >
>> >> > >
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> System Analyst Programmer
>> >> PTI Lab
>> >> Indiana University
>> >>
>>
>>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
plan is to reuse the same code. I have looked at PGETaskInstance where most
of these pre/post task execution implementation resides. I believe
FileManageFileStager class should be able to reuse easily. First I'll
focus on the input fileStaging and then I am planning to focus on ingesting
products and updating metadata as post execution task.
Best Regards,
Sanjaya
On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> I would seriously recommend looking at OODT CAS-PGE, which already does
> file staging, and connects to the file manager using queries. You could
> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
> the existing FM to WM to RM and crawl infrastructure from OODT.
>
> Cheers,
> Chris
>
> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Thanks Lahiru! I have gone through the test classes and the classes in
> >package org.apache.airavata.gfac. It was really helpful to understand the
> >new architecture. I have listed down my approach based on new architecture
> >to use Apache OODT products as an input to Airavata.
> > 1. Introduce new Data Type to represent Apache OODT Product as an
> >DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> > 2. With new Architecture In Handlers and Out Handlers replaces
> >the
> >Pre/Post execution chains in old architecture. For the moment I am
> >focusing
> >on using Apache OODT product ID or file path as an input and stage the
> >file
> >(product) into host where actual execution happens. File staging requires
> >to retrieve product from a File Manager server to the host where execution
> >occurs. File staging can be implemented as an* In Handler* and needs to be
> >configured as a new item in the list of configured In Handlers.
> > 3. Handler should first verify the input parameter types listed
> >in
> >Service Description of the Application context of the
> >*JobExecutionContext*.
> >If input type matches the new parameter type, in handler stage file into
> >host machine using Apache OODT file manager component. Corresponding input
> >value can be retrieved from *In MessageContex*t. If a parameter type in
> >MessageContext matches the new input type, then corresponding value is the
> >id or file path to product managed by Apache OODT File Manager server.
> >
> >Best Regards,
> >Sanjaya
> >
> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
> ><gl...@gmail.com>wrote:
> >
> >> Hi Sanjaya,
> >>
> >> If you want to understand the new architecture by looking in to the
> >>code,
> >> please just refer the package org.apache.airavata.gfac, do not refer
> >>any of
> >> the classes in org.apache.airavata.core.gfac.
> >>
> >> Best place to start with is referring is from test classes
> >> (LocalProviderTest,GramProviderTest), from tehre you can start looking
> >>in
> >> to GFacAPI class and see how to execution is flawing.
> >>
> >> if you have further questions please post on the list and more than
> >>happy
> >> to help. I will be doing some documentation about the architecture,
> >>once I
> >> am done, will post in to the list. And we will be having an architecture
> >> review this week, so please watch the mailing list, if possible please
> >>try
> >> to join us.
> >>
> >> Regards
> >> Lahiru
> >>
> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >> >wrote:
> >>
> >> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
> >>have
> >> > followed the email thread on improved GFac architecture. Though I am
> >>not
> >> > entirely clear on the improved GFac architecture, proposed integration
> >> with
> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
> >>which
> >> > has not been modified with the Architecture improvements (As per one
> >>of
> >> the
> >> > replies from Lahiru, output extension is supported with new
> >> Architecture. I
> >> > assume input extension is also supported).
> >> >
> >> > I have looked into provenance manager and related implementation.
> >>Still I
> >> > am unclear how Airavata support provenance aware work flow processing.
> >> >
> >> > Best Regards,
> >> > Sanjaya
> >> >
> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> >>wrote:
> >> >
> >> > > Hi Sanjaya,
> >> > >
> >> > > This sounds very exciting. Both Airavata and OODT projects have good
> >> > > synergies and have been long looking for volunteers who can bridge
> >>them
> >> > > both. Please do not hesitate to ask any questions to either or both
> >>the
> >> > dev
> >> > > lists. The more engaged you are, you will find use cases and
> >>feedback
> >> > which
> >> > > should help your MSc project.
> >> > >
> >> > > Your plan sounds good. If you are following dev list, you may have
> >> > > noticed, the GFac architecture has been improved to properly support
> >> this
> >> > > kind of handler architecture.
> >> > >
> >> > > You may also want to look at Airavata Registry API which has
> >> organically
> >> > > emerged with the need, but can greatly benefit for OODT's provenance
> >> > > capabilities.
> >> > >
> >> > > Suresh
> >> > >
> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> > >
> >> > > > +1, sounds like a great idea Sanjaya!
> >> > > >
> >> > > > I'm copying dev@oodt so they can be in the conversation too.
> >> > > >
> >> > > > Cheers,
> >> > > > Chris
> >> > > >
> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> >>wrote:
> >> > > >
> >> > > >> Hi Dev Team,
> >> > > >> As I have posted previously, I am working on a Apache Airavata +
> >> > Apache
> >> > > >> OODT integration as my MSc project. Following is one of the
> >>possible
> >> > > >> integration to leverage Apache OODT file management capability
> >>into
> >> > > Apache
> >> > > >> Airavata. Please review the proposal and let me know your
> >>feedback.
> >> > > >>
> >> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> >> > > Workflows
> >> > > >> and staging product files into node where execution happens
> >> > > >>
> >> > >
> >> >
> >>
> >>=========================================================================
> >>=
> >> > > >> ==============================
> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
> >>New
> >> > > "Apache
> >> > > >> OODT Product" input type can sepcify "Product ID" or "File Path
> >>to
> >> > > >> Product"
> >> > > >> as an input to Apache Airavata workflows.
> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> >> from
> >> > > >> File
> >> > > >> Manager Server managed using Apache OODT.
> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
> >>from
> >> > > Server
> >> > > >> to input directory of the application as configured using
> >>XBaya-GUI
> >> > > under
> >> > > >> advanced configuration. (Here the assumption is that Products are
> >> > > >> accesible
> >> > > >> through Apache OODT File Manager server)
> >> > > >> 2. Finally reset the input value to local file path. I think we
> >>can
> >> > > remove
> >> > > >> the OODT Product parameter from invocation context and add new
> >>file
> >> > > >> parameter with value set to 'local path of the transferred
> >> product'. I
> >> > > am
> >> > > >> not quite sure what are the implications of changing input
> >>parameter
> >> > > type
> >> > > >> during the execution.
> >> > > >>
> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
> >> > > >>
> >> > > >> Best Regards,
> >> > > >> Sanjaya
> >> > > >
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
> >>
>
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
plan is to reuse the same code. I have looked at PGETaskInstance where most
of these pre/post task execution implementation resides. I believe
FileManageFileStager class should be able to reuse easily. First I'll
focus on the input fileStaging and then I am planning to focus on ingesting
products and updating metadata as post execution task.
Best Regards,
Sanjaya
On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sanjaya,
>
> I would seriously recommend looking at OODT CAS-PGE, which already does
> file staging, and connects to the file manager using queries. You could
> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
> the existing FM to WM to RM and crawl infrastructure from OODT.
>
> Cheers,
> Chris
>
> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
> >Thanks Lahiru! I have gone through the test classes and the classes in
> >package org.apache.airavata.gfac. It was really helpful to understand the
> >new architecture. I have listed down my approach based on new architecture
> >to use Apache OODT products as an input to Airavata.
> > 1. Introduce new Data Type to represent Apache OODT Product as an
> >DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> > 2. With new Architecture In Handlers and Out Handlers replaces
> >the
> >Pre/Post execution chains in old architecture. For the moment I am
> >focusing
> >on using Apache OODT product ID or file path as an input and stage the
> >file
> >(product) into host where actual execution happens. File staging requires
> >to retrieve product from a File Manager server to the host where execution
> >occurs. File staging can be implemented as an* In Handler* and needs to be
> >configured as a new item in the list of configured In Handlers.
> > 3. Handler should first verify the input parameter types listed
> >in
> >Service Description of the Application context of the
> >*JobExecutionContext*.
> >If input type matches the new parameter type, in handler stage file into
> >host machine using Apache OODT file manager component. Corresponding input
> >value can be retrieved from *In MessageContex*t. If a parameter type in
> >MessageContext matches the new input type, then corresponding value is the
> >id or file path to product managed by Apache OODT File Manager server.
> >
> >Best Regards,
> >Sanjaya
> >
> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
> ><gl...@gmail.com>wrote:
> >
> >> Hi Sanjaya,
> >>
> >> If you want to understand the new architecture by looking in to the
> >>code,
> >> please just refer the package org.apache.airavata.gfac, do not refer
> >>any of
> >> the classes in org.apache.airavata.core.gfac.
> >>
> >> Best place to start with is referring is from test classes
> >> (LocalProviderTest,GramProviderTest), from tehre you can start looking
> >>in
> >> to GFacAPI class and see how to execution is flawing.
> >>
> >> if you have further questions please post on the list and more than
> >>happy
> >> to help. I will be doing some documentation about the architecture,
> >>once I
> >> am done, will post in to the list. And we will be having an architecture
> >> review this week, so please watch the mailing list, if possible please
> >>try
> >> to join us.
> >>
> >> Regards
> >> Lahiru
> >>
> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >> >wrote:
> >>
> >> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
> >>have
> >> > followed the email thread on improved GFac architecture. Though I am
> >>not
> >> > entirely clear on the improved GFac architecture, proposed integration
> >> with
> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
> >>which
> >> > has not been modified with the Architecture improvements (As per one
> >>of
> >> the
> >> > replies from Lahiru, output extension is supported with new
> >> Architecture. I
> >> > assume input extension is also supported).
> >> >
> >> > I have looked into provenance manager and related implementation.
> >>Still I
> >> > am unclear how Airavata support provenance aware work flow processing.
> >> >
> >> > Best Regards,
> >> > Sanjaya
> >> >
> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> >>wrote:
> >> >
> >> > > Hi Sanjaya,
> >> > >
> >> > > This sounds very exciting. Both Airavata and OODT projects have good
> >> > > synergies and have been long looking for volunteers who can bridge
> >>them
> >> > > both. Please do not hesitate to ask any questions to either or both
> >>the
> >> > dev
> >> > > lists. The more engaged you are, you will find use cases and
> >>feedback
> >> > which
> >> > > should help your MSc project.
> >> > >
> >> > > Your plan sounds good. If you are following dev list, you may have
> >> > > noticed, the GFac architecture has been improved to properly support
> >> this
> >> > > kind of handler architecture.
> >> > >
> >> > > You may also want to look at Airavata Registry API which has
> >> organically
> >> > > emerged with the need, but can greatly benefit for OODT's provenance
> >> > > capabilities.
> >> > >
> >> > > Suresh
> >> > >
> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> > >
> >> > > > +1, sounds like a great idea Sanjaya!
> >> > > >
> >> > > > I'm copying dev@oodt so they can be in the conversation too.
> >> > > >
> >> > > > Cheers,
> >> > > > Chris
> >> > > >
> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> >>wrote:
> >> > > >
> >> > > >> Hi Dev Team,
> >> > > >> As I have posted previously, I am working on a Apache Airavata +
> >> > Apache
> >> > > >> OODT integration as my MSc project. Following is one of the
> >>possible
> >> > > >> integration to leverage Apache OODT file management capability
> >>into
> >> > > Apache
> >> > > >> Airavata. Please review the proposal and let me know your
> >>feedback.
> >> > > >>
> >> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> >> > > Workflows
> >> > > >> and staging product files into node where execution happens
> >> > > >>
> >> > >
> >> >
> >>
> >>=========================================================================
> >>=
> >> > > >> ==============================
> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
> >>New
> >> > > "Apache
> >> > > >> OODT Product" input type can sepcify "Product ID" or "File Path
> >>to
> >> > > >> Product"
> >> > > >> as an input to Apache Airavata workflows.
> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> >> from
> >> > > >> File
> >> > > >> Manager Server managed using Apache OODT.
> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
> >>from
> >> > > Server
> >> > > >> to input directory of the application as configured using
> >>XBaya-GUI
> >> > > under
> >> > > >> advanced configuration. (Here the assumption is that Products are
> >> > > >> accesible
> >> > > >> through Apache OODT File Manager server)
> >> > > >> 2. Finally reset the input value to local file path. I think we
> >>can
> >> > > remove
> >> > > >> the OODT Product parameter from invocation context and add new
> >>file
> >> > > >> parameter with value set to 'local path of the transferred
> >> product'. I
> >> > > am
> >> > > >> not quite sure what are the implications of changing input
> >>parameter
> >> > > type
> >> > > >> during the execution.
> >> > > >>
> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
> >> > > >>
> >> > > >> Best Regards,
> >> > > >> Sanjaya
> >> > > >
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
> >>
>
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Posted by Suresh Marru <sm...@apache.org>.
On Feb 19, 2013, at 1:52 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Thanks Lahiru! I have gone through the test classes and the classes in
> package org.apache.airavata.gfac. It was really helpful to understand the
> new architecture. I have listed down my approach based on new architecture
> to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
> DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces the
> Pre/Post execution chains in old architecture. For the moment I am focusing
> on using Apache OODT product ID or file path as an input and stage the file
> (product) into host where actual execution happens. File staging requires
> to retrieve product from a File Manager server to the host where execution
> occurs. File staging can be implemented as an* In Handler* and needs to be
> configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed in
> Service Description of the Application context of the *JobExecutionContext*.
> If input type matches the new parameter type, in handler stage file into
> host machine using Apache OODT file manager component. Corresponding input
> value can be retrieved from *In MessageContex*t. If a parameter type in
> MessageContext matches the new input type, then corresponding value is the
> id or file path to product managed by Apache OODT File Manager server.
Hi Sanjaya,
+ 1 to this plan. You are right on!! great job in dissecting the code and identifying where you need to plug in. Since you are bridging both the projects, these kind of details will greatly help every one understand what you are doing and offer early help.
Suresh
> Best Regards,
> Sanjaya
>
> On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> Hi Sanjaya,
>>
>> If you want to understand the new architecture by looking in to the code,
>> please just refer the package org.apache.airavata.gfac, do not refer any of
>> the classes in org.apache.airavata.core.gfac.
>>
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
>> to GFacAPI class and see how to execution is flawing.
>>
>> if you have further questions please post on the list and more than happy
>> to help. I will be doing some documentation about the architecture, once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please try
>> to join us.
>>
>> Regards
>> Lahiru
>>
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
>>> wrote:
>>
>>> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
>>> followed the email thread on improved GFac architecture. Though I am not
>>> entirely clear on the improved GFac architecture, proposed integration
>> with
>>> OODT is primarily based on the GFac extension, PreExecustionChain, which
>>> has not been modified with the Architecture improvements (As per one of
>> the
>>> replies from Lahiru, output extension is supported with new
>> Architecture. I
>>> assume input extension is also supported).
>>>
>>> I have looked into provenance manager and related implementation. Still I
>>> am unclear how Airavata support provenance aware work flow processing.
>>>
>>> Best Regards,
>>> Sanjaya
>>>
>>> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Sanjaya,
>>>>
>>>> This sounds very exciting. Both Airavata and OODT projects have good
>>>> synergies and have been long looking for volunteers who can bridge them
>>>> both. Please do not hesitate to ask any questions to either or both the
>>> dev
>>>> lists. The more engaged you are, you will find use cases and feedback
>>> which
>>>> should help your MSc project.
>>>>
>>>> Your plan sounds good. If you are following dev list, you may have
>>>> noticed, the GFac architecture has been improved to properly support
>> this
>>>> kind of handler architecture.
>>>>
>>>> You may also want to look at Airavata Registry API which has
>> organically
>>>> emerged with the need, but can greatly benefit for OODT's provenance
>>>> capabilities.
>>>>
>>>> Suresh
>>>>
>>>> On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>>> +1, sounds like a great idea Sanjaya!
>>>>>
>>>>> I'm copying dev@oodt so they can be in the conversation too.
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>> On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>>>>
>>>>>> Hi Dev Team,
>>>>>> As I have posted previously, I am working on a Apache Airavata +
>>> Apache
>>>>>> OODT integration as my MSc project. Following is one of the possible
>>>>>> integration to leverage Apache OODT file management capability into
>>>> Apache
>>>>>> Airavata. Please review the proposal and let me know your feedback.
>>>>>>
>>>>>> Proposal To Use Apache OODT products as input to Apache Airavata
>>>> Workflows
>>>>>> and staging product files into node where execution happens
>>>>>>
>>>>
>>>
>> ==========================================================================
>>>>>> ==============================
>>>>>> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
>>>> "Apache
>>>>>> OODT Product" input type can sepcify "Product ID" or "File Path to
>>>>>> Product"
>>>>>> as an input to Apache Airavata workflows.
>>>>>> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
>> from
>>>>>> File
>>>>>> Manager Server managed using Apache OODT.
>>>>>> 1. Using Apache OODT File Manager componenet transfer Product from
>>>> Server
>>>>>> to input directory of the application as configured using XBaya-GUI
>>>> under
>>>>>> advanced configuration. (Here the assumption is that Products are
>>>>>> accesible
>>>>>> through Apache OODT File Manager server)
>>>>>> 2. Finally reset the input value to local file path. I think we can
>>>> remove
>>>>>> the OODT Product parameter from invocation context and add new file
>>>>>> parameter with value set to 'local path of the transferred
>> product'. I
>>>> am
>>>>>> not quite sure what are the implications of changing input parameter
>>>> type
>>>>>> during the execution.
>>>>>>
>>>>>> Similar approach has been implemented for GridFTP and HTTP.
>>>>>>
>>>>>> Best Regards,
>>>>>> Sanjaya
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Posted by Suresh Marru <sm...@apache.org>.
On Feb 19, 2013, at 1:52 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Thanks Lahiru! I have gone through the test classes and the classes in
> package org.apache.airavata.gfac. It was really helpful to understand the
> new architecture. I have listed down my approach based on new architecture
> to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
> DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces the
> Pre/Post execution chains in old architecture. For the moment I am focusing
> on using Apache OODT product ID or file path as an input and stage the file
> (product) into host where actual execution happens. File staging requires
> to retrieve product from a File Manager server to the host where execution
> occurs. File staging can be implemented as an* In Handler* and needs to be
> configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed in
> Service Description of the Application context of the *JobExecutionContext*.
> If input type matches the new parameter type, in handler stage file into
> host machine using Apache OODT file manager component. Corresponding input
> value can be retrieved from *In MessageContex*t. If a parameter type in
> MessageContext matches the new input type, then corresponding value is the
> id or file path to product managed by Apache OODT File Manager server.
Hi Sanjaya,
+ 1 to this plan. You are right on!! great job in dissecting the code and identifying where you need to plug in. Since you are bridging both the projects, these kind of details will greatly help every one understand what you are doing and offer early help.
Suresh
> Best Regards,
> Sanjaya
>
> On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> Hi Sanjaya,
>>
>> If you want to understand the new architecture by looking in to the code,
>> please just refer the package org.apache.airavata.gfac, do not refer any of
>> the classes in org.apache.airavata.core.gfac.
>>
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
>> to GFacAPI class and see how to execution is flawing.
>>
>> if you have further questions please post on the list and more than happy
>> to help. I will be doing some documentation about the architecture, once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please try
>> to join us.
>>
>> Regards
>> Lahiru
>>
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
>>> wrote:
>>
>>> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
>>> followed the email thread on improved GFac architecture. Though I am not
>>> entirely clear on the improved GFac architecture, proposed integration
>> with
>>> OODT is primarily based on the GFac extension, PreExecustionChain, which
>>> has not been modified with the Architecture improvements (As per one of
>> the
>>> replies from Lahiru, output extension is supported with new
>> Architecture. I
>>> assume input extension is also supported).
>>>
>>> I have looked into provenance manager and related implementation. Still I
>>> am unclear how Airavata support provenance aware work flow processing.
>>>
>>> Best Regards,
>>> Sanjaya
>>>
>>> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Sanjaya,
>>>>
>>>> This sounds very exciting. Both Airavata and OODT projects have good
>>>> synergies and have been long looking for volunteers who can bridge them
>>>> both. Please do not hesitate to ask any questions to either or both the
>>> dev
>>>> lists. The more engaged you are, you will find use cases and feedback
>>> which
>>>> should help your MSc project.
>>>>
>>>> Your plan sounds good. If you are following dev list, you may have
>>>> noticed, the GFac architecture has been improved to properly support
>> this
>>>> kind of handler architecture.
>>>>
>>>> You may also want to look at Airavata Registry API which has
>> organically
>>>> emerged with the need, but can greatly benefit for OODT's provenance
>>>> capabilities.
>>>>
>>>> Suresh
>>>>
>>>> On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>>> +1, sounds like a great idea Sanjaya!
>>>>>
>>>>> I'm copying dev@oodt so they can be in the conversation too.
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>> On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>>>>>
>>>>>> Hi Dev Team,
>>>>>> As I have posted previously, I am working on a Apache Airavata +
>>> Apache
>>>>>> OODT integration as my MSc project. Following is one of the possible
>>>>>> integration to leverage Apache OODT file management capability into
>>>> Apache
>>>>>> Airavata. Please review the proposal and let me know your feedback.
>>>>>>
>>>>>> Proposal To Use Apache OODT products as input to Apache Airavata
>>>> Workflows
>>>>>> and staging product files into node where execution happens
>>>>>>
>>>>
>>>
>> ==========================================================================
>>>>>> ==============================
>>>>>> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
>>>> "Apache
>>>>>> OODT Product" input type can sepcify "Product ID" or "File Path to
>>>>>> Product"
>>>>>> as an input to Apache Airavata workflows.
>>>>>> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
>> from
>>>>>> File
>>>>>> Manager Server managed using Apache OODT.
>>>>>> 1. Using Apache OODT File Manager componenet transfer Product from
>>>> Server
>>>>>> to input directory of the application as configured using XBaya-GUI
>>>> under
>>>>>> advanced configuration. (Here the assumption is that Products are
>>>>>> accesible
>>>>>> through Apache OODT File Manager server)
>>>>>> 2. Finally reset the input value to local file path. I think we can
>>>> remove
>>>>>> the OODT Product parameter from invocation context and add new file
>>>>>> parameter with value set to 'local path of the transferred
>> product'. I
>>>> am
>>>>>> not quite sure what are the implications of changing input parameter
>>>> type
>>>>>> during the execution.
>>>>>>
>>>>>> Similar approach has been implemented for GridFTP and HTTP.
>>>>>>
>>>>>> Best Regards,
>>>>>> Sanjaya
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Sanjaya,
This looks like a clean approach for me.
Regards
Lahiru
On Tue, Feb 19, 2013 at 1:52 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Lahiru! I have gone through the test classes and the classes in
> package org.apache.airavata.gfac. It was really helpful to understand the
> new architecture. I have listed down my approach based on new architecture
> to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
> DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces the
> Pre/Post execution chains in old architecture. For the moment I am focusing
> on using Apache OODT product ID or file path as an input and stage the file
> (product) into host where actual execution happens. File staging requires
> to retrieve product from a File Manager server to the host where execution
> occurs. File staging can be implemented as an* In Handler* and needs to be
> configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed in
> Service Description of the Application context of the
> *JobExecutionContext*.
> If input type matches the new parameter type, in handler stage file into
> host machine using Apache OODT file manager component. Corresponding input
> value can be retrieved from *In MessageContex*t. If a parameter type in
> MessageContext matches the new input type, then corresponding value is the
> id or file path to product managed by Apache OODT File Manager server.
>
> Best Regards,
> Sanjaya
>
> On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Sanjaya,
> >
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
> >
> > Best place to start with is referring is from test classes
> > (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> > to GFacAPI class and see how to execution is flawing.
> >
> > if you have further questions please post on the list and more than happy
> > to help. I will be doing some documentation about the architecture, once
> I
> > am done, will post in to the list. And we will be having an architecture
> > review this week, so please watch the mailing list, if possible please
> try
> > to join us.
> >
> > Regards
> > Lahiru
> >
> > On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> > >wrote:
> >
> > > Thanks Suresh and Chris! It seems I am moving on the correct path. I
> have
> > > followed the email thread on improved GFac architecture. Though I am
> not
> > > entirely clear on the improved GFac architecture, proposed integration
> > with
> > > OODT is primarily based on the GFac extension, PreExecustionChain,
> which
> > > has not been modified with the Architecture improvements (As per one of
> > the
> > > replies from Lahiru, output extension is supported with new
> > Architecture. I
> > > assume input extension is also supported).
> > >
> > > I have looked into provenance manager and related implementation.
> Still I
> > > am unclear how Airavata support provenance aware work flow processing.
> > >
> > > Best Regards,
> > > Sanjaya
> > >
> > > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> wrote:
> > >
> > > > Hi Sanjaya,
> > > >
> > > > This sounds very exciting. Both Airavata and OODT projects have good
> > > > synergies and have been long looking for volunteers who can bridge
> them
> > > > both. Please do not hesitate to ask any questions to either or both
> the
> > > dev
> > > > lists. The more engaged you are, you will find use cases and feedback
> > > which
> > > > should help your MSc project.
> > > >
> > > > Your plan sounds good. If you are following dev list, you may have
> > > > noticed, the GFac architecture has been improved to properly support
> > this
> > > > kind of handler architecture.
> > > >
> > > > You may also want to look at Airavata Registry API which has
> > organically
> > > > emerged with the need, but can greatly benefit for OODT's provenance
> > > > capabilities.
> > > >
> > > > Suresh
> > > >
> > > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >
> > > > > +1, sounds like a great idea Sanjaya!
> > > > >
> > > > > I'm copying dev@oodt so they can be in the conversation too.
> > > > >
> > > > > Cheers,
> > > > > Chris
> > > > >
> > > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> wrote:
> > > > >
> > > > >> Hi Dev Team,
> > > > >> As I have posted previously, I am working on a Apache Airavata +
> > > Apache
> > > > >> OODT integration as my MSc project. Following is one of the
> possible
> > > > >> integration to leverage Apache OODT file management capability
> into
> > > > Apache
> > > > >> Airavata. Please review the proposal and let me know your
> feedback.
> > > > >>
> > > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > > > Workflows
> > > > >> and staging product files into node where execution happens
> > > > >>
> > > >
> > >
> >
> ==========================================================================
> > > > >> ==============================
> > > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> > > > "Apache
> > > > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> > > > >> Product"
> > > > >> as an input to Apache Airavata workflows.
> > > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> > from
> > > > >> File
> > > > >> Manager Server managed using Apache OODT.
> > > > >> 1. Using Apache OODT File Manager componenet transfer Product from
> > > > Server
> > > > >> to input directory of the application as configured using
> XBaya-GUI
> > > > under
> > > > >> advanced configuration. (Here the assumption is that Products are
> > > > >> accesible
> > > > >> through Apache OODT File Manager server)
> > > > >> 2. Finally reset the input value to local file path. I think we
> can
> > > > remove
> > > > >> the OODT Product parameter from invocation context and add new
> file
> > > > >> parameter with value set to 'local path of the transferred
> > product'. I
> > > > am
> > > > >> not quite sure what are the implications of changing input
> parameter
> > > > type
> > > > >> during the execution.
> > > > >>
> > > > >> Similar approach has been implemented for GridFTP and HTTP.
> > > > >>
> > > > >> Best Regards,
> > > > >> Sanjaya
> > > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > System Analyst Programmer
> > PTI Lab
> > Indiana University
> >
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Sanjaya,
This looks like a clean approach for me.
Regards
Lahiru
On Tue, Feb 19, 2013 at 1:52 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Lahiru! I have gone through the test classes and the classes in
> package org.apache.airavata.gfac. It was really helpful to understand the
> new architecture. I have listed down my approach based on new architecture
> to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
> DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces the
> Pre/Post execution chains in old architecture. For the moment I am focusing
> on using Apache OODT product ID or file path as an input and stage the file
> (product) into host where actual execution happens. File staging requires
> to retrieve product from a File Manager server to the host where execution
> occurs. File staging can be implemented as an* In Handler* and needs to be
> configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed in
> Service Description of the Application context of the
> *JobExecutionContext*.
> If input type matches the new parameter type, in handler stage file into
> host machine using Apache OODT file manager component. Corresponding input
> value can be retrieved from *In MessageContex*t. If a parameter type in
> MessageContext matches the new input type, then corresponding value is the
> id or file path to product managed by Apache OODT File Manager server.
>
> Best Regards,
> Sanjaya
>
> On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Sanjaya,
> >
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
> >
> > Best place to start with is referring is from test classes
> > (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> > to GFacAPI class and see how to execution is flawing.
> >
> > if you have further questions please post on the list and more than happy
> > to help. I will be doing some documentation about the architecture, once
> I
> > am done, will post in to the list. And we will be having an architecture
> > review this week, so please watch the mailing list, if possible please
> try
> > to join us.
> >
> > Regards
> > Lahiru
> >
> > On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> > >wrote:
> >
> > > Thanks Suresh and Chris! It seems I am moving on the correct path. I
> have
> > > followed the email thread on improved GFac architecture. Though I am
> not
> > > entirely clear on the improved GFac architecture, proposed integration
> > with
> > > OODT is primarily based on the GFac extension, PreExecustionChain,
> which
> > > has not been modified with the Architecture improvements (As per one of
> > the
> > > replies from Lahiru, output extension is supported with new
> > Architecture. I
> > > assume input extension is also supported).
> > >
> > > I have looked into provenance manager and related implementation.
> Still I
> > > am unclear how Airavata support provenance aware work flow processing.
> > >
> > > Best Regards,
> > > Sanjaya
> > >
> > > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
> wrote:
> > >
> > > > Hi Sanjaya,
> > > >
> > > > This sounds very exciting. Both Airavata and OODT projects have good
> > > > synergies and have been long looking for volunteers who can bridge
> them
> > > > both. Please do not hesitate to ask any questions to either or both
> the
> > > dev
> > > > lists. The more engaged you are, you will find use cases and feedback
> > > which
> > > > should help your MSc project.
> > > >
> > > > Your plan sounds good. If you are following dev list, you may have
> > > > noticed, the GFac architecture has been improved to properly support
> > this
> > > > kind of handler architecture.
> > > >
> > > > You may also want to look at Airavata Registry API which has
> > organically
> > > > emerged with the need, but can greatly benefit for OODT's provenance
> > > > capabilities.
> > > >
> > > > Suresh
> > > >
> > > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >
> > > > > +1, sounds like a great idea Sanjaya!
> > > > >
> > > > > I'm copying dev@oodt so they can be in the conversation too.
> > > > >
> > > > > Cheers,
> > > > > Chris
> > > > >
> > > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
> wrote:
> > > > >
> > > > >> Hi Dev Team,
> > > > >> As I have posted previously, I am working on a Apache Airavata +
> > > Apache
> > > > >> OODT integration as my MSc project. Following is one of the
> possible
> > > > >> integration to leverage Apache OODT file management capability
> into
> > > > Apache
> > > > >> Airavata. Please review the proposal and let me know your
> feedback.
> > > > >>
> > > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > > > Workflows
> > > > >> and staging product files into node where execution happens
> > > > >>
> > > >
> > >
> >
> ==========================================================================
> > > > >> ==============================
> > > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> > > > "Apache
> > > > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> > > > >> Product"
> > > > >> as an input to Apache Airavata workflows.
> > > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> > from
> > > > >> File
> > > > >> Manager Server managed using Apache OODT.
> > > > >> 1. Using Apache OODT File Manager componenet transfer Product from
> > > > Server
> > > > >> to input directory of the application as configured using
> XBaya-GUI
> > > > under
> > > > >> advanced configuration. (Here the assumption is that Products are
> > > > >> accesible
> > > > >> through Apache OODT File Manager server)
> > > > >> 2. Finally reset the input value to local file path. I think we
> can
> > > > remove
> > > > >> the OODT Product parameter from invocation context and add new
> file
> > > > >> parameter with value set to 'local path of the transferred
> > product'. I
> > > > am
> > > > >> not quite sure what are the implications of changing input
> parameter
> > > > type
> > > > >> during the execution.
> > > > >>
> > > > >> Similar approach has been implemented for GridFTP and HTTP.
> > > > >>
> > > > >> Best Regards,
> > > > >> Sanjaya
> > > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > System Analyst Programmer
> > PTI Lab
> > Indiana University
> >
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Proposal To Use Apache OODT products as input to Apache
Airavata Workflows and staging product files into node where execution
happens
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
I would seriously recommend looking at OODT CAS-PGE, which already does
file staging, and connects to the file manager using queries. You could
wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
the existing FM to WM to RM and crawl infrastructure from OODT.
Cheers,
Chris
On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Lahiru! I have gone through the test classes and the classes in
>package org.apache.airavata.gfac. It was really helpful to understand the
>new architecture. I have listed down my approach based on new architecture
>to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
>DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces
>the
>Pre/Post execution chains in old architecture. For the moment I am
>focusing
>on using Apache OODT product ID or file path as an input and stage the
>file
>(product) into host where actual execution happens. File staging requires
>to retrieve product from a File Manager server to the host where execution
>occurs. File staging can be implemented as an* In Handler* and needs to be
>configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed
>in
>Service Description of the Application context of the
>*JobExecutionContext*.
>If input type matches the new parameter type, in handler stage file into
>host machine using Apache OODT file manager component. Corresponding input
>value can be retrieved from *In MessageContex*t. If a parameter type in
>MessageContext matches the new input type, then corresponding value is the
>id or file path to product managed by Apache OODT File Manager server.
>
>Best Regards,
>Sanjaya
>
>On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
><gl...@gmail.com>wrote:
>
>> Hi Sanjaya,
>>
>> If you want to understand the new architecture by looking in to the
>>code,
>> please just refer the package org.apache.airavata.gfac, do not refer
>>any of
>> the classes in org.apache.airavata.core.gfac.
>>
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking
>>in
>> to GFacAPI class and see how to execution is flawing.
>>
>> if you have further questions please post on the list and more than
>>happy
>> to help. I will be doing some documentation about the architecture,
>>once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please
>>try
>> to join us.
>>
>> Regards
>> Lahiru
>>
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
>> >wrote:
>>
>> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
>>have
>> > followed the email thread on improved GFac architecture. Though I am
>>not
>> > entirely clear on the improved GFac architecture, proposed integration
>> with
>> > OODT is primarily based on the GFac extension, PreExecustionChain,
>>which
>> > has not been modified with the Architecture improvements (As per one
>>of
>> the
>> > replies from Lahiru, output extension is supported with new
>> Architecture. I
>> > assume input extension is also supported).
>> >
>> > I have looked into provenance manager and related implementation.
>>Still I
>> > am unclear how Airavata support provenance aware work flow processing.
>> >
>> > Best Regards,
>> > Sanjaya
>> >
>> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
>>wrote:
>> >
>> > > Hi Sanjaya,
>> > >
>> > > This sounds very exciting. Both Airavata and OODT projects have good
>> > > synergies and have been long looking for volunteers who can bridge
>>them
>> > > both. Please do not hesitate to ask any questions to either or both
>>the
>> > dev
>> > > lists. The more engaged you are, you will find use cases and
>>feedback
>> > which
>> > > should help your MSc project.
>> > >
>> > > Your plan sounds good. If you are following dev list, you may have
>> > > noticed, the GFac architecture has been improved to properly support
>> this
>> > > kind of handler architecture.
>> > >
>> > > You may also want to look at Airavata Registry API which has
>> organically
>> > > emerged with the need, but can greatly benefit for OODT's provenance
>> > > capabilities.
>> > >
>> > > Suresh
>> > >
>> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> > > chris.a.mattmann@jpl.nasa.gov> wrote:
>> > >
>> > > > +1, sounds like a great idea Sanjaya!
>> > > >
>> > > > I'm copying dev@oodt so they can be in the conversation too.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
>>wrote:
>> > > >
>> > > >> Hi Dev Team,
>> > > >> As I have posted previously, I am working on a Apache Airavata +
>> > Apache
>> > > >> OODT integration as my MSc project. Following is one of the
>>possible
>> > > >> integration to leverage Apache OODT file management capability
>>into
>> > > Apache
>> > > >> Airavata. Please review the proposal and let me know your
>>feedback.
>> > > >>
>> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
>> > > Workflows
>> > > >> and staging product files into node where execution happens
>> > > >>
>> > >
>> >
>>
>>=========================================================================
>>=
>> > > >> ==============================
>> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
>>New
>> > > "Apache
>> > > >> OODT Product" input type can sepcify "Product ID" or "File Path
>>to
>> > > >> Product"
>> > > >> as an input to Apache Airavata workflows.
>> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
>> from
>> > > >> File
>> > > >> Manager Server managed using Apache OODT.
>> > > >> 1. Using Apache OODT File Manager componenet transfer Product
>>from
>> > > Server
>> > > >> to input directory of the application as configured using
>>XBaya-GUI
>> > > under
>> > > >> advanced configuration. (Here the assumption is that Products are
>> > > >> accesible
>> > > >> through Apache OODT File Manager server)
>> > > >> 2. Finally reset the input value to local file path. I think we
>>can
>> > > remove
>> > > >> the OODT Product parameter from invocation context and add new
>>file
>> > > >> parameter with value set to 'local path of the transferred
>> product'. I
>> > > am
>> > > >> not quite sure what are the implications of changing input
>>parameter
>> > > type
>> > > >> during the execution.
>> > > >>
>> > > >> Similar approach has been implemented for GridFTP and HTTP.
>> > > >>
>> > > >> Best Regards,
>> > > >> Sanjaya
>> > > >
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
Re: Proposal To Use Apache OODT products as input to Apache
Airavata Workflows and staging product files into node where execution
happens
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sanjaya,
I would seriously recommend looking at OODT CAS-PGE, which already does
file staging, and connects to the file manager using queries. You could
wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
the existing FM to WM to RM and crawl infrastructure from OODT.
Cheers,
Chris
On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Thanks Lahiru! I have gone through the test classes and the classes in
>package org.apache.airavata.gfac. It was really helpful to understand the
>new architecture. I have listed down my approach based on new architecture
>to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
>DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces
>the
>Pre/Post execution chains in old architecture. For the moment I am
>focusing
>on using Apache OODT product ID or file path as an input and stage the
>file
>(product) into host where actual execution happens. File staging requires
>to retrieve product from a File Manager server to the host where execution
>occurs. File staging can be implemented as an* In Handler* and needs to be
>configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed
>in
>Service Description of the Application context of the
>*JobExecutionContext*.
>If input type matches the new parameter type, in handler stage file into
>host machine using Apache OODT file manager component. Corresponding input
>value can be retrieved from *In MessageContex*t. If a parameter type in
>MessageContext matches the new input type, then corresponding value is the
>id or file path to product managed by Apache OODT File Manager server.
>
>Best Regards,
>Sanjaya
>
>On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
><gl...@gmail.com>wrote:
>
>> Hi Sanjaya,
>>
>> If you want to understand the new architecture by looking in to the
>>code,
>> please just refer the package org.apache.airavata.gfac, do not refer
>>any of
>> the classes in org.apache.airavata.core.gfac.
>>
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking
>>in
>> to GFacAPI class and see how to execution is flawing.
>>
>> if you have further questions please post on the list and more than
>>happy
>> to help. I will be doing some documentation about the architecture,
>>once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please
>>try
>> to join us.
>>
>> Regards
>> Lahiru
>>
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
>> >wrote:
>>
>> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
>>have
>> > followed the email thread on improved GFac architecture. Though I am
>>not
>> > entirely clear on the improved GFac architecture, proposed integration
>> with
>> > OODT is primarily based on the GFac extension, PreExecustionChain,
>>which
>> > has not been modified with the Architecture improvements (As per one
>>of
>> the
>> > replies from Lahiru, output extension is supported with new
>> Architecture. I
>> > assume input extension is also supported).
>> >
>> > I have looked into provenance manager and related implementation.
>>Still I
>> > am unclear how Airavata support provenance aware work flow processing.
>> >
>> > Best Regards,
>> > Sanjaya
>> >
>> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org>
>>wrote:
>> >
>> > > Hi Sanjaya,
>> > >
>> > > This sounds very exciting. Both Airavata and OODT projects have good
>> > > synergies and have been long looking for volunteers who can bridge
>>them
>> > > both. Please do not hesitate to ask any questions to either or both
>>the
>> > dev
>> > > lists. The more engaged you are, you will find use cases and
>>feedback
>> > which
>> > > should help your MSc project.
>> > >
>> > > Your plan sounds good. If you are following dev list, you may have
>> > > noticed, the GFac architecture has been improved to properly support
>> this
>> > > kind of handler architecture.
>> > >
>> > > You may also want to look at Airavata Registry API which has
>> organically
>> > > emerged with the need, but can greatly benefit for OODT's provenance
>> > > capabilities.
>> > >
>> > > Suresh
>> > >
>> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> > > chris.a.mattmann@jpl.nasa.gov> wrote:
>> > >
>> > > > +1, sounds like a great idea Sanjaya!
>> > > >
>> > > > I'm copying dev@oodt so they can be in the conversation too.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com>
>>wrote:
>> > > >
>> > > >> Hi Dev Team,
>> > > >> As I have posted previously, I am working on a Apache Airavata +
>> > Apache
>> > > >> OODT integration as my MSc project. Following is one of the
>>possible
>> > > >> integration to leverage Apache OODT file management capability
>>into
>> > > Apache
>> > > >> Airavata. Please review the proposal and let me know your
>>feedback.
>> > > >>
>> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
>> > > Workflows
>> > > >> and staging product files into node where execution happens
>> > > >>
>> > >
>> >
>>
>>=========================================================================
>>=
>> > > >> ==============================
>> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
>>New
>> > > "Apache
>> > > >> OODT Product" input type can sepcify "Product ID" or "File Path
>>to
>> > > >> Product"
>> > > >> as an input to Apache Airavata workflows.
>> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
>> from
>> > > >> File
>> > > >> Manager Server managed using Apache OODT.
>> > > >> 1. Using Apache OODT File Manager componenet transfer Product
>>from
>> > > Server
>> > > >> to input directory of the application as configured using
>>XBaya-GUI
>> > > under
>> > > >> advanced configuration. (Here the assumption is that Products are
>> > > >> accesible
>> > > >> through Apache OODT File Manager server)
>> > > >> 2. Finally reset the input value to local file path. I think we
>>can
>> > > remove
>> > > >> the OODT Product parameter from invocation context and add new
>>file
>> > > >> parameter with value set to 'local path of the transferred
>> product'. I
>> > > am
>> > > >> not quite sure what are the implications of changing input
>>parameter
>> > > type
>> > > >> during the execution.
>> > > >>
>> > > >> Similar approach has been implemented for GridFTP and HTTP.
>> > > >>
>> > > >> Best Regards,
>> > > >> Sanjaya
>> > > >
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Lahiru! I have gone through the test classes and the classes in
package org.apache.airavata.gfac. It was really helpful to understand the
new architecture. I have listed down my approach based on new architecture
to use Apache OODT products as an input to Airavata.
1. Introduce new Data Type to represent Apache OODT Product as an
DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
2. With new Architecture In Handlers and Out Handlers replaces the
Pre/Post execution chains in old architecture. For the moment I am focusing
on using Apache OODT product ID or file path as an input and stage the file
(product) into host where actual execution happens. File staging requires
to retrieve product from a File Manager server to the host where execution
occurs. File staging can be implemented as an* In Handler* and needs to be
configured as a new item in the list of configured In Handlers.
3. Handler should first verify the input parameter types listed in
Service Description of the Application context of the *JobExecutionContext*.
If input type matches the new parameter type, in handler stage file into
host machine using Apache OODT file manager component. Corresponding input
value can be retrieved from *In MessageContex*t. If a parameter type in
MessageContext matches the new input type, then corresponding value is the
id or file path to product managed by Apache OODT File Manager server.
Best Regards,
Sanjaya
On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Sanjaya,
>
> If you want to understand the new architecture by looking in to the code,
> please just refer the package org.apache.airavata.gfac, do not refer any of
> the classes in org.apache.airavata.core.gfac.
>
> Best place to start with is referring is from test classes
> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> to GFacAPI class and see how to execution is flawing.
>
> if you have further questions please post on the list and more than happy
> to help. I will be doing some documentation about the architecture, once I
> am done, will post in to the list. And we will be having an architecture
> review this week, so please watch the mailing list, if possible please try
> to join us.
>
> Regards
> Lahiru
>
> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >wrote:
>
> > Thanks Suresh and Chris! It seems I am moving on the correct path. I have
> > followed the email thread on improved GFac architecture. Though I am not
> > entirely clear on the improved GFac architecture, proposed integration
> with
> > OODT is primarily based on the GFac extension, PreExecustionChain, which
> > has not been modified with the Architecture improvements (As per one of
> the
> > replies from Lahiru, output extension is supported with new
> Architecture. I
> > assume input extension is also supported).
> >
> > I have looked into provenance manager and related implementation. Still I
> > am unclear how Airavata support provenance aware work flow processing.
> >
> > Best Regards,
> > Sanjaya
> >
> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
> >
> > > Hi Sanjaya,
> > >
> > > This sounds very exciting. Both Airavata and OODT projects have good
> > > synergies and have been long looking for volunteers who can bridge them
> > > both. Please do not hesitate to ask any questions to either or both the
> > dev
> > > lists. The more engaged you are, you will find use cases and feedback
> > which
> > > should help your MSc project.
> > >
> > > Your plan sounds good. If you are following dev list, you may have
> > > noticed, the GFac architecture has been improved to properly support
> this
> > > kind of handler architecture.
> > >
> > > You may also want to look at Airavata Registry API which has
> organically
> > > emerged with the need, but can greatly benefit for OODT's provenance
> > > capabilities.
> > >
> > > Suresh
> > >
> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > >
> > > > +1, sounds like a great idea Sanjaya!
> > > >
> > > > I'm copying dev@oodt so they can be in the conversation too.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> > > >
> > > >> Hi Dev Team,
> > > >> As I have posted previously, I am working on a Apache Airavata +
> > Apache
> > > >> OODT integration as my MSc project. Following is one of the possible
> > > >> integration to leverage Apache OODT file management capability into
> > > Apache
> > > >> Airavata. Please review the proposal and let me know your feedback.
> > > >>
> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > > Workflows
> > > >> and staging product files into node where execution happens
> > > >>
> > >
> >
> ==========================================================================
> > > >> ==============================
> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> > > "Apache
> > > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> > > >> Product"
> > > >> as an input to Apache Airavata workflows.
> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> from
> > > >> File
> > > >> Manager Server managed using Apache OODT.
> > > >> 1. Using Apache OODT File Manager componenet transfer Product from
> > > Server
> > > >> to input directory of the application as configured using XBaya-GUI
> > > under
> > > >> advanced configuration. (Here the assumption is that Products are
> > > >> accesible
> > > >> through Apache OODT File Manager server)
> > > >> 2. Finally reset the input value to local file path. I think we can
> > > remove
> > > >> the OODT Product parameter from invocation context and add new file
> > > >> parameter with value set to 'local path of the transferred
> product'. I
> > > am
> > > >> not quite sure what are the implications of changing input parameter
> > > type
> > > >> during the execution.
> > > >>
> > > >> Similar approach has been implemented for GridFTP and HTTP.
> > > >>
> > > >> Best Regards,
> > > >> Sanjaya
> > > >
> > >
> > >
> >
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Shahbaz Memon <m....@fz-juelich.de>.
> If you want to understand the new architecture by looking in to the code,
> please just refer the package org.apache.airavata.gfac, do not refer any of
> the classes in org.apache.airavata.core.gfac.
IMHO the unused or outdated code should be moved to any temporary
branch, so that people checking-out trunk would get the actual/latest
source of the architecture realization. In that sense it will not be a
problem for devs if later the outdated code is reused.
Cheers,
Shahbaz
On Mon, Feb 18, 2013 at 8:59 PM, Lahiru Gunathilake <gl...@gmail.com> wrote:
> Hi Sanjaya,
>
> If you want to understand the new architecture by looking in to the code,
> please just refer the package org.apache.airavata.gfac, do not refer any of
> the classes in org.apache.airavata.core.gfac.
>
> Best place to start with is referring is from test classes
> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> to GFacAPI class and see how to execution is flawing.
>
> if you have further questions please post on the list and more than happy
> to help. I will be doing some documentation about the architecture, once I
> am done, will post in to the list. And we will be having an architecture
> review this week, so please watch the mailing list, if possible please try
> to join us.
>
> Regards
> Lahiru
>
> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
>
>> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
>> followed the email thread on improved GFac architecture. Though I am not
>> entirely clear on the improved GFac architecture, proposed integration with
>> OODT is primarily based on the GFac extension, PreExecustionChain, which
>> has not been modified with the Architecture improvements (As per one of the
>> replies from Lahiru, output extension is supported with new Architecture. I
>> assume input extension is also supported).
>>
>> I have looked into provenance manager and related implementation. Still I
>> am unclear how Airavata support provenance aware work flow processing.
>>
>> Best Regards,
>> Sanjaya
>>
>> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>> > Hi Sanjaya,
>> >
>> > This sounds very exciting. Both Airavata and OODT projects have good
>> > synergies and have been long looking for volunteers who can bridge them
>> > both. Please do not hesitate to ask any questions to either or both the
>> dev
>> > lists. The more engaged you are, you will find use cases and feedback
>> which
>> > should help your MSc project.
>> >
>> > Your plan sounds good. If you are following dev list, you may have
>> > noticed, the GFac architecture has been improved to properly support this
>> > kind of handler architecture.
>> >
>> > You may also want to look at Airavata Registry API which has organically
>> > emerged with the need, but can greatly benefit for OODT's provenance
>> > capabilities.
>> >
>> > Suresh
>> >
>> > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > > +1, sounds like a great idea Sanjaya!
>> > >
>> > > I'm copying dev@oodt so they can be in the conversation too.
>> > >
>> > > Cheers,
>> > > Chris
>> > >
>> > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>> > >
>> > >> Hi Dev Team,
>> > >> As I have posted previously, I am working on a Apache Airavata +
>> Apache
>> > >> OODT integration as my MSc project. Following is one of the possible
>> > >> integration to leverage Apache OODT file management capability into
>> > Apache
>> > >> Airavata. Please review the proposal and let me know your feedback.
>> > >>
>> > >> Proposal To Use Apache OODT products as input to Apache Airavata
>> > Workflows
>> > >> and staging product files into node where execution happens
>> > >>
>> >
>> ==========================================================================
>> > >> ==============================
>> > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
>> > "Apache
>> > >> OODT Product" input type can sepcify "Product ID" or "File Path to
>> > >> Product"
>> > >> as an input to Apache Airavata workflows.
>> > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>> > >> File
>> > >> Manager Server managed using Apache OODT.
>> > >> 1. Using Apache OODT File Manager componenet transfer Product from
>> > Server
>> > >> to input directory of the application as configured using XBaya-GUI
>> > under
>> > >> advanced configuration. (Here the assumption is that Products are
>> > >> accesible
>> > >> through Apache OODT File Manager server)
>> > >> 2. Finally reset the input value to local file path. I think we can
>> > remove
>> > >> the OODT Product parameter from invocation context and add new file
>> > >> parameter with value set to 'local path of the transferred product'. I
>> > am
>> > >> not quite sure what are the implications of changing input parameter
>> > type
>> > >> during the execution.
>> > >>
>> > >> Similar approach has been implemented for GridFTP and HTTP.
>> > >>
>> > >> Best Regards,
>> > >> Sanjaya
>> > >
>> >
>> >
>>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Lahiru! I have gone through the test classes and the classes in
package org.apache.airavata.gfac. It was really helpful to understand the
new architecture. I have listed down my approach based on new architecture
to use Apache OODT products as an input to Airavata.
1. Introduce new Data Type to represent Apache OODT Product as an
DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
2. With new Architecture In Handlers and Out Handlers replaces the
Pre/Post execution chains in old architecture. For the moment I am focusing
on using Apache OODT product ID or file path as an input and stage the file
(product) into host where actual execution happens. File staging requires
to retrieve product from a File Manager server to the host where execution
occurs. File staging can be implemented as an* In Handler* and needs to be
configured as a new item in the list of configured In Handlers.
3. Handler should first verify the input parameter types listed in
Service Description of the Application context of the *JobExecutionContext*.
If input type matches the new parameter type, in handler stage file into
host machine using Apache OODT file manager component. Corresponding input
value can be retrieved from *In MessageContex*t. If a parameter type in
MessageContext matches the new input type, then corresponding value is the
id or file path to product managed by Apache OODT File Manager server.
Best Regards,
Sanjaya
On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Sanjaya,
>
> If you want to understand the new architecture by looking in to the code,
> please just refer the package org.apache.airavata.gfac, do not refer any of
> the classes in org.apache.airavata.core.gfac.
>
> Best place to start with is referring is from test classes
> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> to GFacAPI class and see how to execution is flawing.
>
> if you have further questions please post on the list and more than happy
> to help. I will be doing some documentation about the architecture, once I
> am done, will post in to the list. And we will be having an architecture
> review this week, so please watch the mailing list, if possible please try
> to join us.
>
> Regards
> Lahiru
>
> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >wrote:
>
> > Thanks Suresh and Chris! It seems I am moving on the correct path. I have
> > followed the email thread on improved GFac architecture. Though I am not
> > entirely clear on the improved GFac architecture, proposed integration
> with
> > OODT is primarily based on the GFac extension, PreExecustionChain, which
> > has not been modified with the Architecture improvements (As per one of
> the
> > replies from Lahiru, output extension is supported with new
> Architecture. I
> > assume input extension is also supported).
> >
> > I have looked into provenance manager and related implementation. Still I
> > am unclear how Airavata support provenance aware work flow processing.
> >
> > Best Regards,
> > Sanjaya
> >
> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
> >
> > > Hi Sanjaya,
> > >
> > > This sounds very exciting. Both Airavata and OODT projects have good
> > > synergies and have been long looking for volunteers who can bridge them
> > > both. Please do not hesitate to ask any questions to either or both the
> > dev
> > > lists. The more engaged you are, you will find use cases and feedback
> > which
> > > should help your MSc project.
> > >
> > > Your plan sounds good. If you are following dev list, you may have
> > > noticed, the GFac architecture has been improved to properly support
> this
> > > kind of handler architecture.
> > >
> > > You may also want to look at Airavata Registry API which has
> organically
> > > emerged with the need, but can greatly benefit for OODT's provenance
> > > capabilities.
> > >
> > > Suresh
> > >
> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > >
> > > > +1, sounds like a great idea Sanjaya!
> > > >
> > > > I'm copying dev@oodt so they can be in the conversation too.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> > > >
> > > >> Hi Dev Team,
> > > >> As I have posted previously, I am working on a Apache Airavata +
> > Apache
> > > >> OODT integration as my MSc project. Following is one of the possible
> > > >> integration to leverage Apache OODT file management capability into
> > > Apache
> > > >> Airavata. Please review the proposal and let me know your feedback.
> > > >>
> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > > Workflows
> > > >> and staging product files into node where execution happens
> > > >>
> > >
> >
> ==========================================================================
> > > >> ==============================
> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> > > "Apache
> > > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> > > >> Product"
> > > >> as an input to Apache Airavata workflows.
> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> from
> > > >> File
> > > >> Manager Server managed using Apache OODT.
> > > >> 1. Using Apache OODT File Manager componenet transfer Product from
> > > Server
> > > >> to input directory of the application as configured using XBaya-GUI
> > > under
> > > >> advanced configuration. (Here the assumption is that Products are
> > > >> accesible
> > > >> through Apache OODT File Manager server)
> > > >> 2. Finally reset the input value to local file path. I think we can
> > > remove
> > > >> the OODT Product parameter from invocation context and add new file
> > > >> parameter with value set to 'local path of the transferred
> product'. I
> > > am
> > > >> not quite sure what are the implications of changing input parameter
> > > type
> > > >> during the execution.
> > > >>
> > > >> Similar approach has been implemented for GridFTP and HTTP.
> > > >>
> > > >> Best Regards,
> > > >> Sanjaya
> > > >
> > >
> > >
> >
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Sanjaya,
If you want to understand the new architecture by looking in to the code,
please just refer the package org.apache.airavata.gfac, do not refer any of
the classes in org.apache.airavata.core.gfac.
Best place to start with is referring is from test classes
(LocalProviderTest,GramProviderTest), from tehre you can start looking in
to GFacAPI class and see how to execution is flawing.
if you have further questions please post on the list and more than happy
to help. I will be doing some documentation about the architecture, once I
am done, will post in to the list. And we will be having an architecture
review this week, so please watch the mailing list, if possible please try
to join us.
Regards
Lahiru
On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
> followed the email thread on improved GFac architecture. Though I am not
> entirely clear on the improved GFac architecture, proposed integration with
> OODT is primarily based on the GFac extension, PreExecustionChain, which
> has not been modified with the Architecture improvements (As per one of the
> replies from Lahiru, output extension is supported with new Architecture. I
> assume input extension is also supported).
>
> I have looked into provenance manager and related implementation. Still I
> am unclear how Airavata support provenance aware work flow processing.
>
> Best Regards,
> Sanjaya
>
> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
>
> > Hi Sanjaya,
> >
> > This sounds very exciting. Both Airavata and OODT projects have good
> > synergies and have been long looking for volunteers who can bridge them
> > both. Please do not hesitate to ask any questions to either or both the
> dev
> > lists. The more engaged you are, you will find use cases and feedback
> which
> > should help your MSc project.
> >
> > Your plan sounds good. If you are following dev list, you may have
> > noticed, the GFac architecture has been improved to properly support this
> > kind of handler architecture.
> >
> > You may also want to look at Airavata Registry API which has organically
> > emerged with the need, but can greatly benefit for OODT's provenance
> > capabilities.
> >
> > Suresh
> >
> > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > +1, sounds like a great idea Sanjaya!
> > >
> > > I'm copying dev@oodt so they can be in the conversation too.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> > >
> > >> Hi Dev Team,
> > >> As I have posted previously, I am working on a Apache Airavata +
> Apache
> > >> OODT integration as my MSc project. Following is one of the possible
> > >> integration to leverage Apache OODT file management capability into
> > Apache
> > >> Airavata. Please review the proposal and let me know your feedback.
> > >>
> > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > Workflows
> > >> and staging product files into node where execution happens
> > >>
> >
> ==========================================================================
> > >> ==============================
> > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> > "Apache
> > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> > >> Product"
> > >> as an input to Apache Airavata workflows.
> > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
> > >> File
> > >> Manager Server managed using Apache OODT.
> > >> 1. Using Apache OODT File Manager componenet transfer Product from
> > Server
> > >> to input directory of the application as configured using XBaya-GUI
> > under
> > >> advanced configuration. (Here the assumption is that Products are
> > >> accesible
> > >> through Apache OODT File Manager server)
> > >> 2. Finally reset the input value to local file path. I think we can
> > remove
> > >> the OODT Product parameter from invocation context and add new file
> > >> parameter with value set to 'local path of the transferred product'. I
> > am
> > >> not quite sure what are the implications of changing input parameter
> > type
> > >> during the execution.
> > >>
> > >> Similar approach has been implemented for GridFTP and HTTP.
> > >>
> > >> Best Regards,
> > >> Sanjaya
> > >
> >
> >
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Sanjaya,
If you want to understand the new architecture by looking in to the code,
please just refer the package org.apache.airavata.gfac, do not refer any of
the classes in org.apache.airavata.core.gfac.
Best place to start with is referring is from test classes
(LocalProviderTest,GramProviderTest), from tehre you can start looking in
to GFacAPI class and see how to execution is flawing.
if you have further questions please post on the list and more than happy
to help. I will be doing some documentation about the architecture, once I
am done, will post in to the list. And we will be having an architecture
review this week, so please watch the mailing list, if possible please try
to join us.
Regards
Lahiru
On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
> followed the email thread on improved GFac architecture. Though I am not
> entirely clear on the improved GFac architecture, proposed integration with
> OODT is primarily based on the GFac extension, PreExecustionChain, which
> has not been modified with the Architecture improvements (As per one of the
> replies from Lahiru, output extension is supported with new Architecture. I
> assume input extension is also supported).
>
> I have looked into provenance manager and related implementation. Still I
> am unclear how Airavata support provenance aware work flow processing.
>
> Best Regards,
> Sanjaya
>
> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
>
> > Hi Sanjaya,
> >
> > This sounds very exciting. Both Airavata and OODT projects have good
> > synergies and have been long looking for volunteers who can bridge them
> > both. Please do not hesitate to ask any questions to either or both the
> dev
> > lists. The more engaged you are, you will find use cases and feedback
> which
> > should help your MSc project.
> >
> > Your plan sounds good. If you are following dev list, you may have
> > noticed, the GFac architecture has been improved to properly support this
> > kind of handler architecture.
> >
> > You may also want to look at Airavata Registry API which has organically
> > emerged with the need, but can greatly benefit for OODT's provenance
> > capabilities.
> >
> > Suresh
> >
> > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > +1, sounds like a great idea Sanjaya!
> > >
> > > I'm copying dev@oodt so they can be in the conversation too.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> > >
> > >> Hi Dev Team,
> > >> As I have posted previously, I am working on a Apache Airavata +
> Apache
> > >> OODT integration as my MSc project. Following is one of the possible
> > >> integration to leverage Apache OODT file management capability into
> > Apache
> > >> Airavata. Please review the proposal and let me know your feedback.
> > >>
> > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > Workflows
> > >> and staging product files into node where execution happens
> > >>
> >
> ==========================================================================
> > >> ==============================
> > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> > "Apache
> > >> OODT Product" input type can sepcify "Product ID" or "File Path to
> > >> Product"
> > >> as an input to Apache Airavata workflows.
> > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
> > >> File
> > >> Manager Server managed using Apache OODT.
> > >> 1. Using Apache OODT File Manager componenet transfer Product from
> > Server
> > >> to input directory of the application as configured using XBaya-GUI
> > under
> > >> advanced configuration. (Here the assumption is that Products are
> > >> accesible
> > >> through Apache OODT File Manager server)
> > >> 2. Finally reset the input value to local file path. I think we can
> > remove
> > >> the OODT Product parameter from invocation context and add new file
> > >> parameter with value set to 'local path of the transferred product'. I
> > am
> > >> not quite sure what are the implications of changing input parameter
> > type
> > >> during the execution.
> > >>
> > >> Similar approach has been implemented for GridFTP and HTTP.
> > >>
> > >> Best Regards,
> > >> Sanjaya
> > >
> >
> >
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh and Chris! It seems I am moving on the correct path. I have
followed the email thread on improved GFac architecture. Though I am not
entirely clear on the improved GFac architecture, proposed integration with
OODT is primarily based on the GFac extension, PreExecustionChain, which
has not been modified with the Architecture improvements (As per one of the
replies from Lahiru, output extension is supported with new Architecture. I
assume input extension is also supported).
I have looked into provenance manager and related implementation. Still I
am unclear how Airavata support provenance aware work flow processing.
Best Regards,
Sanjaya
On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya,
>
> This sounds very exciting. Both Airavata and OODT projects have good
> synergies and have been long looking for volunteers who can bridge them
> both. Please do not hesitate to ask any questions to either or both the dev
> lists. The more engaged you are, you will find use cases and feedback which
> should help your MSc project.
>
> Your plan sounds good. If you are following dev list, you may have
> noticed, the GFac architecture has been improved to properly support this
> kind of handler architecture.
>
> You may also want to look at Airavata Registry API which has organically
> emerged with the need, but can greatly benefit for OODT's provenance
> capabilities.
>
> Suresh
>
> On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > +1, sounds like a great idea Sanjaya!
> >
> > I'm copying dev@oodt so they can be in the conversation too.
> >
> > Cheers,
> > Chris
> >
> > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> >
> >> Hi Dev Team,
> >> As I have posted previously, I am working on a Apache Airavata + Apache
> >> OODT integration as my MSc project. Following is one of the possible
> >> integration to leverage Apache OODT file management capability into
> Apache
> >> Airavata. Please review the proposal and let me know your feedback.
> >>
> >> Proposal To Use Apache OODT products as input to Apache Airavata
> Workflows
> >> and staging product files into node where execution happens
> >>
> ==========================================================================
> >> ==============================
> >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> "Apache
> >> OODT Product" input type can sepcify "Product ID" or "File Path to
> >> Product"
> >> as an input to Apache Airavata workflows.
> >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
> >> File
> >> Manager Server managed using Apache OODT.
> >> 1. Using Apache OODT File Manager componenet transfer Product from
> Server
> >> to input directory of the application as configured using XBaya-GUI
> under
> >> advanced configuration. (Here the assumption is that Products are
> >> accesible
> >> through Apache OODT File Manager server)
> >> 2. Finally reset the input value to local file path. I think we can
> remove
> >> the OODT Product parameter from invocation context and add new file
> >> parameter with value set to 'local path of the transferred product'. I
> am
> >> not quite sure what are the implications of changing input parameter
> type
> >> during the execution.
> >>
> >> Similar approach has been implemented for GridFTP and HTTP.
> >>
> >> Best Regards,
> >> Sanjaya
> >
>
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata
Workflows and staging product files into node where execution happens
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh and Chris! It seems I am moving on the correct path. I have
followed the email thread on improved GFac architecture. Though I am not
entirely clear on the improved GFac architecture, proposed integration with
OODT is primarily based on the GFac extension, PreExecustionChain, which
has not been modified with the Architecture improvements (As per one of the
replies from Lahiru, output extension is supported with new Architecture. I
assume input extension is also supported).
I have looked into provenance manager and related implementation. Still I
am unclear how Airavata support provenance aware work flow processing.
Best Regards,
Sanjaya
On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya,
>
> This sounds very exciting. Both Airavata and OODT projects have good
> synergies and have been long looking for volunteers who can bridge them
> both. Please do not hesitate to ask any questions to either or both the dev
> lists. The more engaged you are, you will find use cases and feedback which
> should help your MSc project.
>
> Your plan sounds good. If you are following dev list, you may have
> noticed, the GFac architecture has been improved to properly support this
> kind of handler architecture.
>
> You may also want to look at Airavata Registry API which has organically
> emerged with the need, but can greatly benefit for OODT's provenance
> capabilities.
>
> Suresh
>
> On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > +1, sounds like a great idea Sanjaya!
> >
> > I'm copying dev@oodt so they can be in the conversation too.
> >
> > Cheers,
> > Chris
> >
> > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
> >
> >> Hi Dev Team,
> >> As I have posted previously, I am working on a Apache Airavata + Apache
> >> OODT integration as my MSc project. Following is one of the possible
> >> integration to leverage Apache OODT file management capability into
> Apache
> >> Airavata. Please review the proposal and let me know your feedback.
> >>
> >> Proposal To Use Apache OODT products as input to Apache Airavata
> Workflows
> >> and staging product files into node where execution happens
> >>
> ==========================================================================
> >> ==============================
> >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
> "Apache
> >> OODT Product" input type can sepcify "Product ID" or "File Path to
> >> Product"
> >> as an input to Apache Airavata workflows.
> >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
> >> File
> >> Manager Server managed using Apache OODT.
> >> 1. Using Apache OODT File Manager componenet transfer Product from
> Server
> >> to input directory of the application as configured using XBaya-GUI
> under
> >> advanced configuration. (Here the assumption is that Products are
> >> accesible
> >> through Apache OODT File Manager server)
> >> 2. Finally reset the input value to local file path. I think we can
> remove
> >> the OODT Product parameter from invocation context and add new file
> >> parameter with value set to 'local path of the transferred product'. I
> am
> >> not quite sure what are the implications of changing input parameter
> type
> >> during the execution.
> >>
> >> Similar approach has been implemented for GridFTP and HTTP.
> >>
> >> Best Regards,
> >> Sanjaya
> >
>
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya,
This sounds very exciting. Both Airavata and OODT projects have good synergies and have been long looking for volunteers who can bridge them both. Please do not hesitate to ask any questions to either or both the dev lists. The more engaged you are, you will find use cases and feedback which should help your MSc project.
Your plan sounds good. If you are following dev list, you may have noticed, the GFac architecture has been improved to properly support this kind of handler architecture.
You may also want to look at Airavata Registry API which has organically emerged with the need, but can greatly benefit for OODT's provenance capabilities.
Suresh
On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
> +1, sounds like a great idea Sanjaya!
>
> I'm copying dev@oodt so they can be in the conversation too.
>
> Cheers,
> Chris
>
> On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
>> Hi Dev Team,
>> As I have posted previously, I am working on a Apache Airavata + Apache
>> OODT integration as my MSc project. Following is one of the possible
>> integration to leverage Apache OODT file management capability into Apache
>> Airavata. Please review the proposal and let me know your feedback.
>>
>> Proposal To Use Apache OODT products as input to Apache Airavata Workflows
>> and staging product files into node where execution happens
>> ==========================================================================
>> ==============================
>> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New "Apache
>> OODT Product" input type can sepcify "Product ID" or "File Path to
>> Product"
>> as an input to Apache Airavata workflows.
>> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>> File
>> Manager Server managed using Apache OODT.
>> 1. Using Apache OODT File Manager componenet transfer Product from Server
>> to input directory of the application as configured using XBaya-GUI under
>> advanced configuration. (Here the assumption is that Products are
>> accesible
>> through Apache OODT File Manager server)
>> 2. Finally reset the input value to local file path. I think we can remove
>> the OODT Product parameter from invocation context and add new file
>> parameter with value set to 'local path of the transferred product'. I am
>> not quite sure what are the implications of changing input parameter type
>> during the execution.
>>
>> Similar approach has been implemented for GridFTP and HTTP.
>>
>> Best Regards,
>> Sanjaya
>
Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya,
This sounds very exciting. Both Airavata and OODT projects have good synergies and have been long looking for volunteers who can bridge them both. Please do not hesitate to ask any questions to either or both the dev lists. The more engaged you are, you will find use cases and feedback which should help your MSc project.
Your plan sounds good. If you are following dev list, you may have noticed, the GFac architecture has been improved to properly support this kind of handler architecture.
You may also want to look at Airavata Registry API which has organically emerged with the need, but can greatly benefit for OODT's provenance capabilities.
Suresh
On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
> +1, sounds like a great idea Sanjaya!
>
> I'm copying dev@oodt so they can be in the conversation too.
>
> Cheers,
> Chris
>
> On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>
>> Hi Dev Team,
>> As I have posted previously, I am working on a Apache Airavata + Apache
>> OODT integration as my MSc project. Following is one of the possible
>> integration to leverage Apache OODT file management capability into Apache
>> Airavata. Please review the proposal and let me know your feedback.
>>
>> Proposal To Use Apache OODT products as input to Apache Airavata Workflows
>> and staging product files into node where execution happens
>> ==========================================================================
>> ==============================
>> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New "Apache
>> OODT Product" input type can sepcify "Product ID" or "File Path to
>> Product"
>> as an input to Apache Airavata workflows.
>> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>> File
>> Manager Server managed using Apache OODT.
>> 1. Using Apache OODT File Manager componenet transfer Product from Server
>> to input directory of the application as configured using XBaya-GUI under
>> advanced configuration. (Here the assumption is that Products are
>> accesible
>> through Apache OODT File Manager server)
>> 2. Finally reset the input value to local file path. I think we can remove
>> the OODT Product parameter from invocation context and add new file
>> parameter with value set to 'local path of the transferred product'. I am
>> not quite sure what are the implications of changing input parameter type
>> during the execution.
>>
>> Similar approach has been implemented for GridFTP and HTTP.
>>
>> Best Regards,
>> Sanjaya
>
Re: Proposal To Use Apache OODT products as input to Apache
Airavata Workflows and staging product files into node where execution
happens
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, sounds like a great idea Sanjaya!
I'm copying dev@oodt so they can be in the conversation too.
Cheers,
Chris
On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Hi Dev Team,
>As I have posted previously, I am working on a Apache Airavata + Apache
>OODT integration as my MSc project. Following is one of the possible
>integration to leverage Apache OODT file management capability into Apache
>Airavata. Please review the proposal and let me know your feedback.
>
>Proposal To Use Apache OODT products as input to Apache Airavata Workflows
>and staging product files into node where execution happens
>==========================================================================
>==============================
>1. Introduce "Apache OODT Product" as a new GFacParameterType. New "Apache
>OODT Product" input type can sepcify "Product ID" or "File Path to
>Product"
>as an input to Apache Airavata workflows.
>2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>File
>Manager Server managed using Apache OODT.
>1. Using Apache OODT File Manager componenet transfer Product from Server
>to input directory of the application as configured using XBaya-GUI under
>advanced configuration. (Here the assumption is that Products are
>accesible
>through Apache OODT File Manager server)
>2. Finally reset the input value to local file path. I think we can remove
>the OODT Product parameter from invocation context and add new file
>parameter with value set to 'local path of the transferred product'. I am
>not quite sure what are the implications of changing input parameter type
>during the execution.
>
>Similar approach has been implemented for GridFTP and HTTP.
>
>Best Regards,
>Sanjaya
Re: Proposal To Use Apache OODT products as input to Apache
Airavata Workflows and staging product files into node where execution
happens
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, sounds like a great idea Sanjaya!
I'm copying dev@oodt so they can be in the conversation too.
Cheers,
Chris
On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sa...@gmail.com> wrote:
>Hi Dev Team,
>As I have posted previously, I am working on a Apache Airavata + Apache
>OODT integration as my MSc project. Following is one of the possible
>integration to leverage Apache OODT file management capability into Apache
>Airavata. Please review the proposal and let me know your feedback.
>
>Proposal To Use Apache OODT products as input to Apache Airavata Workflows
>and staging product files into node where execution happens
>==========================================================================
>==============================
>1. Introduce "Apache OODT Product" as a new GFacParameterType. New "Apache
>OODT Product" input type can sepcify "Product ID" or "File Path to
>Product"
>as an input to Apache Airavata workflows.
>2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>File
>Manager Server managed using Apache OODT.
>1. Using Apache OODT File Manager componenet transfer Product from Server
>to input directory of the application as configured using XBaya-GUI under
>advanced configuration. (Here the assumption is that Products are
>accesible
>through Apache OODT File Manager server)
>2. Finally reset the input value to local file path. I think we can remove
>the OODT Product parameter from invocation context and add new file
>parameter with value set to 'local path of the transferred product'. I am
>not quite sure what are the implications of changing input parameter type
>during the execution.
>
>Similar approach has been implemented for GridFTP and HTTP.
>
>Best Regards,
>Sanjaya