You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Sanjaya Medonsa <sa...@gmail.com> on 2013/07/22 13:09:20 UTC
Povenance Aware Workflow Processing
Hi,
I am trying to find out ways to improve Airavata provenance aware workflow
processing based on rich metadata available in Apache OODT. As part of this
task, I am investigating how Apache Airavata is currently performing
provenance aware workflow processing. I have looked into the provenance
manager implementation, where I could see lot of data related to workflow
execution is saved in registry. But I could see very little of use of those
data. Could some one please explain the details of provenance aware
processing in Airavata and any future plans on this area?
Best Regards,
Sanjaya
Re: Povenance Aware Workflow Processing
Posted by Marlon Pierce <ma...@iu.edu>.
Hi Sanjaya--
Could you file a Jira ticket for this? We also welcome patches if you
want to contribute it.
Marlon
On 10/15/13 9:32 PM, Sanjaya Medonsa wrote:
> Hi Suresh,
> I have further looked into the code to understand, how provenance aware
> workflow processing has been implemented in current code base. As you have
> stated, it seems functionality is broken. In WorkflowInterpreter class,
> there is a method called readProvenance (only place where ProvenanceReader
> is used) . It is a private method, no longer a part of the actual
> execution. I think, it's better to go back to version where provenance
> aware workflow processing working as expected to get better understand on
> the expected flow. Then fix the issues in current version. Could you please
> let me know, possible version that this particular functionality worked as
> expected ?
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
>
>> Thanks Suresh! I'll do further investigations and let the team know my
>> findings.
>>
>> Best Regards,
>> Sanjaya
>>
>>
>> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Sanjaya,
>>>
>>> XBaya and Workflow Interpreter went through some changes which probably
>>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>>> AFAIK, the feature did not get deprecated it should be one of those cases
>>> which was not actively tested during releases and fell through the cracks.
>>> Lets see if any one gets motivated to debug and fix this issue. And as
>>> always, we welcome your patches. Sorry about pointing to a unverified
>>> feature.
>>>
>>> Suresh
>>>
>>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com>
>>> wrote:
>>>
>>>> Hi Suresh,
>>>> I have gone through the research paper in item 6 and try to match it to
>>> the current Airavata implementation. I can see that Airavata collects
>>> provenance data and stored them in a registry. I couldn't able to find out
>>> a place where these data has been actually used to improve the workflow
>>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>>> It seems this particular configuration is no longer used. Since provenance
>>> aware data is already available in registry, is it the responsibility of
>>> the application to use these data to improve workflow processing based on
>>> provenance aware data ?
>>>> Thanks
>>>> Sanjaya
>>>>
>>>>
>>>> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
>>> wrote:
>>>> Hey Sanjaya,
>>>>
>>>> You may want to check out the Traceable Product product type policy
>>>> [1] as well as OODT-139 [2] which includes the PCS services that serve
>>>> up the information and provenance.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> [1]
>>>>
>>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>>>> les/trace/
>>>> [2] http://issues.apache.org/jira/browse/OODT-139
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Sanjaya Medonsa <sa...@gmail.com>
>>>> Reply-To: <de...@oodt.apache.org>
>>>> Date: Friday, August 23, 2013 6:30 PM
>>>> To: Airavata Dev <de...@airavata.apache.org>
>>>> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>> Subject: Re: Povenance Aware Workflow Processing
>>>>
>>>>> Thanks Suresh! I am interested in Provenance aware workflow processing
>>>>> where OODT integration could help greatly with rich metadata available.
>>>>> I'll go through the item 6 and let you know my understanding.
>>>>>
>>>>> Sanjaya
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org>
>>> wrote:
>>>>>> Hi Sanjaya, Lewis,
>>>>>>
>>>>>> Appreciate your interest on this topic, this is a very good question
>>> and
>>>>>> sorry for the delay in responding, these days I feel I am always in
>>>>>> catch
>>>>>> up mode.
>>>>>>
>>>>>> Let me separate this discussion into four parts: Provenance
>>> Gathering,
>>>>>> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>>> executions.
>>>>>> Gathering: Airavata is instrumented with rich provenance centric
>>>>>> information model which captures each step and publishes to an event
>>>>>> bus.
>>>>>> This is encoded in a "workflow tracking" schema [1]. So the current
>>>>>> trunk
>>>>>> of Airavata publishes provenance rich information but not all is
>>>>>> guaranteed
>>>>>> to be captured in the registry.
>>>>>>
>>>>>> Analyzing: Significant work on provenance aspect related to Airavata
>>> is
>>>>>> a
>>>>>> by product of doctoral dissertation of Yogesh Simmhan, currently a
>>>>>> research
>>>>>> asst. professor at USC. His thesis [2] is a good source of reference
>>> on
>>>>>> this topic. Both the research and tools are in active development by
>>>>>> Prof.
>>>>>> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>>>>>> uses
>>>>>> the same workflow tracking schema and event bus, the tools should be
>>>>>> easily
>>>>>> integratable but did not have a motivating use case to do so.
>>>>>>
>>>>>> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>>>>>> version) and Karma framework which provides context to the gathered
>>>>>> provenance. Couple of papers on this topic [4], [5].
>>>>>>
>>>>>> Executions: This is an active topic within Airavata. Current release
>>> of
>>>>>> Airavata supports Provenance aware workflow executions and is
>>> discussed
>>>>>> in
>>>>>> [6]. I think the OODT and Airavata integration will greatly benefit
>>> this
>>>>>> aspect. Airavata Registry metadata cataloging is very primitive. And
>>>>>> Rich
>>>>>> data management capabilities of OODT should nicely enhance airavata
>>>>>> provenance aware executions.
>>>>>>
>>>>>> Please let me know which of the above aspects you are interested in
>>> and
>>>>>> we
>>>>>> can discuss further.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> [1] -
>>>>>>
>>>>>>
>>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>>>>>> tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>>>>>> [2] - http://gradworks.umi.com/3297094.pdf
>>>>>> [3] - http://d2i.indiana.edu/provenance_karma
>>>>>> [4] -
>>>>>>
>>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>>>>>> [5] -
>>>>>>
>>>>>>
>>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>>>>>> 38325269&CFTOKEN=73364358
>>>>>> [6] - http://dl.acm.org/citation.cfm?id=2359844
>>>>>>
>>>>>>
>>>>>> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sanjaya,
>>>>>>> I am seeing some traffic floating around RE: Airavata OODT
>>> integration
>>>>>> but I am not clued up.
>>>>>>> Can you point me to the code? The issues? The mailing list threads?
>>>>>>> I am also extremely interested in provenance tracking of
>>> (record???)
>>>>>> data which passes through OODT. I would like to expose this as some
>>>>>> stage
>>>>>> to people who access the data.
>>>>>>> Thank you so much.
>>>>>>>
>>>>>>> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>> I am trying to find out ways to improve Airavata provenance aware
>>>>>> workflow
>>>>>>>> processing based on rich metadata available in Apache OODT. As
>>> part
>>>>>> of
>>>>>> this
>>>>>>>> task, I am investigating how Apache Airavata is currently
>>> performing
>>>>>>>> provenance aware workflow processing. I have looked into the
>>>>>> provenance
>>>>>>>> manager implementation, where I could see lot of data related to
>>>>>> workflow
>>>>>>>> execution is saved in registry. But I could see very little of
>>> use
>>>>>> of
>>>>>> those
>>>>>>>> data. Could some one please explain the details of provenance
>>> aware
>>>>>>>> processing in Airavata and any future plans on this area?
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Sanjaya
>>>>>>>>
>>>>>>> --
>>>>>>> Lewis
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Suresh,
I have further looked into the code to understand, how provenance aware
workflow processing has been implemented in current code base. As you have
stated, it seems functionality is broken. In WorkflowInterpreter class,
there is a method called readProvenance (only place where ProvenanceReader
is used) . It is a private method, no longer a part of the actual
execution. I think, it's better to go back to version where provenance
aware workflow processing working as expected to get better understand on
the expected flow. Then fix the issues in current version. Could you please
let me know, possible version that this particular functionality worked as
expected ?
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Suresh! I'll do further investigations and let the team know my
> findings.
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Sanjaya,
>>
>> XBaya and Workflow Interpreter went through some changes which probably
>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>> AFAIK, the feature did not get deprecated it should be one of those cases
>> which was not actively tested during releases and fell through the cracks.
>> Lets see if any one gets motivated to debug and fix this issue. And as
>> always, we welcome your patches. Sorry about pointing to a unverified
>> feature.
>>
>> Suresh
>>
>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com>
>> wrote:
>>
>> > Hi Suresh,
>> > I have gone through the research paper in item 6 and try to match it to
>> the current Airavata implementation. I can see that Airavata collects
>> provenance data and stored them in a registry. I couldn't able to find out
>> a place where these data has been actually used to improve the workflow
>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>> It seems this particular configuration is no longer used. Since provenance
>> aware data is already available in registry, is it the responsibility of
>> the application to use these data to improve workflow processing based on
>> provenance aware data ?
>> >
>> > Thanks
>> > Sanjaya
>> >
>> >
>> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
>> wrote:
>> > Hey Sanjaya,
>> >
>> > You may want to check out the Traceable Product product type policy
>> > [1] as well as OODT-139 [2] which includes the PCS services that serve
>> > up the information and provenance.
>> >
>> > Cheers,
>> > Chris
>> >
>> > [1]
>> >
>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>> > les/trace/
>> > [2] http://issues.apache.org/jira/browse/OODT-139
>> >
>> >
>> > -----Original Message-----
>> > From: Sanjaya Medonsa <sa...@gmail.com>
>> > Reply-To: <de...@oodt.apache.org>
>> > Date: Friday, August 23, 2013 6:30 PM
>> > To: Airavata Dev <de...@airavata.apache.org>
>> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > Subject: Re: Povenance Aware Workflow Processing
>> >
>> > >Thanks Suresh! I am interested in Provenance aware workflow processing
>> > >where OODT integration could help greatly with rich metadata available.
>> > >I'll go through the item 6 and let you know my understanding.
>> > >
>> > >Sanjaya
>> > >
>> > >
>> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org>
>> wrote:
>> > >
>> > >> Hi Sanjaya, Lewis,
>> > >>
>> > >> Appreciate your interest on this topic, this is a very good question
>> and
>> > >> sorry for the delay in responding, these days I feel I am always in
>> > >>catch
>> > >> up mode.
>> > >>
>> > >> Let me separate this discussion into four parts: Provenance
>> Gathering,
>> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>> executions.
>> > >>
>> > >> Gathering: Airavata is instrumented with rich provenance centric
>> > >> information model which captures each step and publishes to an event
>> > >>bus.
>> > >> This is encoded in a "workflow tracking" schema [1]. So the current
>> > >>trunk
>> > >> of Airavata publishes provenance rich information but not all is
>> > >>guaranteed
>> > >> to be captured in the registry.
>> > >>
>> > >> Analyzing: Significant work on provenance aspect related to Airavata
>> is
>> > >>a
>> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
>> > >>research
>> > >> asst. professor at USC. His thesis [2] is a good source of reference
>> on
>> > >> this topic. Both the research and tools are in active development by
>> > >>Prof.
>> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>> > >>uses
>> > >> the same workflow tracking schema and event bus, the tools should be
>> > >>easily
>> > >> integratable but did not have a motivating use case to do so.
>> > >>
>> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> > >> version) and Karma framework which provides context to the gathered
>> > >> provenance. Couple of papers on this topic [4], [5].
>> > >>
>> > >> Executions: This is an active topic within Airavata. Current release
>> of
>> > >> Airavata supports Provenance aware workflow executions and is
>> discussed
>> > >>in
>> > >> [6]. I think the OODT and Airavata integration will greatly benefit
>> this
>> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
>> > >>Rich
>> > >> data management capabilities of OODT should nicely enhance airavata
>> > >> provenance aware executions.
>> > >>
>> > >> Please let me know which of the above aspects you are interested in
>> and
>> > >>we
>> > >> can discuss further.
>> > >>
>> > >> Suresh
>> > >>
>> > >> [1] -
>> > >>
>> > >>
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> > >> [2] - http://gradworks.umi.com/3297094.pdf
>> > >> [3] - http://d2i.indiana.edu/provenance_karma
>> > >> [4] -
>> > >>
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> > >> [5] -
>> > >>
>> > >>
>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>> > >>38325269&CFTOKEN=73364358
>> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
>> > >>
>> > >>
>> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> > >> lewis.mcgibbney@gmail.com> wrote:
>> > >>
>> > >> > Hi Sanjaya,
>> > >> > I am seeing some traffic floating around RE: Airavata OODT
>> integration
>> > >> but I am not clued up.
>> > >> > Can you point me to the code? The issues? The mailing list threads?
>> > >> > I am also extremely interested in provenance tracking of
>> (record???)
>> > >> data which passes through OODT. I would like to expose this as some
>> > >>stage
>> > >> to people who access the data.
>> > >> > Thank you so much.
>> > >> >
>> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>> > >>wrote:
>> > >> > > Hi,
>> > >> > > I am trying to find out ways to improve Airavata provenance aware
>> > >> workflow
>> > >> > > processing based on rich metadata available in Apache OODT. As
>> part
>> > >>of
>> > >> this
>> > >> > > task, I am investigating how Apache Airavata is currently
>> performing
>> > >> > > provenance aware workflow processing. I have looked into the
>> > >>provenance
>> > >> > > manager implementation, where I could see lot of data related to
>> > >> workflow
>> > >> > > execution is saved in registry. But I could see very little of
>> use
>> > >>of
>> > >> those
>> > >> > > data. Could some one please explain the details of provenance
>> aware
>> > >> > > processing in Airavata and any future plans on this area?
>> > >> > >
>> > >> > > Best Regards,
>> > >> > > Sanjaya
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Lewis
>> > >> >
>> > >>
>> > >>
>> >
>> >
>> >
>>
>>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Suresh,
I have further looked into the code to understand, how provenance aware
workflow processing has been implemented in current code base. As you have
stated, it seems functionality is broken. In WorkflowInterpreter class,
there is a method called readProvenance (only place where ProvenanceReader
is used) . It is a private method, no longer a part of the actual
execution. I think, it's better to go back to version where provenance
aware workflow processing working as expected to get better understand on
the expected flow. Then fix the issues in current version. Could you please
let me know, possible version that this particular functionality worked as
expected ?
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Suresh! I'll do further investigations and let the team know my
> findings.
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Sanjaya,
>>
>> XBaya and Workflow Interpreter went through some changes which probably
>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>> AFAIK, the feature did not get deprecated it should be one of those cases
>> which was not actively tested during releases and fell through the cracks.
>> Lets see if any one gets motivated to debug and fix this issue. And as
>> always, we welcome your patches. Sorry about pointing to a unverified
>> feature.
>>
>> Suresh
>>
>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com>
>> wrote:
>>
>> > Hi Suresh,
>> > I have gone through the research paper in item 6 and try to match it to
>> the current Airavata implementation. I can see that Airavata collects
>> provenance data and stored them in a registry. I couldn't able to find out
>> a place where these data has been actually used to improve the workflow
>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>> It seems this particular configuration is no longer used. Since provenance
>> aware data is already available in registry, is it the responsibility of
>> the application to use these data to improve workflow processing based on
>> provenance aware data ?
>> >
>> > Thanks
>> > Sanjaya
>> >
>> >
>> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
>> wrote:
>> > Hey Sanjaya,
>> >
>> > You may want to check out the Traceable Product product type policy
>> > [1] as well as OODT-139 [2] which includes the PCS services that serve
>> > up the information and provenance.
>> >
>> > Cheers,
>> > Chris
>> >
>> > [1]
>> >
>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>> > les/trace/
>> > [2] http://issues.apache.org/jira/browse/OODT-139
>> >
>> >
>> > -----Original Message-----
>> > From: Sanjaya Medonsa <sa...@gmail.com>
>> > Reply-To: <de...@oodt.apache.org>
>> > Date: Friday, August 23, 2013 6:30 PM
>> > To: Airavata Dev <de...@airavata.apache.org>
>> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > Subject: Re: Povenance Aware Workflow Processing
>> >
>> > >Thanks Suresh! I am interested in Provenance aware workflow processing
>> > >where OODT integration could help greatly with rich metadata available.
>> > >I'll go through the item 6 and let you know my understanding.
>> > >
>> > >Sanjaya
>> > >
>> > >
>> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org>
>> wrote:
>> > >
>> > >> Hi Sanjaya, Lewis,
>> > >>
>> > >> Appreciate your interest on this topic, this is a very good question
>> and
>> > >> sorry for the delay in responding, these days I feel I am always in
>> > >>catch
>> > >> up mode.
>> > >>
>> > >> Let me separate this discussion into four parts: Provenance
>> Gathering,
>> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>> executions.
>> > >>
>> > >> Gathering: Airavata is instrumented with rich provenance centric
>> > >> information model which captures each step and publishes to an event
>> > >>bus.
>> > >> This is encoded in a "workflow tracking" schema [1]. So the current
>> > >>trunk
>> > >> of Airavata publishes provenance rich information but not all is
>> > >>guaranteed
>> > >> to be captured in the registry.
>> > >>
>> > >> Analyzing: Significant work on provenance aspect related to Airavata
>> is
>> > >>a
>> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
>> > >>research
>> > >> asst. professor at USC. His thesis [2] is a good source of reference
>> on
>> > >> this topic. Both the research and tools are in active development by
>> > >>Prof.
>> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>> > >>uses
>> > >> the same workflow tracking schema and event bus, the tools should be
>> > >>easily
>> > >> integratable but did not have a motivating use case to do so.
>> > >>
>> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> > >> version) and Karma framework which provides context to the gathered
>> > >> provenance. Couple of papers on this topic [4], [5].
>> > >>
>> > >> Executions: This is an active topic within Airavata. Current release
>> of
>> > >> Airavata supports Provenance aware workflow executions and is
>> discussed
>> > >>in
>> > >> [6]. I think the OODT and Airavata integration will greatly benefit
>> this
>> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
>> > >>Rich
>> > >> data management capabilities of OODT should nicely enhance airavata
>> > >> provenance aware executions.
>> > >>
>> > >> Please let me know which of the above aspects you are interested in
>> and
>> > >>we
>> > >> can discuss further.
>> > >>
>> > >> Suresh
>> > >>
>> > >> [1] -
>> > >>
>> > >>
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> > >> [2] - http://gradworks.umi.com/3297094.pdf
>> > >> [3] - http://d2i.indiana.edu/provenance_karma
>> > >> [4] -
>> > >>
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> > >> [5] -
>> > >>
>> > >>
>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>> > >>38325269&CFTOKEN=73364358
>> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
>> > >>
>> > >>
>> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> > >> lewis.mcgibbney@gmail.com> wrote:
>> > >>
>> > >> > Hi Sanjaya,
>> > >> > I am seeing some traffic floating around RE: Airavata OODT
>> integration
>> > >> but I am not clued up.
>> > >> > Can you point me to the code? The issues? The mailing list threads?
>> > >> > I am also extremely interested in provenance tracking of
>> (record???)
>> > >> data which passes through OODT. I would like to expose this as some
>> > >>stage
>> > >> to people who access the data.
>> > >> > Thank you so much.
>> > >> >
>> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>> > >>wrote:
>> > >> > > Hi,
>> > >> > > I am trying to find out ways to improve Airavata provenance aware
>> > >> workflow
>> > >> > > processing based on rich metadata available in Apache OODT. As
>> part
>> > >>of
>> > >> this
>> > >> > > task, I am investigating how Apache Airavata is currently
>> performing
>> > >> > > provenance aware workflow processing. I have looked into the
>> > >>provenance
>> > >> > > manager implementation, where I could see lot of data related to
>> > >> workflow
>> > >> > > execution is saved in registry. But I could see very little of
>> use
>> > >>of
>> > >> those
>> > >> > > data. Could some one please explain the details of provenance
>> aware
>> > >> > > processing in Airavata and any future plans on this area?
>> > >> > >
>> > >> > > Best Regards,
>> > >> > > Sanjaya
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Lewis
>> > >> >
>> > >>
>> > >>
>> >
>> >
>> >
>>
>>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh! I'll do further investigations and let the team know my
findings.
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya,
>
> XBaya and Workflow Interpreter went through some changes which probably
> dropped the condition which makes use of the enableProvenanceSmartRun flag.
> AFAIK, the feature did not get deprecated it should be one of those cases
> which was not actively tested during releases and fell through the cracks.
> Lets see if any one gets motivated to debug and fix this issue. And as
> always, we welcome your patches. Sorry about pointing to a unverified
> feature.
>
> Suresh
>
> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
>
> > Hi Suresh,
> > I have gone through the research paper in item 6 and try to match it to
> the current Airavata implementation. I can see that Airavata collects
> provenance data and stored them in a registry. I couldn't able to find out
> a place where these data has been actually used to improve the workflow
> processing. Xbaya can be configured with option enableProvenanceSmartRun.
> It seems this particular configuration is no longer used. Since provenance
> aware data is already available in registry, is it the responsibility of
> the application to use these data to improve workflow processing based on
> provenance aware data ?
> >
> > Thanks
> > Sanjaya
> >
> >
> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
> wrote:
> > Hey Sanjaya,
> >
> > You may want to check out the Traceable Product product type policy
> > [1] as well as OODT-139 [2] which includes the PCS services that serve
> > up the information and provenance.
> >
> > Cheers,
> > Chris
> >
> > [1]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> > les/trace/
> > [2] http://issues.apache.org/jira/browse/OODT-139
> >
> >
> > -----Original Message-----
> > From: Sanjaya Medonsa <sa...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Friday, August 23, 2013 6:30 PM
> > To: Airavata Dev <de...@airavata.apache.org>
> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: Re: Povenance Aware Workflow Processing
> >
> > >Thanks Suresh! I am interested in Provenance aware workflow processing
> > >where OODT integration could help greatly with rich metadata available.
> > >I'll go through the item 6 and let you know my understanding.
> > >
> > >Sanjaya
> > >
> > >
> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> > >
> > >> Hi Sanjaya, Lewis,
> > >>
> > >> Appreciate your interest on this topic, this is a very good question
> and
> > >> sorry for the delay in responding, these days I feel I am always in
> > >>catch
> > >> up mode.
> > >>
> > >> Let me separate this discussion into four parts: Provenance Gathering,
> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
> executions.
> > >>
> > >> Gathering: Airavata is instrumented with rich provenance centric
> > >> information model which captures each step and publishes to an event
> > >>bus.
> > >> This is encoded in a "workflow tracking" schema [1]. So the current
> > >>trunk
> > >> of Airavata publishes provenance rich information but not all is
> > >>guaranteed
> > >> to be captured in the registry.
> > >>
> > >> Analyzing: Significant work on provenance aspect related to Airavata
> is
> > >>a
> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> > >>research
> > >> asst. professor at USC. His thesis [2] is a good source of reference
> on
> > >> this topic. Both the research and tools are in active development by
> > >>Prof.
> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> > >>uses
> > >> the same workflow tracking schema and event bus, the tools should be
> > >>easily
> > >> integratable but did not have a motivating use case to do so.
> > >>
> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> > >> version) and Karma framework which provides context to the gathered
> > >> provenance. Couple of papers on this topic [4], [5].
> > >>
> > >> Executions: This is an active topic within Airavata. Current release
> of
> > >> Airavata supports Provenance aware workflow executions and is
> discussed
> > >>in
> > >> [6]. I think the OODT and Airavata integration will greatly benefit
> this
> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
> > >>Rich
> > >> data management capabilities of OODT should nicely enhance airavata
> > >> provenance aware executions.
> > >>
> > >> Please let me know which of the above aspects you are interested in
> and
> > >>we
> > >> can discuss further.
> > >>
> > >> Suresh
> > >>
> > >> [1] -
> > >>
> > >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> > >> [2] - http://gradworks.umi.com/3297094.pdf
> > >> [3] - http://d2i.indiana.edu/provenance_karma
> > >> [4] -
> > >>
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> > >> [5] -
> > >>
> > >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> > >>38325269&CFTOKEN=73364358
> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> > >>
> > >>
> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> > >> lewis.mcgibbney@gmail.com> wrote:
> > >>
> > >> > Hi Sanjaya,
> > >> > I am seeing some traffic floating around RE: Airavata OODT
> integration
> > >> but I am not clued up.
> > >> > Can you point me to the code? The issues? The mailing list threads?
> > >> > I am also extremely interested in provenance tracking of (record???)
> > >> data which passes through OODT. I would like to expose this as some
> > >>stage
> > >> to people who access the data.
> > >> > Thank you so much.
> > >> >
> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> > >>wrote:
> > >> > > Hi,
> > >> > > I am trying to find out ways to improve Airavata provenance aware
> > >> workflow
> > >> > > processing based on rich metadata available in Apache OODT. As
> part
> > >>of
> > >> this
> > >> > > task, I am investigating how Apache Airavata is currently
> performing
> > >> > > provenance aware workflow processing. I have looked into the
> > >>provenance
> > >> > > manager implementation, where I could see lot of data related to
> > >> workflow
> > >> > > execution is saved in registry. But I could see very little of use
> > >>of
> > >> those
> > >> > > data. Could some one please explain the details of provenance
> aware
> > >> > > processing in Airavata and any future plans on this area?
> > >> > >
> > >> > > Best Regards,
> > >> > > Sanjaya
> > >> > >
> > >> >
> > >> > --
> > >> > Lewis
> > >> >
> > >>
> > >>
> >
> >
> >
>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh! I'll do further investigations and let the team know my
findings.
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya,
>
> XBaya and Workflow Interpreter went through some changes which probably
> dropped the condition which makes use of the enableProvenanceSmartRun flag.
> AFAIK, the feature did not get deprecated it should be one of those cases
> which was not actively tested during releases and fell through the cracks.
> Lets see if any one gets motivated to debug and fix this issue. And as
> always, we welcome your patches. Sorry about pointing to a unverified
> feature.
>
> Suresh
>
> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
>
> > Hi Suresh,
> > I have gone through the research paper in item 6 and try to match it to
> the current Airavata implementation. I can see that Airavata collects
> provenance data and stored them in a registry. I couldn't able to find out
> a place where these data has been actually used to improve the workflow
> processing. Xbaya can be configured with option enableProvenanceSmartRun.
> It seems this particular configuration is no longer used. Since provenance
> aware data is already available in registry, is it the responsibility of
> the application to use these data to improve workflow processing based on
> provenance aware data ?
> >
> > Thanks
> > Sanjaya
> >
> >
> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
> wrote:
> > Hey Sanjaya,
> >
> > You may want to check out the Traceable Product product type policy
> > [1] as well as OODT-139 [2] which includes the PCS services that serve
> > up the information and provenance.
> >
> > Cheers,
> > Chris
> >
> > [1]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> > les/trace/
> > [2] http://issues.apache.org/jira/browse/OODT-139
> >
> >
> > -----Original Message-----
> > From: Sanjaya Medonsa <sa...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Friday, August 23, 2013 6:30 PM
> > To: Airavata Dev <de...@airavata.apache.org>
> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: Re: Povenance Aware Workflow Processing
> >
> > >Thanks Suresh! I am interested in Provenance aware workflow processing
> > >where OODT integration could help greatly with rich metadata available.
> > >I'll go through the item 6 and let you know my understanding.
> > >
> > >Sanjaya
> > >
> > >
> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> > >
> > >> Hi Sanjaya, Lewis,
> > >>
> > >> Appreciate your interest on this topic, this is a very good question
> and
> > >> sorry for the delay in responding, these days I feel I am always in
> > >>catch
> > >> up mode.
> > >>
> > >> Let me separate this discussion into four parts: Provenance Gathering,
> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
> executions.
> > >>
> > >> Gathering: Airavata is instrumented with rich provenance centric
> > >> information model which captures each step and publishes to an event
> > >>bus.
> > >> This is encoded in a "workflow tracking" schema [1]. So the current
> > >>trunk
> > >> of Airavata publishes provenance rich information but not all is
> > >>guaranteed
> > >> to be captured in the registry.
> > >>
> > >> Analyzing: Significant work on provenance aspect related to Airavata
> is
> > >>a
> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> > >>research
> > >> asst. professor at USC. His thesis [2] is a good source of reference
> on
> > >> this topic. Both the research and tools are in active development by
> > >>Prof.
> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> > >>uses
> > >> the same workflow tracking schema and event bus, the tools should be
> > >>easily
> > >> integratable but did not have a motivating use case to do so.
> > >>
> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> > >> version) and Karma framework which provides context to the gathered
> > >> provenance. Couple of papers on this topic [4], [5].
> > >>
> > >> Executions: This is an active topic within Airavata. Current release
> of
> > >> Airavata supports Provenance aware workflow executions and is
> discussed
> > >>in
> > >> [6]. I think the OODT and Airavata integration will greatly benefit
> this
> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
> > >>Rich
> > >> data management capabilities of OODT should nicely enhance airavata
> > >> provenance aware executions.
> > >>
> > >> Please let me know which of the above aspects you are interested in
> and
> > >>we
> > >> can discuss further.
> > >>
> > >> Suresh
> > >>
> > >> [1] -
> > >>
> > >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> > >> [2] - http://gradworks.umi.com/3297094.pdf
> > >> [3] - http://d2i.indiana.edu/provenance_karma
> > >> [4] -
> > >>
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> > >> [5] -
> > >>
> > >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> > >>38325269&CFTOKEN=73364358
> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> > >>
> > >>
> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> > >> lewis.mcgibbney@gmail.com> wrote:
> > >>
> > >> > Hi Sanjaya,
> > >> > I am seeing some traffic floating around RE: Airavata OODT
> integration
> > >> but I am not clued up.
> > >> > Can you point me to the code? The issues? The mailing list threads?
> > >> > I am also extremely interested in provenance tracking of (record???)
> > >> data which passes through OODT. I would like to expose this as some
> > >>stage
> > >> to people who access the data.
> > >> > Thank you so much.
> > >> >
> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> > >>wrote:
> > >> > > Hi,
> > >> > > I am trying to find out ways to improve Airavata provenance aware
> > >> workflow
> > >> > > processing based on rich metadata available in Apache OODT. As
> part
> > >>of
> > >> this
> > >> > > task, I am investigating how Apache Airavata is currently
> performing
> > >> > > provenance aware workflow processing. I have looked into the
> > >>provenance
> > >> > > manager implementation, where I could see lot of data related to
> > >> workflow
> > >> > > execution is saved in registry. But I could see very little of use
> > >>of
> > >> those
> > >> > > data. Could some one please explain the details of provenance
> aware
> > >> > > processing in Airavata and any future plans on this area?
> > >> > >
> > >> > > Best Regards,
> > >> > > Sanjaya
> > >> > >
> > >> >
> > >> > --
> > >> > Lewis
> > >> >
> > >>
> > >>
> >
> >
> >
>
>
Re: Povenance Aware Workflow Processing
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya,
XBaya and Workflow Interpreter went through some changes which probably dropped the condition which makes use of the enableProvenanceSmartRun flag. AFAIK, the feature did not get deprecated it should be one of those cases which was not actively tested during releases and fell through the cracks. Lets see if any one gets motivated to debug and fix this issue. And as always, we welcome your patches. Sorry about pointing to a unverified feature.
Suresh
On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Hi Suresh,
> I have gone through the research paper in item 6 and try to match it to the current Airavata implementation. I can see that Airavata collects provenance data and stored them in a registry. I couldn't able to find out a place where these data has been actually used to improve the workflow processing. Xbaya can be configured with option enableProvenanceSmartRun. It seems this particular configuration is no longer used. Since provenance aware data is already available in registry, is it the responsibility of the application to use these data to improve workflow processing based on provenance aware data ?
>
> Thanks
> Sanjaya
>
>
> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>
Re: Povenance Aware Workflow Processing
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya,
XBaya and Workflow Interpreter went through some changes which probably dropped the condition which makes use of the enableProvenanceSmartRun flag. AFAIK, the feature did not get deprecated it should be one of those cases which was not actively tested during releases and fell through the cracks. Lets see if any one gets motivated to debug and fix this issue. And as always, we welcome your patches. Sorry about pointing to a unverified feature.
Suresh
On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Hi Suresh,
> I have gone through the research paper in item 6 and try to match it to the current Airavata implementation. I can see that Airavata collects provenance data and stored them in a registry. I couldn't able to find out a place where these data has been actually used to improve the workflow processing. Xbaya can be configured with option enableProvenanceSmartRun. It seems this particular configuration is no longer used. Since provenance aware data is already available in registry, is it the responsibility of the application to use these data to improve workflow processing based on provenance aware data ?
>
> Thanks
> Sanjaya
>
>
> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Suresh,
I have gone through the research paper in item 6 and try to match it to the
current Airavata implementation. I can see that Airavata collects
provenance data and stored them in a registry. I couldn't able to find out
a place where these data has been actually used to improve the workflow
processing. Xbaya can be configured with option enableProvenanceSmartRun.
It seems this particular configuration is no longer used. Since provenance
aware data is already available in registry, is it the responsibility of
the application to use these data to improve workflow processing based on
provenance aware data ?
Thanks
Sanjaya
On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Suresh,
I have gone through the research paper in item 6 and try to match it to the
current Airavata implementation. I can see that Airavata collects
provenance data and stored them in a registry. I couldn't able to find out
a place where these data has been actually used to improve the workflow
processing. Xbaya can be configured with option enableProvenanceSmartRun.
It seems this particular configuration is no longer used. Since provenance
aware data is already available in registry, is it the responsibility of
the application to use these data to improve workflow processing based on
provenance aware data ?
Thanks
Sanjaya
On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>
Re: Povenance Aware Workflow Processing
Posted by Chris Mattmann <ma...@apache.org>.
Hey Sanjaya,
You may want to check out the Traceable Product product type policy
[1] as well as OODT-139 [2] which includes the PCS services that serve
up the information and provenance.
Cheers,
Chris
[1]
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
les/trace/
[2] http://issues.apache.org/jira/browse/OODT-139
-----Original Message-----
From: Sanjaya Medonsa <sa...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Friday, August 23, 2013 6:30 PM
To: Airavata Dev <de...@airavata.apache.org>
Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Povenance Aware Workflow Processing
>Thanks Suresh! I am interested in Provenance aware workflow processing
>where OODT integration could help greatly with rich metadata available.
>I'll go through the item 6 and let you know my understanding.
>
>Sanjaya
>
>
>On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Sanjaya, Lewis,
>>
>> Appreciate your interest on this topic, this is a very good question and
>> sorry for the delay in responding, these days I feel I am always in
>>catch
>> up mode.
>>
>> Let me separate this discussion into four parts: Provenance Gathering,
>> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
>>
>> Gathering: Airavata is instrumented with rich provenance centric
>> information model which captures each step and publishes to an event
>>bus.
>> This is encoded in a "workflow tracking" schema [1]. So the current
>>trunk
>> of Airavata publishes provenance rich information but not all is
>>guaranteed
>> to be captured in the registry.
>>
>> Analyzing: Significant work on provenance aspect related to Airavata is
>>a
>> by product of doctoral dissertation of Yogesh Simmhan, currently a
>>research
>> asst. professor at USC. His thesis [2] is a good source of reference on
>> this topic. Both the research and tools are in active development by
>>Prof.
>> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>>uses
>> the same workflow tracking schema and event bus, the tools should be
>>easily
>> integratable but did not have a motivating use case to do so.
>>
>> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> version) and Karma framework which provides context to the gathered
>> provenance. Couple of papers on this topic [4], [5].
>>
>> Executions: This is an active topic within Airavata. Current release of
>> Airavata supports Provenance aware workflow executions and is discussed
>>in
>> [6]. I think the OODT and Airavata integration will greatly benefit this
>> aspect. Airavata Registry metadata cataloging is very primitive. And
>>Rich
>> data management capabilities of OODT should nicely enhance airavata
>> provenance aware executions.
>>
>> Please let me know which of the above aspects you are interested in and
>>we
>> can discuss further.
>>
>> Suresh
>>
>> [1] -
>>
>>https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> [2] - http://gradworks.umi.com/3297094.pdf
>> [3] - http://d2i.indiana.edu/provenance_karma
>> [4] -
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> [5] -
>>
>>http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>>38325269&CFTOKEN=73364358
>> [6] - http://dl.acm.org/citation.cfm?id=2359844
>>
>>
>> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>> > Hi Sanjaya,
>> > I am seeing some traffic floating around RE: Airavata OODT integration
>> but I am not clued up.
>> > Can you point me to the code? The issues? The mailing list threads?
>> > I am also extremely interested in provenance tracking of (record???)
>> data which passes through OODT. I would like to expose this as some
>>stage
>> to people who access the data.
>> > Thank you so much.
>> >
>> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>>wrote:
>> > > Hi,
>> > > I am trying to find out ways to improve Airavata provenance aware
>> workflow
>> > > processing based on rich metadata available in Apache OODT. As part
>>of
>> this
>> > > task, I am investigating how Apache Airavata is currently performing
>> > > provenance aware workflow processing. I have looked into the
>>provenance
>> > > manager implementation, where I could see lot of data related to
>> workflow
>> > > execution is saved in registry. But I could see very little of use
>>of
>> those
>> > > data. Could some one please explain the details of provenance aware
>> > > processing in Airavata and any future plans on this area?
>> > >
>> > > Best Regards,
>> > > Sanjaya
>> > >
>> >
>> > --
>> > Lewis
>> >
>>
>>
Re: Povenance Aware Workflow Processing
Posted by Chris Mattmann <ma...@apache.org>.
Hey Sanjaya,
You may want to check out the Traceable Product product type policy
[1] as well as OODT-139 [2] which includes the PCS services that serve
up the information and provenance.
Cheers,
Chris
[1]
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
les/trace/
[2] http://issues.apache.org/jira/browse/OODT-139
-----Original Message-----
From: Sanjaya Medonsa <sa...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Friday, August 23, 2013 6:30 PM
To: Airavata Dev <de...@airavata.apache.org>
Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Povenance Aware Workflow Processing
>Thanks Suresh! I am interested in Provenance aware workflow processing
>where OODT integration could help greatly with rich metadata available.
>I'll go through the item 6 and let you know my understanding.
>
>Sanjaya
>
>
>On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Sanjaya, Lewis,
>>
>> Appreciate your interest on this topic, this is a very good question and
>> sorry for the delay in responding, these days I feel I am always in
>>catch
>> up mode.
>>
>> Let me separate this discussion into four parts: Provenance Gathering,
>> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
>>
>> Gathering: Airavata is instrumented with rich provenance centric
>> information model which captures each step and publishes to an event
>>bus.
>> This is encoded in a "workflow tracking" schema [1]. So the current
>>trunk
>> of Airavata publishes provenance rich information but not all is
>>guaranteed
>> to be captured in the registry.
>>
>> Analyzing: Significant work on provenance aspect related to Airavata is
>>a
>> by product of doctoral dissertation of Yogesh Simmhan, currently a
>>research
>> asst. professor at USC. His thesis [2] is a good source of reference on
>> this topic. Both the research and tools are in active development by
>>Prof.
>> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>>uses
>> the same workflow tracking schema and event bus, the tools should be
>>easily
>> integratable but did not have a motivating use case to do so.
>>
>> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> version) and Karma framework which provides context to the gathered
>> provenance. Couple of papers on this topic [4], [5].
>>
>> Executions: This is an active topic within Airavata. Current release of
>> Airavata supports Provenance aware workflow executions and is discussed
>>in
>> [6]. I think the OODT and Airavata integration will greatly benefit this
>> aspect. Airavata Registry metadata cataloging is very primitive. And
>>Rich
>> data management capabilities of OODT should nicely enhance airavata
>> provenance aware executions.
>>
>> Please let me know which of the above aspects you are interested in and
>>we
>> can discuss further.
>>
>> Suresh
>>
>> [1] -
>>
>>https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> [2] - http://gradworks.umi.com/3297094.pdf
>> [3] - http://d2i.indiana.edu/provenance_karma
>> [4] -
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> [5] -
>>
>>http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>>38325269&CFTOKEN=73364358
>> [6] - http://dl.acm.org/citation.cfm?id=2359844
>>
>>
>> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>> > Hi Sanjaya,
>> > I am seeing some traffic floating around RE: Airavata OODT integration
>> but I am not clued up.
>> > Can you point me to the code? The issues? The mailing list threads?
>> > I am also extremely interested in provenance tracking of (record???)
>> data which passes through OODT. I would like to expose this as some
>>stage
>> to people who access the data.
>> > Thank you so much.
>> >
>> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>>wrote:
>> > > Hi,
>> > > I am trying to find out ways to improve Airavata provenance aware
>> workflow
>> > > processing based on rich metadata available in Apache OODT. As part
>>of
>> this
>> > > task, I am investigating how Apache Airavata is currently performing
>> > > provenance aware workflow processing. I have looked into the
>>provenance
>> > > manager implementation, where I could see lot of data related to
>> workflow
>> > > execution is saved in registry. But I could see very little of use
>>of
>> those
>> > > data. Could some one please explain the details of provenance aware
>> > > processing in Airavata and any future plans on this area?
>> > >
>> > > Best Regards,
>> > > Sanjaya
>> > >
>> >
>> > --
>> > Lewis
>> >
>>
>>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh! I am interested in Provenance aware workflow processing
where OODT integration could help greatly with rich metadata available.
I'll go through the item 6 and let you know my understanding.
Sanjaya
On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya, Lewis,
>
> Appreciate your interest on this topic, this is a very good question and
> sorry for the delay in responding, these days I feel I am always in catch
> up mode.
>
> Let me separate this discussion into four parts: Provenance Gathering,
> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
>
> Gathering: Airavata is instrumented with rich provenance centric
> information model which captures each step and publishes to an event bus.
> This is encoded in a "workflow tracking" schema [1]. So the current trunk
> of Airavata publishes provenance rich information but not all is guaranteed
> to be captured in the registry.
>
> Analyzing: Significant work on provenance aspect related to Airavata is a
> by product of doctoral dissertation of Yogesh Simmhan, currently a research
> asst. professor at USC. His thesis [2] is a good source of reference on
> this topic. Both the research and tools are in active development by Prof.
> Beth Plale's group at IU [3]. Theoretically speaking since both Karma uses
> the same workflow tracking schema and event bus, the tools should be easily
> integratable but did not have a motivating use case to do so.
>
> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> version) and Karma framework which provides context to the gathered
> provenance. Couple of papers on this topic [4], [5].
>
> Executions: This is an active topic within Airavata. Current release of
> Airavata supports Provenance aware workflow executions and is discussed in
> [6]. I think the OODT and Airavata integration will greatly benefit this
> aspect. Airavata Registry metadata cataloging is very primitive. And Rich
> data management capabilities of OODT should nicely enhance airavata
> provenance aware executions.
>
> Please let me know which of the above aspects you are interested in and we
> can discuss further.
>
> Suresh
>
> [1] -
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> [2] - http://gradworks.umi.com/3297094.pdf
> [3] - http://d2i.indiana.edu/provenance_karma
> [4] -
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> [5] -
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=238325269&CFTOKEN=73364358
> [6] - http://dl.acm.org/citation.cfm?id=2359844
>
>
> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi Sanjaya,
> > I am seeing some traffic floating around RE: Airavata OODT integration
> but I am not clued up.
> > Can you point me to the code? The issues? The mailing list threads?
> > I am also extremely interested in provenance tracking of (record???)
> data which passes through OODT. I would like to expose this as some stage
> to people who access the data.
> > Thank you so much.
> >
> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> > > Hi,
> > > I am trying to find out ways to improve Airavata provenance aware
> workflow
> > > processing based on rich metadata available in Apache OODT. As part of
> this
> > > task, I am investigating how Apache Airavata is currently performing
> > > provenance aware workflow processing. I have looked into the provenance
> > > manager implementation, where I could see lot of data related to
> workflow
> > > execution is saved in registry. But I could see very little of use of
> those
> > > data. Could some one please explain the details of provenance aware
> > > processing in Airavata and any future plans on this area?
> > >
> > > Best Regards,
> > > Sanjaya
> > >
> >
> > --
> > Lewis
> >
>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh! I am interested in Provenance aware workflow processing
where OODT integration could help greatly with rich metadata available.
I'll go through the item 6 and let you know my understanding.
Sanjaya
On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya, Lewis,
>
> Appreciate your interest on this topic, this is a very good question and
> sorry for the delay in responding, these days I feel I am always in catch
> up mode.
>
> Let me separate this discussion into four parts: Provenance Gathering,
> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
>
> Gathering: Airavata is instrumented with rich provenance centric
> information model which captures each step and publishes to an event bus.
> This is encoded in a "workflow tracking" schema [1]. So the current trunk
> of Airavata publishes provenance rich information but not all is guaranteed
> to be captured in the registry.
>
> Analyzing: Significant work on provenance aspect related to Airavata is a
> by product of doctoral dissertation of Yogesh Simmhan, currently a research
> asst. professor at USC. His thesis [2] is a good source of reference on
> this topic. Both the research and tools are in active development by Prof.
> Beth Plale's group at IU [3]. Theoretically speaking since both Karma uses
> the same workflow tracking schema and event bus, the tools should be easily
> integratable but did not have a motivating use case to do so.
>
> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> version) and Karma framework which provides context to the gathered
> provenance. Couple of papers on this topic [4], [5].
>
> Executions: This is an active topic within Airavata. Current release of
> Airavata supports Provenance aware workflow executions and is discussed in
> [6]. I think the OODT and Airavata integration will greatly benefit this
> aspect. Airavata Registry metadata cataloging is very primitive. And Rich
> data management capabilities of OODT should nicely enhance airavata
> provenance aware executions.
>
> Please let me know which of the above aspects you are interested in and we
> can discuss further.
>
> Suresh
>
> [1] -
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> [2] - http://gradworks.umi.com/3297094.pdf
> [3] - http://d2i.indiana.edu/provenance_karma
> [4] -
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> [5] -
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=238325269&CFTOKEN=73364358
> [6] - http://dl.acm.org/citation.cfm?id=2359844
>
>
> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > Hi Sanjaya,
> > I am seeing some traffic floating around RE: Airavata OODT integration
> but I am not clued up.
> > Can you point me to the code? The issues? The mailing list threads?
> > I am also extremely interested in provenance tracking of (record???)
> data which passes through OODT. I would like to expose this as some stage
> to people who access the data.
> > Thank you so much.
> >
> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> > > Hi,
> > > I am trying to find out ways to improve Airavata provenance aware
> workflow
> > > processing based on rich metadata available in Apache OODT. As part of
> this
> > > task, I am investigating how Apache Airavata is currently performing
> > > provenance aware workflow processing. I have looked into the provenance
> > > manager implementation, where I could see lot of data related to
> workflow
> > > execution is saved in registry. But I could see very little of use of
> those
> > > data. Could some one please explain the details of provenance aware
> > > processing in Airavata and any future plans on this area?
> > >
> > > Best Regards,
> > > Sanjaya
> > >
> >
> > --
> > Lewis
> >
>
>
Re: Povenance Aware Workflow Processing
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya, Lewis,
Appreciate your interest on this topic, this is a very good question and sorry for the delay in responding, these days I feel I am always in catch up mode.
Let me separate this discussion into four parts: Provenance Gathering, Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
Gathering: Airavata is instrumented with rich provenance centric information model which captures each step and publishes to an event bus. This is encoded in a "workflow tracking" schema [1]. So the current trunk of Airavata publishes provenance rich information but not all is guaranteed to be captured in the registry.
Analyzing: Significant work on provenance aspect related to Airavata is a by product of doctoral dissertation of Yogesh Simmhan, currently a research asst. professor at USC. His thesis [2] is a good source of reference on this topic. Both the research and tools are in active development by Prof. Beth Plale's group at IU [3]. Theoretically speaking since both Karma uses the same workflow tracking schema and event bus, the tools should be easily integratable but did not have a motivating use case to do so.
Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD version) and Karma framework which provides context to the gathered provenance. Couple of papers on this topic [4], [5].
Executions: This is an active topic within Airavata. Current release of Airavata supports Provenance aware workflow executions and is discussed in [6]. I think the OODT and Airavata integration will greatly benefit this aspect. Airavata Registry metadata cataloging is very primitive. And Rich data management capabilities of OODT should nicely enhance airavata provenance aware executions.
Please let me know which of the above aspects you are interested in and we can discuss further.
Suresh
[1] - https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
[2] - http://gradworks.umi.com/3297094.pdf
[3] - http://d2i.indiana.edu/provenance_karma
[4] - http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
[5] - http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=238325269&CFTOKEN=73364358
[6] - http://dl.acm.org/citation.cfm?id=2359844
On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> Hi Sanjaya,
> I am seeing some traffic floating around RE: Airavata OODT integration but I am not clued up.
> Can you point me to the code? The issues? The mailing list threads?
> I am also extremely interested in provenance tracking of (record???) data which passes through OODT. I would like to expose this as some stage to people who access the data.
> Thank you so much.
>
> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> > Hi,
> > I am trying to find out ways to improve Airavata provenance aware workflow
> > processing based on rich metadata available in Apache OODT. As part of this
> > task, I am investigating how Apache Airavata is currently performing
> > provenance aware workflow processing. I have looked into the provenance
> > manager implementation, where I could see lot of data related to workflow
> > execution is saved in registry. But I could see very little of use of those
> > data. Could some one please explain the details of provenance aware
> > processing in Airavata and any future plans on this area?
> >
> > Best Regards,
> > Sanjaya
> >
>
> --
> Lewis
>
Re: Povenance Aware Workflow Processing
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya, Lewis,
Appreciate your interest on this topic, this is a very good question and sorry for the delay in responding, these days I feel I am always in catch up mode.
Let me separate this discussion into four parts: Provenance Gathering, Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
Gathering: Airavata is instrumented with rich provenance centric information model which captures each step and publishes to an event bus. This is encoded in a "workflow tracking" schema [1]. So the current trunk of Airavata publishes provenance rich information but not all is guaranteed to be captured in the registry.
Analyzing: Significant work on provenance aspect related to Airavata is a by product of doctoral dissertation of Yogesh Simmhan, currently a research asst. professor at USC. His thesis [2] is a good source of reference on this topic. Both the research and tools are in active development by Prof. Beth Plale's group at IU [3]. Theoretically speaking since both Karma uses the same workflow tracking schema and event bus, the tools should be easily integratable but did not have a motivating use case to do so.
Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD version) and Karma framework which provides context to the gathered provenance. Couple of papers on this topic [4], [5].
Executions: This is an active topic within Airavata. Current release of Airavata supports Provenance aware workflow executions and is discussed in [6]. I think the OODT and Airavata integration will greatly benefit this aspect. Airavata Registry metadata cataloging is very primitive. And Rich data management capabilities of OODT should nicely enhance airavata provenance aware executions.
Please let me know which of the above aspects you are interested in and we can discuss further.
Suresh
[1] - https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
[2] - http://gradworks.umi.com/3297094.pdf
[3] - http://d2i.indiana.edu/provenance_karma
[4] - http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
[5] - http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=238325269&CFTOKEN=73364358
[6] - http://dl.acm.org/citation.cfm?id=2359844
On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <le...@gmail.com> wrote:
> Hi Sanjaya,
> I am seeing some traffic floating around RE: Airavata OODT integration but I am not clued up.
> Can you point me to the code? The issues? The mailing list threads?
> I am also extremely interested in provenance tracking of (record???) data which passes through OODT. I would like to expose this as some stage to people who access the data.
> Thank you so much.
>
> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> > Hi,
> > I am trying to find out ways to improve Airavata provenance aware workflow
> > processing based on rich metadata available in Apache OODT. As part of this
> > task, I am investigating how Apache Airavata is currently performing
> > provenance aware workflow processing. I have looked into the provenance
> > manager implementation, where I could see lot of data related to workflow
> > execution is saved in registry. But I could see very little of use of those
> > data. Could some one please explain the details of provenance aware
> > processing in Airavata and any future plans on this area?
> >
> > Best Regards,
> > Sanjaya
> >
>
> --
> Lewis
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi,
As per my understanding, in simple terms, provenance aware workflow
processing means if output is available from previous execution, then that
particular task doesn't require to execute again as output is already
available. Then that output can be used for rest of the workflow execution.
Provenance aware workflow processing avoids re executing already executed
tasks, thus improving overall performance. Please correct if my
understanding is wrong.
When it comes to Airavata, what is the best way to identify previously
executed tasks. Is it from input and the service name ? Please let me know
your thoughts. Here I am trying to understand best way to use the data
available in Airavata Registry and the OODT metadata catalog. As part of
the OODT-integration, output of task execution ingest into OODT file
manager server together with corresponding meta data. Basically I need to
understand what sort of Metadata that needs to be stored together with the
output. At the moment I am planning to store service name and workflow
instance id together with other metadata extracted based on output by OODT
metadata extractors. I guess I may need to store input as well.
Please let me know your valuable thoughts, it would be really helpful.
Best Regards,
Sanjaya
On Mon, Jul 22, 2013 at 11:13 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Hi Lewis,
> As you mentioned, RE: Airavata OODT integration is the main email
> thread related to my work. Still code has not been checked in. Earlier I
> have published a review request for my code changes. I have got some
> feedback from Chris and I am planning to publish a new review request
> during next week with the updates. In short, I am using OODT task wrapper
> capabilities to improve Airavata workflow execution. Basically there are
> two main tasks.
> 1. Stage input file from OODT file manager server as input for Airavata
> workflow execution
> 2. Ingest generated output of a Airavata workflow execution to OODT
> file manager server with metadata
>
> Improving provenance aware workflow processing is my next task. Hope this
> helps you. Please let me know if you need more details.
>
> Best Regards,
> Sanjaya
>
>
> On Mon, Jul 22, 2013 at 8:27 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Sanjaya,
>> I am seeing some traffic floating around RE: Airavata OODT integration
>> but I am not clued up.
>> Can you point me to the code? The issues? The mailing list threads?
>> I am also extremely interested in provenance tracking of (record???) data
>> which passes through OODT. I would like to expose this as some stage to
>> people who access the data.
>> Thank you so much.
>>
>>
>> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
>> > Hi,
>> > I am trying to find out ways to improve Airavata provenance aware
>> workflow
>> > processing based on rich metadata available in Apache OODT. As part of
>> this
>> > task, I am investigating how Apache Airavata is currently performing
>> > provenance aware workflow processing. I have looked into the provenance
>> > manager implementation, where I could see lot of data related to
>> workflow
>> > execution is saved in registry. But I could see very little of use of
>> those
>> > data. Could some one please explain the details of provenance aware
>> > processing in Airavata and any future plans on this area?
>> >
>> > Best Regards,
>> > Sanjaya
>> >
>>
>> --
>> *Lewis*
>>
>>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi,
As per my understanding, in simple terms, provenance aware workflow
processing means if output is available from previous execution, then that
particular task doesn't require to execute again as output is already
available. Then that output can be used for rest of the workflow execution.
Provenance aware workflow processing avoids re executing already executed
tasks, thus improving overall performance. Please correct if my
understanding is wrong.
When it comes to Airavata, what is the best way to identify previously
executed tasks. Is it from input and the service name ? Please let me know
your thoughts. Here I am trying to understand best way to use the data
available in Airavata Registry and the OODT metadata catalog. As part of
the OODT-integration, output of task execution ingest into OODT file
manager server together with corresponding meta data. Basically I need to
understand what sort of Metadata that needs to be stored together with the
output. At the moment I am planning to store service name and workflow
instance id together with other metadata extracted based on output by OODT
metadata extractors. I guess I may need to store input as well.
Please let me know your valuable thoughts, it would be really helpful.
Best Regards,
Sanjaya
On Mon, Jul 22, 2013 at 11:13 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Hi Lewis,
> As you mentioned, RE: Airavata OODT integration is the main email
> thread related to my work. Still code has not been checked in. Earlier I
> have published a review request for my code changes. I have got some
> feedback from Chris and I am planning to publish a new review request
> during next week with the updates. In short, I am using OODT task wrapper
> capabilities to improve Airavata workflow execution. Basically there are
> two main tasks.
> 1. Stage input file from OODT file manager server as input for Airavata
> workflow execution
> 2. Ingest generated output of a Airavata workflow execution to OODT
> file manager server with metadata
>
> Improving provenance aware workflow processing is my next task. Hope this
> helps you. Please let me know if you need more details.
>
> Best Regards,
> Sanjaya
>
>
> On Mon, Jul 22, 2013 at 8:27 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Sanjaya,
>> I am seeing some traffic floating around RE: Airavata OODT integration
>> but I am not clued up.
>> Can you point me to the code? The issues? The mailing list threads?
>> I am also extremely interested in provenance tracking of (record???) data
>> which passes through OODT. I would like to expose this as some stage to
>> people who access the data.
>> Thank you so much.
>>
>>
>> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
>> > Hi,
>> > I am trying to find out ways to improve Airavata provenance aware
>> workflow
>> > processing based on rich metadata available in Apache OODT. As part of
>> this
>> > task, I am investigating how Apache Airavata is currently performing
>> > provenance aware workflow processing. I have looked into the provenance
>> > manager implementation, where I could see lot of data related to
>> workflow
>> > execution is saved in registry. But I could see very little of use of
>> those
>> > data. Could some one please explain the details of provenance aware
>> > processing in Airavata and any future plans on this area?
>> >
>> > Best Regards,
>> > Sanjaya
>> >
>>
>> --
>> *Lewis*
>>
>>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Lewis,
As you mentioned, RE: Airavata OODT integration is the main email
thread related to my work. Still code has not been checked in. Earlier I
have published a review request for my code changes. I have got some
feedback from Chris and I am planning to publish a new review request
during next week with the updates. In short, I am using OODT task wrapper
capabilities to improve Airavata workflow execution. Basically there are
two main tasks.
1. Stage input file from OODT file manager server as input for Airavata
workflow execution
2. Ingest generated output of a Airavata workflow execution to OODT file
manager server with metadata
Improving provenance aware workflow processing is my next task. Hope this
helps you. Please let me know if you need more details.
Best Regards,
Sanjaya
On Mon, Jul 22, 2013 at 8:27 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Sanjaya,
> I am seeing some traffic floating around RE: Airavata OODT integration but
> I am not clued up.
> Can you point me to the code? The issues? The mailing list threads?
> I am also extremely interested in provenance tracking of (record???) data
> which passes through OODT. I would like to expose this as some stage to
> people who access the data.
> Thank you so much.
>
>
> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> > Hi,
> > I am trying to find out ways to improve Airavata provenance aware
> workflow
> > processing based on rich metadata available in Apache OODT. As part of
> this
> > task, I am investigating how Apache Airavata is currently performing
> > provenance aware workflow processing. I have looked into the provenance
> > manager implementation, where I could see lot of data related to workflow
> > execution is saved in registry. But I could see very little of use of
> those
> > data. Could some one please explain the details of provenance aware
> > processing in Airavata and any future plans on this area?
> >
> > Best Regards,
> > Sanjaya
> >
>
> --
> *Lewis*
>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Lewis,
As you mentioned, RE: Airavata OODT integration is the main email
thread related to my work. Still code has not been checked in. Earlier I
have published a review request for my code changes. I have got some
feedback from Chris and I am planning to publish a new review request
during next week with the updates. In short, I am using OODT task wrapper
capabilities to improve Airavata workflow execution. Basically there are
two main tasks.
1. Stage input file from OODT file manager server as input for Airavata
workflow execution
2. Ingest generated output of a Airavata workflow execution to OODT file
manager server with metadata
Improving provenance aware workflow processing is my next task. Hope this
helps you. Please let me know if you need more details.
Best Regards,
Sanjaya
On Mon, Jul 22, 2013 at 8:27 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Sanjaya,
> I am seeing some traffic floating around RE: Airavata OODT integration but
> I am not clued up.
> Can you point me to the code? The issues? The mailing list threads?
> I am also extremely interested in provenance tracking of (record???) data
> which passes through OODT. I would like to expose this as some stage to
> people who access the data.
> Thank you so much.
>
>
> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> > Hi,
> > I am trying to find out ways to improve Airavata provenance aware
> workflow
> > processing based on rich metadata available in Apache OODT. As part of
> this
> > task, I am investigating how Apache Airavata is currently performing
> > provenance aware workflow processing. I have looked into the provenance
> > manager implementation, where I could see lot of data related to workflow
> > execution is saved in registry. But I could see very little of use of
> those
> > data. Could some one please explain the details of provenance aware
> > processing in Airavata and any future plans on this area?
> >
> > Best Regards,
> > Sanjaya
> >
>
> --
> *Lewis*
>
>
Re: Povenance Aware Workflow Processing
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Sanjaya,
I am seeing some traffic floating around RE: Airavata OODT integration but
I am not clued up.
Can you point me to the code? The issues? The mailing list threads?
I am also extremely interested in provenance tracking of (record???) data
which passes through OODT. I would like to expose this as some stage to
people who access the data.
Thank you so much.
On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Hi,
> I am trying to find out ways to improve Airavata provenance aware workflow
> processing based on rich metadata available in Apache OODT. As part of
this
> task, I am investigating how Apache Airavata is currently performing
> provenance aware workflow processing. I have looked into the provenance
> manager implementation, where I could see lot of data related to workflow
> execution is saved in registry. But I could see very little of use of
those
> data. Could some one please explain the details of provenance aware
> processing in Airavata and any future plans on this area?
>
> Best Regards,
> Sanjaya
>
--
*Lewis*
Re: Povenance Aware Workflow Processing
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Sanjaya,
I am seeing some traffic floating around RE: Airavata OODT integration but
I am not clued up.
Can you point me to the code? The issues? The mailing list threads?
I am also extremely interested in provenance tracking of (record???) data
which passes through OODT. I would like to expose this as some stage to
people who access the data.
Thank you so much.
On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Hi,
> I am trying to find out ways to improve Airavata provenance aware workflow
> processing based on rich metadata available in Apache OODT. As part of
this
> task, I am investigating how Apache Airavata is currently performing
> provenance aware workflow processing. I have looked into the provenance
> manager implementation, where I could see lot of data related to workflow
> execution is saved in registry. But I could see very little of use of
those
> data. Could some one please explain the details of provenance aware
> processing in Airavata and any future plans on this area?
>
> Best Regards,
> Sanjaya
>
--
*Lewis*