You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Sanjaya Medonsa <sa...@gmail.com> on 2013/10/15 03:01:37 UTC
Re: Povenance Aware Workflow Processing
Hi Suresh,
I have gone through the research paper in item 6 and try to match it to the
current Airavata implementation. I can see that Airavata collects
provenance data and stored them in a registry. I couldn't able to find out
a place where these data has been actually used to improve the workflow
processing. Xbaya can be configured with option enableProvenanceSmartRun.
It seems this particular configuration is no longer used. Since provenance
aware data is already available in registry, is it the responsibility of
the application to use these data to improve workflow processing based on
provenance aware data ?
Thanks
Sanjaya
On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>
Re: Povenance Aware Workflow Processing
Posted by Marlon Pierce <ma...@iu.edu>.
Hi Sanjaya--
Could you file a Jira ticket for this? We also welcome patches if you
want to contribute it.
Marlon
On 10/15/13 9:32 PM, Sanjaya Medonsa wrote:
> Hi Suresh,
> I have further looked into the code to understand, how provenance aware
> workflow processing has been implemented in current code base. As you have
> stated, it seems functionality is broken. In WorkflowInterpreter class,
> there is a method called readProvenance (only place where ProvenanceReader
> is used) . It is a private method, no longer a part of the actual
> execution. I think, it's better to go back to version where provenance
> aware workflow processing working as expected to get better understand on
> the expected flow. Then fix the issues in current version. Could you please
> let me know, possible version that this particular functionality worked as
> expected ?
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
>
>> Thanks Suresh! I'll do further investigations and let the team know my
>> findings.
>>
>> Best Regards,
>> Sanjaya
>>
>>
>> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Sanjaya,
>>>
>>> XBaya and Workflow Interpreter went through some changes which probably
>>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>>> AFAIK, the feature did not get deprecated it should be one of those cases
>>> which was not actively tested during releases and fell through the cracks.
>>> Lets see if any one gets motivated to debug and fix this issue. And as
>>> always, we welcome your patches. Sorry about pointing to a unverified
>>> feature.
>>>
>>> Suresh
>>>
>>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com>
>>> wrote:
>>>
>>>> Hi Suresh,
>>>> I have gone through the research paper in item 6 and try to match it to
>>> the current Airavata implementation. I can see that Airavata collects
>>> provenance data and stored them in a registry. I couldn't able to find out
>>> a place where these data has been actually used to improve the workflow
>>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>>> It seems this particular configuration is no longer used. Since provenance
>>> aware data is already available in registry, is it the responsibility of
>>> the application to use these data to improve workflow processing based on
>>> provenance aware data ?
>>>> Thanks
>>>> Sanjaya
>>>>
>>>>
>>>> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
>>> wrote:
>>>> Hey Sanjaya,
>>>>
>>>> You may want to check out the Traceable Product product type policy
>>>> [1] as well as OODT-139 [2] which includes the PCS services that serve
>>>> up the information and provenance.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> [1]
>>>>
>>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>>>> les/trace/
>>>> [2] http://issues.apache.org/jira/browse/OODT-139
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Sanjaya Medonsa <sa...@gmail.com>
>>>> Reply-To: <de...@oodt.apache.org>
>>>> Date: Friday, August 23, 2013 6:30 PM
>>>> To: Airavata Dev <de...@airavata.apache.org>
>>>> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
>>>> Subject: Re: Povenance Aware Workflow Processing
>>>>
>>>>> Thanks Suresh! I am interested in Provenance aware workflow processing
>>>>> where OODT integration could help greatly with rich metadata available.
>>>>> I'll go through the item 6 and let you know my understanding.
>>>>>
>>>>> Sanjaya
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org>
>>> wrote:
>>>>>> Hi Sanjaya, Lewis,
>>>>>>
>>>>>> Appreciate your interest on this topic, this is a very good question
>>> and
>>>>>> sorry for the delay in responding, these days I feel I am always in
>>>>>> catch
>>>>>> up mode.
>>>>>>
>>>>>> Let me separate this discussion into four parts: Provenance
>>> Gathering,
>>>>>> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>>> executions.
>>>>>> Gathering: Airavata is instrumented with rich provenance centric
>>>>>> information model which captures each step and publishes to an event
>>>>>> bus.
>>>>>> This is encoded in a "workflow tracking" schema [1]. So the current
>>>>>> trunk
>>>>>> of Airavata publishes provenance rich information but not all is
>>>>>> guaranteed
>>>>>> to be captured in the registry.
>>>>>>
>>>>>> Analyzing: Significant work on provenance aspect related to Airavata
>>> is
>>>>>> a
>>>>>> by product of doctoral dissertation of Yogesh Simmhan, currently a
>>>>>> research
>>>>>> asst. professor at USC. His thesis [2] is a good source of reference
>>> on
>>>>>> this topic. Both the research and tools are in active development by
>>>>>> Prof.
>>>>>> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>>>>>> uses
>>>>>> the same workflow tracking schema and event bus, the tools should be
>>>>>> easily
>>>>>> integratable but did not have a motivating use case to do so.
>>>>>>
>>>>>> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>>>>>> version) and Karma framework which provides context to the gathered
>>>>>> provenance. Couple of papers on this topic [4], [5].
>>>>>>
>>>>>> Executions: This is an active topic within Airavata. Current release
>>> of
>>>>>> Airavata supports Provenance aware workflow executions and is
>>> discussed
>>>>>> in
>>>>>> [6]. I think the OODT and Airavata integration will greatly benefit
>>> this
>>>>>> aspect. Airavata Registry metadata cataloging is very primitive. And
>>>>>> Rich
>>>>>> data management capabilities of OODT should nicely enhance airavata
>>>>>> provenance aware executions.
>>>>>>
>>>>>> Please let me know which of the above aspects you are interested in
>>> and
>>>>>> we
>>>>>> can discuss further.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> [1] -
>>>>>>
>>>>>>
>>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>>>>>> tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>>>>>> [2] - http://gradworks.umi.com/3297094.pdf
>>>>>> [3] - http://d2i.indiana.edu/provenance_karma
>>>>>> [4] -
>>>>>>
>>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>>>>>> [5] -
>>>>>>
>>>>>>
>>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>>>>>> 38325269&CFTOKEN=73364358
>>>>>> [6] - http://dl.acm.org/citation.cfm?id=2359844
>>>>>>
>>>>>>
>>>>>> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sanjaya,
>>>>>>> I am seeing some traffic floating around RE: Airavata OODT
>>> integration
>>>>>> but I am not clued up.
>>>>>>> Can you point me to the code? The issues? The mailing list threads?
>>>>>>> I am also extremely interested in provenance tracking of
>>> (record???)
>>>>>> data which passes through OODT. I would like to expose this as some
>>>>>> stage
>>>>>> to people who access the data.
>>>>>>> Thank you so much.
>>>>>>>
>>>>>>> On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>> I am trying to find out ways to improve Airavata provenance aware
>>>>>> workflow
>>>>>>>> processing based on rich metadata available in Apache OODT. As
>>> part
>>>>>> of
>>>>>> this
>>>>>>>> task, I am investigating how Apache Airavata is currently
>>> performing
>>>>>>>> provenance aware workflow processing. I have looked into the
>>>>>> provenance
>>>>>>>> manager implementation, where I could see lot of data related to
>>>>>> workflow
>>>>>>>> execution is saved in registry. But I could see very little of
>>> use
>>>>>> of
>>>>>> those
>>>>>>>> data. Could some one please explain the details of provenance
>>> aware
>>>>>>>> processing in Airavata and any future plans on this area?
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Sanjaya
>>>>>>>>
>>>>>>> --
>>>>>>> Lewis
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Suresh,
I have further looked into the code to understand, how provenance aware
workflow processing has been implemented in current code base. As you have
stated, it seems functionality is broken. In WorkflowInterpreter class,
there is a method called readProvenance (only place where ProvenanceReader
is used) . It is a private method, no longer a part of the actual
execution. I think, it's better to go back to version where provenance
aware workflow processing working as expected to get better understand on
the expected flow. Then fix the issues in current version. Could you please
let me know, possible version that this particular functionality worked as
expected ?
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Suresh! I'll do further investigations and let the team know my
> findings.
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Sanjaya,
>>
>> XBaya and Workflow Interpreter went through some changes which probably
>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>> AFAIK, the feature did not get deprecated it should be one of those cases
>> which was not actively tested during releases and fell through the cracks.
>> Lets see if any one gets motivated to debug and fix this issue. And as
>> always, we welcome your patches. Sorry about pointing to a unverified
>> feature.
>>
>> Suresh
>>
>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com>
>> wrote:
>>
>> > Hi Suresh,
>> > I have gone through the research paper in item 6 and try to match it to
>> the current Airavata implementation. I can see that Airavata collects
>> provenance data and stored them in a registry. I couldn't able to find out
>> a place where these data has been actually used to improve the workflow
>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>> It seems this particular configuration is no longer used. Since provenance
>> aware data is already available in registry, is it the responsibility of
>> the application to use these data to improve workflow processing based on
>> provenance aware data ?
>> >
>> > Thanks
>> > Sanjaya
>> >
>> >
>> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
>> wrote:
>> > Hey Sanjaya,
>> >
>> > You may want to check out the Traceable Product product type policy
>> > [1] as well as OODT-139 [2] which includes the PCS services that serve
>> > up the information and provenance.
>> >
>> > Cheers,
>> > Chris
>> >
>> > [1]
>> >
>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>> > les/trace/
>> > [2] http://issues.apache.org/jira/browse/OODT-139
>> >
>> >
>> > -----Original Message-----
>> > From: Sanjaya Medonsa <sa...@gmail.com>
>> > Reply-To: <de...@oodt.apache.org>
>> > Date: Friday, August 23, 2013 6:30 PM
>> > To: Airavata Dev <de...@airavata.apache.org>
>> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > Subject: Re: Povenance Aware Workflow Processing
>> >
>> > >Thanks Suresh! I am interested in Provenance aware workflow processing
>> > >where OODT integration could help greatly with rich metadata available.
>> > >I'll go through the item 6 and let you know my understanding.
>> > >
>> > >Sanjaya
>> > >
>> > >
>> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org>
>> wrote:
>> > >
>> > >> Hi Sanjaya, Lewis,
>> > >>
>> > >> Appreciate your interest on this topic, this is a very good question
>> and
>> > >> sorry for the delay in responding, these days I feel I am always in
>> > >>catch
>> > >> up mode.
>> > >>
>> > >> Let me separate this discussion into four parts: Provenance
>> Gathering,
>> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>> executions.
>> > >>
>> > >> Gathering: Airavata is instrumented with rich provenance centric
>> > >> information model which captures each step and publishes to an event
>> > >>bus.
>> > >> This is encoded in a "workflow tracking" schema [1]. So the current
>> > >>trunk
>> > >> of Airavata publishes provenance rich information but not all is
>> > >>guaranteed
>> > >> to be captured in the registry.
>> > >>
>> > >> Analyzing: Significant work on provenance aspect related to Airavata
>> is
>> > >>a
>> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
>> > >>research
>> > >> asst. professor at USC. His thesis [2] is a good source of reference
>> on
>> > >> this topic. Both the research and tools are in active development by
>> > >>Prof.
>> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>> > >>uses
>> > >> the same workflow tracking schema and event bus, the tools should be
>> > >>easily
>> > >> integratable but did not have a motivating use case to do so.
>> > >>
>> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> > >> version) and Karma framework which provides context to the gathered
>> > >> provenance. Couple of papers on this topic [4], [5].
>> > >>
>> > >> Executions: This is an active topic within Airavata. Current release
>> of
>> > >> Airavata supports Provenance aware workflow executions and is
>> discussed
>> > >>in
>> > >> [6]. I think the OODT and Airavata integration will greatly benefit
>> this
>> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
>> > >>Rich
>> > >> data management capabilities of OODT should nicely enhance airavata
>> > >> provenance aware executions.
>> > >>
>> > >> Please let me know which of the above aspects you are interested in
>> and
>> > >>we
>> > >> can discuss further.
>> > >>
>> > >> Suresh
>> > >>
>> > >> [1] -
>> > >>
>> > >>
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> > >> [2] - http://gradworks.umi.com/3297094.pdf
>> > >> [3] - http://d2i.indiana.edu/provenance_karma
>> > >> [4] -
>> > >>
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> > >> [5] -
>> > >>
>> > >>
>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>> > >>38325269&CFTOKEN=73364358
>> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
>> > >>
>> > >>
>> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> > >> lewis.mcgibbney@gmail.com> wrote:
>> > >>
>> > >> > Hi Sanjaya,
>> > >> > I am seeing some traffic floating around RE: Airavata OODT
>> integration
>> > >> but I am not clued up.
>> > >> > Can you point me to the code? The issues? The mailing list threads?
>> > >> > I am also extremely interested in provenance tracking of
>> (record???)
>> > >> data which passes through OODT. I would like to expose this as some
>> > >>stage
>> > >> to people who access the data.
>> > >> > Thank you so much.
>> > >> >
>> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>> > >>wrote:
>> > >> > > Hi,
>> > >> > > I am trying to find out ways to improve Airavata provenance aware
>> > >> workflow
>> > >> > > processing based on rich metadata available in Apache OODT. As
>> part
>> > >>of
>> > >> this
>> > >> > > task, I am investigating how Apache Airavata is currently
>> performing
>> > >> > > provenance aware workflow processing. I have looked into the
>> > >>provenance
>> > >> > > manager implementation, where I could see lot of data related to
>> > >> workflow
>> > >> > > execution is saved in registry. But I could see very little of
>> use
>> > >>of
>> > >> those
>> > >> > > data. Could some one please explain the details of provenance
>> aware
>> > >> > > processing in Airavata and any future plans on this area?
>> > >> > >
>> > >> > > Best Regards,
>> > >> > > Sanjaya
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Lewis
>> > >> >
>> > >>
>> > >>
>> >
>> >
>> >
>>
>>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Hi Suresh,
I have further looked into the code to understand, how provenance aware
workflow processing has been implemented in current code base. As you have
stated, it seems functionality is broken. In WorkflowInterpreter class,
there is a method called readProvenance (only place where ProvenanceReader
is used) . It is a private method, no longer a part of the actual
execution. I think, it's better to go back to version where provenance
aware workflow processing working as expected to get better understand on
the expected flow. Then fix the issues in current version. Could you please
let me know, possible version that this particular functionality worked as
expected ?
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sa...@gmail.com>wrote:
> Thanks Suresh! I'll do further investigations and let the team know my
> findings.
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Sanjaya,
>>
>> XBaya and Workflow Interpreter went through some changes which probably
>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>> AFAIK, the feature did not get deprecated it should be one of those cases
>> which was not actively tested during releases and fell through the cracks.
>> Lets see if any one gets motivated to debug and fix this issue. And as
>> always, we welcome your patches. Sorry about pointing to a unverified
>> feature.
>>
>> Suresh
>>
>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com>
>> wrote:
>>
>> > Hi Suresh,
>> > I have gone through the research paper in item 6 and try to match it to
>> the current Airavata implementation. I can see that Airavata collects
>> provenance data and stored them in a registry. I couldn't able to find out
>> a place where these data has been actually used to improve the workflow
>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>> It seems this particular configuration is no longer used. Since provenance
>> aware data is already available in registry, is it the responsibility of
>> the application to use these data to improve workflow processing based on
>> provenance aware data ?
>> >
>> > Thanks
>> > Sanjaya
>> >
>> >
>> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
>> wrote:
>> > Hey Sanjaya,
>> >
>> > You may want to check out the Traceable Product product type policy
>> > [1] as well as OODT-139 [2] which includes the PCS services that serve
>> > up the information and provenance.
>> >
>> > Cheers,
>> > Chris
>> >
>> > [1]
>> >
>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>> > les/trace/
>> > [2] http://issues.apache.org/jira/browse/OODT-139
>> >
>> >
>> > -----Original Message-----
>> > From: Sanjaya Medonsa <sa...@gmail.com>
>> > Reply-To: <de...@oodt.apache.org>
>> > Date: Friday, August 23, 2013 6:30 PM
>> > To: Airavata Dev <de...@airavata.apache.org>
>> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > Subject: Re: Povenance Aware Workflow Processing
>> >
>> > >Thanks Suresh! I am interested in Provenance aware workflow processing
>> > >where OODT integration could help greatly with rich metadata available.
>> > >I'll go through the item 6 and let you know my understanding.
>> > >
>> > >Sanjaya
>> > >
>> > >
>> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org>
>> wrote:
>> > >
>> > >> Hi Sanjaya, Lewis,
>> > >>
>> > >> Appreciate your interest on this topic, this is a very good question
>> and
>> > >> sorry for the delay in responding, these days I feel I am always in
>> > >>catch
>> > >> up mode.
>> > >>
>> > >> Let me separate this discussion into four parts: Provenance
>> Gathering,
>> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>> executions.
>> > >>
>> > >> Gathering: Airavata is instrumented with rich provenance centric
>> > >> information model which captures each step and publishes to an event
>> > >>bus.
>> > >> This is encoded in a "workflow tracking" schema [1]. So the current
>> > >>trunk
>> > >> of Airavata publishes provenance rich information but not all is
>> > >>guaranteed
>> > >> to be captured in the registry.
>> > >>
>> > >> Analyzing: Significant work on provenance aspect related to Airavata
>> is
>> > >>a
>> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
>> > >>research
>> > >> asst. professor at USC. His thesis [2] is a good source of reference
>> on
>> > >> this topic. Both the research and tools are in active development by
>> > >>Prof.
>> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>> > >>uses
>> > >> the same workflow tracking schema and event bus, the tools should be
>> > >>easily
>> > >> integratable but did not have a motivating use case to do so.
>> > >>
>> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> > >> version) and Karma framework which provides context to the gathered
>> > >> provenance. Couple of papers on this topic [4], [5].
>> > >>
>> > >> Executions: This is an active topic within Airavata. Current release
>> of
>> > >> Airavata supports Provenance aware workflow executions and is
>> discussed
>> > >>in
>> > >> [6]. I think the OODT and Airavata integration will greatly benefit
>> this
>> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
>> > >>Rich
>> > >> data management capabilities of OODT should nicely enhance airavata
>> > >> provenance aware executions.
>> > >>
>> > >> Please let me know which of the above aspects you are interested in
>> and
>> > >>we
>> > >> can discuss further.
>> > >>
>> > >> Suresh
>> > >>
>> > >> [1] -
>> > >>
>> > >>
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> > >> [2] - http://gradworks.umi.com/3297094.pdf
>> > >> [3] - http://d2i.indiana.edu/provenance_karma
>> > >> [4] -
>> > >>
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> > >> [5] -
>> > >>
>> > >>
>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>> > >>38325269&CFTOKEN=73364358
>> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
>> > >>
>> > >>
>> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> > >> lewis.mcgibbney@gmail.com> wrote:
>> > >>
>> > >> > Hi Sanjaya,
>> > >> > I am seeing some traffic floating around RE: Airavata OODT
>> integration
>> > >> but I am not clued up.
>> > >> > Can you point me to the code? The issues? The mailing list threads?
>> > >> > I am also extremely interested in provenance tracking of
>> (record???)
>> > >> data which passes through OODT. I would like to expose this as some
>> > >>stage
>> > >> to people who access the data.
>> > >> > Thank you so much.
>> > >> >
>> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
>> > >>wrote:
>> > >> > > Hi,
>> > >> > > I am trying to find out ways to improve Airavata provenance aware
>> > >> workflow
>> > >> > > processing based on rich metadata available in Apache OODT. As
>> part
>> > >>of
>> > >> this
>> > >> > > task, I am investigating how Apache Airavata is currently
>> performing
>> > >> > > provenance aware workflow processing. I have looked into the
>> > >>provenance
>> > >> > > manager implementation, where I could see lot of data related to
>> > >> workflow
>> > >> > > execution is saved in registry. But I could see very little of
>> use
>> > >>of
>> > >> those
>> > >> > > data. Could some one please explain the details of provenance
>> aware
>> > >> > > processing in Airavata and any future plans on this area?
>> > >> > >
>> > >> > > Best Regards,
>> > >> > > Sanjaya
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Lewis
>> > >> >
>> > >>
>> > >>
>> >
>> >
>> >
>>
>>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh! I'll do further investigations and let the team know my
findings.
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya,
>
> XBaya and Workflow Interpreter went through some changes which probably
> dropped the condition which makes use of the enableProvenanceSmartRun flag.
> AFAIK, the feature did not get deprecated it should be one of those cases
> which was not actively tested during releases and fell through the cracks.
> Lets see if any one gets motivated to debug and fix this issue. And as
> always, we welcome your patches. Sorry about pointing to a unverified
> feature.
>
> Suresh
>
> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
>
> > Hi Suresh,
> > I have gone through the research paper in item 6 and try to match it to
> the current Airavata implementation. I can see that Airavata collects
> provenance data and stored them in a registry. I couldn't able to find out
> a place where these data has been actually used to improve the workflow
> processing. Xbaya can be configured with option enableProvenanceSmartRun.
> It seems this particular configuration is no longer used. Since provenance
> aware data is already available in registry, is it the responsibility of
> the application to use these data to improve workflow processing based on
> provenance aware data ?
> >
> > Thanks
> > Sanjaya
> >
> >
> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
> wrote:
> > Hey Sanjaya,
> >
> > You may want to check out the Traceable Product product type policy
> > [1] as well as OODT-139 [2] which includes the PCS services that serve
> > up the information and provenance.
> >
> > Cheers,
> > Chris
> >
> > [1]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> > les/trace/
> > [2] http://issues.apache.org/jira/browse/OODT-139
> >
> >
> > -----Original Message-----
> > From: Sanjaya Medonsa <sa...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Friday, August 23, 2013 6:30 PM
> > To: Airavata Dev <de...@airavata.apache.org>
> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: Re: Povenance Aware Workflow Processing
> >
> > >Thanks Suresh! I am interested in Provenance aware workflow processing
> > >where OODT integration could help greatly with rich metadata available.
> > >I'll go through the item 6 and let you know my understanding.
> > >
> > >Sanjaya
> > >
> > >
> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> > >
> > >> Hi Sanjaya, Lewis,
> > >>
> > >> Appreciate your interest on this topic, this is a very good question
> and
> > >> sorry for the delay in responding, these days I feel I am always in
> > >>catch
> > >> up mode.
> > >>
> > >> Let me separate this discussion into four parts: Provenance Gathering,
> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
> executions.
> > >>
> > >> Gathering: Airavata is instrumented with rich provenance centric
> > >> information model which captures each step and publishes to an event
> > >>bus.
> > >> This is encoded in a "workflow tracking" schema [1]. So the current
> > >>trunk
> > >> of Airavata publishes provenance rich information but not all is
> > >>guaranteed
> > >> to be captured in the registry.
> > >>
> > >> Analyzing: Significant work on provenance aspect related to Airavata
> is
> > >>a
> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> > >>research
> > >> asst. professor at USC. His thesis [2] is a good source of reference
> on
> > >> this topic. Both the research and tools are in active development by
> > >>Prof.
> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> > >>uses
> > >> the same workflow tracking schema and event bus, the tools should be
> > >>easily
> > >> integratable but did not have a motivating use case to do so.
> > >>
> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> > >> version) and Karma framework which provides context to the gathered
> > >> provenance. Couple of papers on this topic [4], [5].
> > >>
> > >> Executions: This is an active topic within Airavata. Current release
> of
> > >> Airavata supports Provenance aware workflow executions and is
> discussed
> > >>in
> > >> [6]. I think the OODT and Airavata integration will greatly benefit
> this
> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
> > >>Rich
> > >> data management capabilities of OODT should nicely enhance airavata
> > >> provenance aware executions.
> > >>
> > >> Please let me know which of the above aspects you are interested in
> and
> > >>we
> > >> can discuss further.
> > >>
> > >> Suresh
> > >>
> > >> [1] -
> > >>
> > >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> > >> [2] - http://gradworks.umi.com/3297094.pdf
> > >> [3] - http://d2i.indiana.edu/provenance_karma
> > >> [4] -
> > >>
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> > >> [5] -
> > >>
> > >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> > >>38325269&CFTOKEN=73364358
> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> > >>
> > >>
> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> > >> lewis.mcgibbney@gmail.com> wrote:
> > >>
> > >> > Hi Sanjaya,
> > >> > I am seeing some traffic floating around RE: Airavata OODT
> integration
> > >> but I am not clued up.
> > >> > Can you point me to the code? The issues? The mailing list threads?
> > >> > I am also extremely interested in provenance tracking of (record???)
> > >> data which passes through OODT. I would like to expose this as some
> > >>stage
> > >> to people who access the data.
> > >> > Thank you so much.
> > >> >
> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> > >>wrote:
> > >> > > Hi,
> > >> > > I am trying to find out ways to improve Airavata provenance aware
> > >> workflow
> > >> > > processing based on rich metadata available in Apache OODT. As
> part
> > >>of
> > >> this
> > >> > > task, I am investigating how Apache Airavata is currently
> performing
> > >> > > provenance aware workflow processing. I have looked into the
> > >>provenance
> > >> > > manager implementation, where I could see lot of data related to
> > >> workflow
> > >> > > execution is saved in registry. But I could see very little of use
> > >>of
> > >> those
> > >> > > data. Could some one please explain the details of provenance
> aware
> > >> > > processing in Airavata and any future plans on this area?
> > >> > >
> > >> > > Best Regards,
> > >> > > Sanjaya
> > >> > >
> > >> >
> > >> > --
> > >> > Lewis
> > >> >
> > >>
> > >>
> >
> >
> >
>
>
Re: Povenance Aware Workflow Processing
Posted by Sanjaya Medonsa <sa...@gmail.com>.
Thanks Suresh! I'll do further investigations and let the team know my
findings.
Best Regards,
Sanjaya
On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Sanjaya,
>
> XBaya and Workflow Interpreter went through some changes which probably
> dropped the condition which makes use of the enableProvenanceSmartRun flag.
> AFAIK, the feature did not get deprecated it should be one of those cases
> which was not actively tested during releases and fell through the cracks.
> Lets see if any one gets motivated to debug and fix this issue. And as
> always, we welcome your patches. Sorry about pointing to a unverified
> feature.
>
> Suresh
>
> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
>
> > Hi Suresh,
> > I have gone through the research paper in item 6 and try to match it to
> the current Airavata implementation. I can see that Airavata collects
> provenance data and stored them in a registry. I couldn't able to find out
> a place where these data has been actually used to improve the workflow
> processing. Xbaya can be configured with option enableProvenanceSmartRun.
> It seems this particular configuration is no longer used. Since provenance
> aware data is already available in registry, is it the responsibility of
> the application to use these data to improve workflow processing based on
> provenance aware data ?
> >
> > Thanks
> > Sanjaya
> >
> >
> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org>
> wrote:
> > Hey Sanjaya,
> >
> > You may want to check out the Traceable Product product type policy
> > [1] as well as OODT-139 [2] which includes the PCS services that serve
> > up the information and provenance.
> >
> > Cheers,
> > Chris
> >
> > [1]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> > les/trace/
> > [2] http://issues.apache.org/jira/browse/OODT-139
> >
> >
> > -----Original Message-----
> > From: Sanjaya Medonsa <sa...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Friday, August 23, 2013 6:30 PM
> > To: Airavata Dev <de...@airavata.apache.org>
> > Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: Re: Povenance Aware Workflow Processing
> >
> > >Thanks Suresh! I am interested in Provenance aware workflow processing
> > >where OODT integration could help greatly with rich metadata available.
> > >I'll go through the item 6 and let you know my understanding.
> > >
> > >Sanjaya
> > >
> > >
> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> > >
> > >> Hi Sanjaya, Lewis,
> > >>
> > >> Appreciate your interest on this topic, this is a very good question
> and
> > >> sorry for the delay in responding, these days I feel I am always in
> > >>catch
> > >> up mode.
> > >>
> > >> Let me separate this discussion into four parts: Provenance Gathering,
> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
> executions.
> > >>
> > >> Gathering: Airavata is instrumented with rich provenance centric
> > >> information model which captures each step and publishes to an event
> > >>bus.
> > >> This is encoded in a "workflow tracking" schema [1]. So the current
> > >>trunk
> > >> of Airavata publishes provenance rich information but not all is
> > >>guaranteed
> > >> to be captured in the registry.
> > >>
> > >> Analyzing: Significant work on provenance aspect related to Airavata
> is
> > >>a
> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> > >>research
> > >> asst. professor at USC. His thesis [2] is a good source of reference
> on
> > >> this topic. Both the research and tools are in active development by
> > >>Prof.
> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> > >>uses
> > >> the same workflow tracking schema and event bus, the tools should be
> > >>easily
> > >> integratable but did not have a motivating use case to do so.
> > >>
> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> > >> version) and Karma framework which provides context to the gathered
> > >> provenance. Couple of papers on this topic [4], [5].
> > >>
> > >> Executions: This is an active topic within Airavata. Current release
> of
> > >> Airavata supports Provenance aware workflow executions and is
> discussed
> > >>in
> > >> [6]. I think the OODT and Airavata integration will greatly benefit
> this
> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
> > >>Rich
> > >> data management capabilities of OODT should nicely enhance airavata
> > >> provenance aware executions.
> > >>
> > >> Please let me know which of the above aspects you are interested in
> and
> > >>we
> > >> can discuss further.
> > >>
> > >> Suresh
> > >>
> > >> [1] -
> > >>
> > >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> > >> [2] - http://gradworks.umi.com/3297094.pdf
> > >> [3] - http://d2i.indiana.edu/provenance_karma
> > >> [4] -
> > >>
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> > >> [5] -
> > >>
> > >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> > >>38325269&CFTOKEN=73364358
> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> > >>
> > >>
> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> > >> lewis.mcgibbney@gmail.com> wrote:
> > >>
> > >> > Hi Sanjaya,
> > >> > I am seeing some traffic floating around RE: Airavata OODT
> integration
> > >> but I am not clued up.
> > >> > Can you point me to the code? The issues? The mailing list threads?
> > >> > I am also extremely interested in provenance tracking of (record???)
> > >> data which passes through OODT. I would like to expose this as some
> > >>stage
> > >> to people who access the data.
> > >> > Thank you so much.
> > >> >
> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> > >>wrote:
> > >> > > Hi,
> > >> > > I am trying to find out ways to improve Airavata provenance aware
> > >> workflow
> > >> > > processing based on rich metadata available in Apache OODT. As
> part
> > >>of
> > >> this
> > >> > > task, I am investigating how Apache Airavata is currently
> performing
> > >> > > provenance aware workflow processing. I have looked into the
> > >>provenance
> > >> > > manager implementation, where I could see lot of data related to
> > >> workflow
> > >> > > execution is saved in registry. But I could see very little of use
> > >>of
> > >> those
> > >> > > data. Could some one please explain the details of provenance
> aware
> > >> > > processing in Airavata and any future plans on this area?
> > >> > >
> > >> > > Best Regards,
> > >> > > Sanjaya
> > >> > >
> > >> >
> > >> > --
> > >> > Lewis
> > >> >
> > >>
> > >>
> >
> >
> >
>
>
Re: Povenance Aware Workflow Processing
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya,
XBaya and Workflow Interpreter went through some changes which probably dropped the condition which makes use of the enableProvenanceSmartRun flag. AFAIK, the feature did not get deprecated it should be one of those cases which was not actively tested during releases and fell through the cracks. Lets see if any one gets motivated to debug and fix this issue. And as always, we welcome your patches. Sorry about pointing to a unverified feature.
Suresh
On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Hi Suresh,
> I have gone through the research paper in item 6 and try to match it to the current Airavata implementation. I can see that Airavata collects provenance data and stored them in a registry. I couldn't able to find out a place where these data has been actually used to improve the workflow processing. Xbaya can be configured with option enableProvenanceSmartRun. It seems this particular configuration is no longer used. Since provenance aware data is already available in registry, is it the responsibility of the application to use these data to improve workflow processing based on provenance aware data ?
>
> Thanks
> Sanjaya
>
>
> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>
Re: Povenance Aware Workflow Processing
Posted by Suresh Marru <sm...@apache.org>.
Hi Sanjaya,
XBaya and Workflow Interpreter went through some changes which probably dropped the condition which makes use of the enableProvenanceSmartRun flag. AFAIK, the feature did not get deprecated it should be one of those cases which was not actively tested during releases and fell through the cracks. Lets see if any one gets motivated to debug and fix this issue. And as always, we welcome your patches. Sorry about pointing to a unverified feature.
Suresh
On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sa...@gmail.com> wrote:
> Hi Suresh,
> I have gone through the research paper in item 6 and try to match it to the current Airavata implementation. I can see that Airavata collects provenance data and stored them in a registry. I couldn't able to find out a place where these data has been actually used to improve the workflow processing. Xbaya can be configured with option enableProvenanceSmartRun. It seems this particular configuration is no longer used. Since provenance aware data is already available in registry, is it the responsibility of the application to use these data to improve workflow processing based on provenance aware data ?
>
> Thanks
> Sanjaya
>
>
> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <ma...@apache.org> wrote:
> Hey Sanjaya,
>
> You may want to check out the Traceable Product product type policy
> [1] as well as OODT-139 [2] which includes the PCS services that serve
> up the information and provenance.
>
> Cheers,
> Chris
>
> [1]
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> les/trace/
> [2] http://issues.apache.org/jira/browse/OODT-139
>
>
> -----Original Message-----
> From: Sanjaya Medonsa <sa...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Friday, August 23, 2013 6:30 PM
> To: Airavata Dev <de...@airavata.apache.org>
> Cc: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Povenance Aware Workflow Processing
>
> >Thanks Suresh! I am interested in Provenance aware workflow processing
> >where OODT integration could help greatly with rich metadata available.
> >I'll go through the item 6 and let you know my understanding.
> >
> >Sanjaya
> >
> >
> >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> Hi Sanjaya, Lewis,
> >>
> >> Appreciate your interest on this topic, this is a very good question and
> >> sorry for the delay in responding, these days I feel I am always in
> >>catch
> >> up mode.
> >>
> >> Let me separate this discussion into four parts: Provenance Gathering,
> >> Provenance Analyzing, Provenance Reasoning, Provenance Aware executions.
> >>
> >> Gathering: Airavata is instrumented with rich provenance centric
> >> information model which captures each step and publishes to an event
> >>bus.
> >> This is encoded in a "workflow tracking" schema [1]. So the current
> >>trunk
> >> of Airavata publishes provenance rich information but not all is
> >>guaranteed
> >> to be captured in the registry.
> >>
> >> Analyzing: Significant work on provenance aspect related to Airavata is
> >>a
> >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> >>research
> >> asst. professor at USC. His thesis [2] is a good source of reference on
> >> this topic. Both the research and tools are in active development by
> >>Prof.
> >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> >>uses
> >> the same workflow tracking schema and event bus, the tools should be
> >>easily
> >> integratable but did not have a motivating use case to do so.
> >>
> >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> >> version) and Karma framework which provides context to the gathered
> >> provenance. Couple of papers on this topic [4], [5].
> >>
> >> Executions: This is an active topic within Airavata. Current release of
> >> Airavata supports Provenance aware workflow executions and is discussed
> >>in
> >> [6]. I think the OODT and Airavata integration will greatly benefit this
> >> aspect. Airavata Registry metadata cataloging is very primitive. And
> >>Rich
> >> data management capabilities of OODT should nicely enhance airavata
> >> provenance aware executions.
> >>
> >> Please let me know which of the above aspects you are interested in and
> >>we
> >> can discuss further.
> >>
> >> Suresh
> >>
> >> [1] -
> >>
> >>https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >> [2] - http://gradworks.umi.com/3297094.pdf
> >> [3] - http://d2i.indiana.edu/provenance_karma
> >> [4] -
> >> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> >> [5] -
> >>
> >>http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> >>38325269&CFTOKEN=73364358
> >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> >>
> >>
> >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> >> lewis.mcgibbney@gmail.com> wrote:
> >>
> >> > Hi Sanjaya,
> >> > I am seeing some traffic floating around RE: Airavata OODT integration
> >> but I am not clued up.
> >> > Can you point me to the code? The issues? The mailing list threads?
> >> > I am also extremely interested in provenance tracking of (record???)
> >> data which passes through OODT. I would like to expose this as some
> >>stage
> >> to people who access the data.
> >> > Thank you so much.
> >> >
> >> > On Monday, July 22, 2013, Sanjaya Medonsa <sa...@gmail.com>
> >>wrote:
> >> > > Hi,
> >> > > I am trying to find out ways to improve Airavata provenance aware
> >> workflow
> >> > > processing based on rich metadata available in Apache OODT. As part
> >>of
> >> this
> >> > > task, I am investigating how Apache Airavata is currently performing
> >> > > provenance aware workflow processing. I have looked into the
> >>provenance
> >> > > manager implementation, where I could see lot of data related to
> >> workflow
> >> > > execution is saved in registry. But I could see very little of use
> >>of
> >> those
> >> > > data. Could some one please explain the details of provenance aware
> >> > > processing in Airavata and any future plans on this area?
> >> > >
> >> > > Best Regards,
> >> > > Sanjaya
> >> > >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
>
>
>