Integrating Custom Workflow Interpreter

[Bendy Lal <be...@gmail.com>]

Hi,

I am Bendy, newbie for Apache Airavata - find it very interesting.

I am looking for some guidance on how to Integrate a custom Workflow
Interpreter something similar to "XBaya Interpretter Service" with XBAYA
GUI to leverage designer, interaction and monitoring support.

I'm sure this would have been attempted before as it seems to be in-line
with the vision for Airavata/ABaya to be agnostic of underlying workflow
runtimes.

Appreciate your support.

Thanks,
Bendy.

Re: Integrating Custom Workflow Interpreter

[Suresh Marru <sm...@apache.org>]

Hi Bendy,

Airavata originated from an academic research lab (Prof. Dennis Gannon’s group at Indiana University) to meet the goals of a dynamically adaptive system [1]. Alongside, we were working with the community towards workflow interoperability with XBaya as one of the reference implementation. So, you are right in the sense that Airavata is designed so components like Workflow Interpreter could be swapped. A schematic of pluggable workflow engines is attached. 

As an Apache community, Airavata is more driven by the user needs which have been more in managing long running applications on diverse computational resources. We have been focused mostly on rest of the Airavata (including making the system multi-tentant and platform as a service ready) and just starting to shift our attention back to workflows. We very much would like to see both Airavata as a robust production ready software and a parallel exploratory laboratory. What you are attempting to do is doable with the following gotcha’s. 

* Airavata exposes an API to clients and GUI’s (XBaya is one such GUI, but other web based examples and reference implementations exist). 
* Internally each of Airavata’s components expose a Component Provider Interface or CPI. These are described at an abstract level in paper [2].
* We can work with you in fleshing out the Workflow Engine CPI (it needs work, especially to craft dynamism calls). You can then implement the API with your custom workflow engine replacing Airavata’s workflow interpreter. 
* Airavata messaging system was recently moved from a WS-Eventing framework to a AMQP (RabbitMQ implementation). So the messaging instrumentation needs to be worked out as well.

How does your interpreter specify workflow definition? There was a long discussion on this topic recently, you might find some relevance at [3].

So yes, you are right that theoretically everything you say is plausible, but making it reality will take a small effort. If you are motivated enough, Airavata community will offer guidance. Ofcourse if you make any contributions to Airavata, you will be meritocratically rewarded :)

Suresh 

[1] - http://link.springer.com/chapter/10.1007/978-1-84628-757-2_9
[2] - http://goo.gl/QZXNJZ
[3] - http://markmail.org/thread/tkpbj3sr4jhg6o6z



On Jan 22, 2015, at 5:34 PM, Bendy Lal <be...@gmail.com> wrote:

> Hi Raminder,
> 
> Thanks for the quick response.
> 
> We have a custom home grown workflow engine and have flexibility on it's APIs. It supports dynamic interpreter based enactment similar to my understanding of "XBaya Interpreter Service". This is engine with it own Workflow language, Activities, validations, rich expression language, testing, etc that we are not looking to swap with other workflow engines - for the near future.
> 
> As it's an dynamic/interpreter based workflow, it has well defined interaction and state transition points. So there is API for Start, Stop, Pause, Resume, etc that seems to align well with Airavata/XBaya. The activity/step completion or any completion/error or start of another state in the workflow DAG can be posted to the Queue that XBaya listens - so it seems to me that the integration should be straight forward (at least theoretically :-) )  
> 
> If I can get to a reference to a sample workflow engine integration with XBaya, I can model the integration with that and share my findings with this group.
> 
> What would be a good starting point.
> 
> Thanks!
> Bendy
> 
> 
> On Thu, Jan 22, 2015 at 11:25 AM, Raminder Singh <ra...@gmail.com> wrote:
> Hi Bendy, 
> 
> Welcome to Airavata!! I am glad you find it interesting. We have done integration with workflow engines like Apache ODE (WS-BPEL) or Taverna in the past etc. We are not actively monitoring those implementation for various reasons. Challenge in such integration were, API’s to interact with engines, to monitor/manage the workflow and use of different version of workflow engine in different projects. Please let us know which engine you are thinking about and what type of API’s it provide. 
> 
> We have developed workflow interpreter to server current usecases in Airavata and plan to enhance it for other usecases. If you describe your usecases then it will be easy to guide you if inbuilt workflow engine can solve the problem. 
> 
> Thanks for your interest.
> Raminder
> 
> On Jan 22, 2015, at 1:49 PM, Bendy Lal <be...@gmail.com> wrote:
> 
>> Hi,
>> 
>> I am Bendy, newbie for Apache Airavata - find it very interesting.
>> 
>> I am looking for some guidance on how to Integrate a custom Workflow Interpreter something similar to "XBaya Interpretter Service" with XBAYA GUI to leverage designer, interaction and monitoring support.
>> 
>> I'm sure this would have been attempted before as it seems to be in-line with the vision for Airavata/ABaya to be agnostic of underlying workflow runtimes.
>> 
>> Appreciate your support.
>> 
>> Thanks,
>> Bendy.
> 
> 

Re: Integrating Custom Workflow Interpreter

[Bendy Lal <be...@gmail.com>]

Hi Raminder,

Thanks for the quick response.

We have a custom home grown workflow engine and have flexibility on it's
APIs. It supports dynamic interpreter based enactment similar to my
understanding of "XBaya Interpreter Service". This is engine with it own
Workflow language, Activities, validations, rich expression language,
testing, etc that we are not looking to swap with other workflow engines -
for the near future.

As it's an dynamic/interpreter based workflow, it has well defined
interaction and state transition points. So there is API for Start, Stop,
Pause, Resume, etc that seems to align well with Airavata/XBaya. The
activity/step completion or any completion/error or start of another state
in the workflow DAG can be posted to the Queue that XBaya listens - so it
seems to me that the integration should be straight forward (at least
theoretically :-) )

If I can get to a reference to a sample workflow engine integration with
XBaya, I can model the integration with that and share my findings with
this group.

What would be a good starting point.

Thanks!
Bendy


On Thu, Jan 22, 2015 at 11:25 AM, Raminder Singh <ra...@gmail.com>
wrote:

> Hi Bendy,
>
> Welcome to Airavata!! I am glad you find it interesting. We have done
> integration with workflow engines like Apache ODE (WS-BPEL) or Taverna in
> the past etc. We are not actively monitoring those implementation for
> various reasons. Challenge in such integration were, API’s to interact with
> engines, to monitor/manage the workflow and use of different version of
> workflow engine in different projects. Please let us know which engine you
> are thinking about and what type of API’s it provide.
>
> We have developed workflow interpreter to server current usecases in
> Airavata and plan to enhance it for other usecases. If you describe your
> usecases then it will be easy to guide you if inbuilt workflow engine can
> solve the problem.
>
> Thanks for your interest.
> Raminder
>
> On Jan 22, 2015, at 1:49 PM, Bendy Lal <be...@gmail.com> wrote:
>
> Hi,
>
> I am Bendy, newbie for Apache Airavata - find it very interesting.
>
> I am looking for some guidance on how to Integrate a custom Workflow
> Interpreter something similar to "XBaya Interpretter Service" with XBAYA
> GUI to leverage designer, interaction and monitoring support.
>
> I'm sure this would have been attempted before as it seems to be in-line
> with the vision for Airavata/ABaya to be agnostic of underlying workflow
> runtimes.
>
> Appreciate your support.
>
> Thanks,
> Bendy.
>
>
>

Re: Integrating Custom Workflow Interpreter

[Raminder Singh <ra...@gmail.com>]

Hi Bendy, 

Welcome to Airavata!! I am glad you find it interesting. We have done integration with workflow engines like Apache ODE (WS-BPEL) or Taverna in the past etc. We are not actively monitoring those implementation for various reasons. Challenge in such integration were, API’s to interact with engines, to monitor/manage the workflow and use of different version of workflow engine in different projects. Please let us know which engine you are thinking about and what type of API’s it provide. 

We have developed workflow interpreter to server current usecases in Airavata and plan to enhance it for other usecases. If you describe your usecases then it will be easy to guide you if inbuilt workflow engine can solve the problem. 

Thanks for your interest.
Raminder

On Jan 22, 2015, at 1:49 PM, Bendy Lal <be...@gmail.com> wrote:

> Hi,
> 
> I am Bendy, newbie for Apache Airavata - find it very interesting.
> 
> I am looking for some guidance on how to Integrate a custom Workflow Interpreter something similar to "XBaya Interpretter Service" with XBAYA GUI to leverage designer, interaction and monitoring support.
> 
> I'm sure this would have been attempted before as it seems to be in-line with the vision for Airavata/ABaya to be agnostic of underlying workflow runtimes.
> 
> Appreciate your support.
> 
> Thanks,
> Bendy.