You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Saminda Wijeratne <sa...@gmail.com> on 2014/06/03 17:38:02 UTC
Re: Workflow handling in the Application Catalog
On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com>
wrote:
>
>
>
> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <sa...@gmail.com>
> wrote:
>
>>
>>
>>
>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> When designing the Application Catalog for single Applications, I ran
>>> into the issue of supporting workflows in the Application Catalog.
>>>
>>> Should we store the workflows ( workflow files) and keep ids, which
>>> would be the handles for those workflows?
>>>
>> Thanks for bringing this up Sachith. In your opinion how do you think the
>> workflow should be represented through the CPI? Workflow string or Thrift
>> object with workflow metadata distinguished or Thrift object with all
>> workflow data distinguished or any other way? I suppose we need to
>> figure-out what should be queried in a workflow.
>>
>
>
> For the functional purposes I think it would be better to store the
> workflow the way it is right now purely because we wouldn't want this to
> affect the Workflow Interpreter behavior. But we can store meta-data about
> the workflows to whatever the features we would want to support. Is there a
> way to get the applications residing in a workflow without explicitly
> passing it to get the meta-data?
>
Workflow nodes specifically does directly correspond to applications but
service descriptors. The current workflow model object have functions to
return these service descriptors.
>
>
>>
>>> We would have to keep track of the applications that are in that
>>> workflow so that the deployment related data would be reused.
>>>
>>> Similarly to the single Applications, the sharing and other features
>>> would be available for the workflows as well.
>>>
>>> Is there any more details that I should be concerned about when
>>> implementing the aforementioned approach?
>>>
>> IMO we can generalize workflows as a composite application with special
>> properties. For example a workflow could simply be another deployment for
>> an Application interface. wdyt?
>>
>
>
> I agree. I'll send you the current design I have which generalize this.
>
>
>>
>>
>>> If we are planning to add searching capability with the workflows as
>>> well, then we'd have to store the applications that are used in the
>>> workflows, separately in the database as well instead of storing it as a
>>> whole.
>>>
>> You mean node information here when you say "applications" right?
>>
>
> yep. wdyt?
>
I think it brings us back to whether want to save the workflow as a single
blob or as separate data for the graph. IMO what we send to the registry
should be a complex workflow object and we should be able to query those
objects from the registry based on their properties. How registry saves
this complex object (as a single blob, metadata+single blob or
metadata+graph data) is immaterial for the user of the registry/app catalog.
>
>>
>>> Any suggestions/comments on the matter is highly appreciated.
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>>
>>>
>>
>
>
> --
> Thanks,
> Sachith Withana
>
>
Re: Workflow handling in the Application Catalog
Posted by Marlon Pierce <ma...@iu.edu>.
Thanks, Eroma. For Step 1, we should also have a programmatic (API) way
to create workflows. For questions 1 and 2, I think the answer is no to
both. Applications in the catalog are available individually and as part
of workflows. #3 is a good point. #4 is part of the larger "delete"
problem that we need to discuss. I think for #5 we have one or the
other (and I personally think single applications are just special cases
of workflows).
Marlon
On 6/9/14 11:35 AM, Eroma Abeysinghe wrote:
> Hi Devs,
>
> I was thinking about workflows from system requirement point of view and
> IMO app catalog should cover below.
> (All given might be taken-care-of already; if so just ignore).
>
> When it comes to WFs (workflows) within App catalog we should be able to;
>
> 1. Create Workflows (More like adding the WF in to App catalog; as
> actual WF creation is in XBAYA)
> - When creating we should be able to give a name, description, ID
> (generated) for the workflow. This would help with searing for
> WFs as well
> as searching for experiments using workflow ID, name, etc....
> - WFs can have a single node or multiple nodes. At least one node has
> to have an application to execute. WFs can have one or many applications
> - In WFs an output from one node can be a input to the next node
> 2. Publish & Un-publish Workflows
> - We need to have the option of publishing and un-publishing them.
> This way we can keep the WFs un-published till they are ready
> and then make
> them available.
> 3. Edit Workflows
> - We should be able to add remove nodes, applications from workflows.
> 4. Clone Workflows
> - This is to assist with workflow creation. If we could clone it can
> save time required to create WFs from scratch. Once cloned user should be
> able to modify the new WF.
> 5. Import Workflows
> - Importing from other gateways or an outside file. (Not so sure how
> the importing works when it comes to XBAYA. If its imported from another
> gateway the WF is already created using XBAYA. I guess then its matter of
> having it inserted into catalog. How would the import work if
> the WF is not
> even created in XBAYA)?
> 6. Search Workflows
> - We should be able to search using
> - Workflow name
> - Workflow description
> - Application (we give an application s the key and all workflows
> using the given app will get listed)
> - Searching for WFs giving experiment details such as exp name,
> exp description, exp status
> - Created user
> - ... any more?
> 7. Delete Workflows
> - This is to completely delete the workflow from app catalog. We may
> not need this if we can un-publish
>
> Few questions
>
> 1. Can we have applications created only to be used in Workflows?
> meaning we cannot use them individually in experiments.
> 2. Can we make an application available in workflows but unavailable for
> individual use at a given time?
> 3. Currently when we search for experiments, we dont say whether we are
> looking for WFs or not. but in future this may be required
> 4. If we are deleting applications in the system if its is been used in
> a WF are we going to allow deletion?
> 5. In an experiment can we use more than one WF? can we use WF and
> application in an experiment?
>
> Thank You,
>
> Best Regards,
>
> Eroma
>
>
>
>
>
> On Mon, Jun 9, 2014 at 12:02 AM, Sachith Withana <sw...@gmail.com>
> wrote:
>
>> Thanks Raman.
>>
>> You are exactly correct.
>>
>> I have three questions.
>>
>> 1. We should store the DAG only. But the problem is that DAG contains
>> which applications are used, and how they are connected.( which output goes
>> to which input). This is a lot of data that is already available in the
>> .wxf files that we use. So basically what you are proposing is shred the
>> .wxf files and store them in the App Catalog?
>>
>> 2. If I remember correctly, the workflows we used had web services as
>> nodes (SimpleMath workflow) , and they were not available in the registry
>> as applications before hand. How do you propose we handle that scenario?
>>
>> 3. Overall, what's the purpose of having .wxf files ( WSDLS) as workflows
>> since now that we will only be using it to transfer data between the
>> client( XBaya) and the server correct?
>>
>>
>> On Thu, Jun 5, 2014 at 7:32 PM, Raminder Singh <ra...@gmail.com>
>> wrote:
>>
>>> Sachith,
>>>
>>> Workflow is a set of applications configured to work together. User need
>>> to define its individual applications and the next step is to create a
>>> workflow to configure the applications together. We should not save
>>> applications used in the workflow as part of workflow. String saved in
>>> workflow is a DAG (Directed acyclic graph) and it has reference to
>>> applications (currently services). Other details saved in the workflow are
>>> conditional blocks (if, for, while conditions), connections (input-output)
>>> and defaults inputs configured by the user. Doing this it will keep the
>>> structure of workflow DAG simple and application defined can be used in
>>> multiple workflows or run standalone if needed. Please let me know if you
>>> have more questions
>>>
>>> Thanks
>>> Raminder
>>>
>>>
>>> On Jun 5, 2014, at 3:59 AM, Sachith Withana <sw...@gmail.com> wrote:
>>>
>>> So what you are suggesting is instead of saving a one big string, we
>>> should shred it and store the nodes and their relationships separately so
>>> that it can be queriable?
>>>
>>> But XBaya and the Workflow Interpreter would still be using the string
>>> right?
>>>
>>>
>>> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <sa...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <samindaw@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> When designing the Application Catalog for single Applications, I ran
>>>>>>> into the issue of supporting workflows in the Application Catalog.
>>>>>>>
>>>>>>> Should we store the workflows ( workflow files) and keep ids, which
>>>>>>> would be the handles for those workflows?
>>>>>>>
>>>>>> Thanks for bringing this up Sachith. In your opinion how do you think
>>>>>> the workflow should be represented through the CPI? Workflow string or
>>>>>> Thrift object with workflow metadata distinguished or Thrift object with
>>>>>> all workflow data distinguished or any other way? I suppose we need to
>>>>>> figure-out what should be queried in a workflow.
>>>>>>
>>>>>
>>>>> For the functional purposes I think it would be better to store the
>>>>> workflow the way it is right now purely because we wouldn't want this to
>>>>> affect the Workflow Interpreter behavior. But we can store meta-data about
>>>>> the workflows to whatever the features we would want to support. Is there a
>>>>> way to get the applications residing in a workflow without explicitly
>>>>> passing it to get the meta-data?
>>>>>
>>>> Workflow nodes specifically does directly correspond to applications but
>>>> service descriptors. The current workflow model object have functions to
>>>> return these service descriptors.
>>>>
>>>>
>>>>>
>>>>>>> We would have to keep track of the applications that are in that
>>>>>>> workflow so that the deployment related data would be reused.
>>>>>>>
>>>>>>> Similarly to the single Applications, the sharing and other features
>>>>>>> would be available for the workflows as well.
>>>>>>>
>>>>>>> Is there any more details that I should be concerned about when
>>>>>>> implementing the aforementioned approach?
>>>>>>>
>>>>>> IMO we can generalize workflows as a composite application with
>>>>>> special properties. For example a workflow could simply be another
>>>>>> deployment for an Application interface. wdyt?
>>>>>>
>>>>>
>>>>> I agree. I'll send you the current design I have which generalize this.
>>>>>
>>>>>
>>>>>>
>>>>>>> If we are planning to add searching capability with the workflows as
>>>>>>> well, then we'd have to store the applications that are used in the
>>>>>>> workflows, separately in the database as well instead of storing it as a
>>>>>>> whole.
>>>>>>>
>>>>>> You mean node information here when you say "applications" right?
>>>>>>
>>>>> yep. wdyt?
>>>>>
>>>> I think it brings us back to whether want to save the workflow as a
>>>> single blob or as separate data for the graph. IMO what we send to the
>>>> registry should be a complex workflow object and we should be able to query
>>>> those objects from the registry based on their properties. How registry
>>>> saves this complex object (as a single blob, metadata+single blob or
>>>> metadata+graph data) is immaterial for the user of the registry/app catalog.
>>>>
>>>>
>>>>
>>>>>>> Any suggestions/comments on the matter is highly appreciated.
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Sachith Withana
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Sachith Withana
>>>>>
>>>>>
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>>
>>>
>>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>>
>
Re: Workflow handling in the Application Catalog
Posted by Eroma Abeysinghe <er...@gmail.com>.
Hi Devs,
I was thinking about workflows from system requirement point of view and
IMO app catalog should cover below.
(All given might be taken-care-of already; if so just ignore).
When it comes to WFs (workflows) within App catalog we should be able to;
1. Create Workflows (More like adding the WF in to App catalog; as
actual WF creation is in XBAYA)
- When creating we should be able to give a name, description, ID
(generated) for the workflow. This would help with searing for
WFs as well
as searching for experiments using workflow ID, name, etc....
- WFs can have a single node or multiple nodes. At least one node has
to have an application to execute. WFs can have one or many applications
- In WFs an output from one node can be a input to the next node
2. Publish & Un-publish Workflows
- We need to have the option of publishing and un-publishing them.
This way we can keep the WFs un-published till they are ready
and then make
them available.
3. Edit Workflows
- We should be able to add remove nodes, applications from workflows.
4. Clone Workflows
- This is to assist with workflow creation. If we could clone it can
save time required to create WFs from scratch. Once cloned user should be
able to modify the new WF.
5. Import Workflows
- Importing from other gateways or an outside file. (Not so sure how
the importing works when it comes to XBAYA. If its imported from another
gateway the WF is already created using XBAYA. I guess then its matter of
having it inserted into catalog. How would the import work if
the WF is not
even created in XBAYA)?
6. Search Workflows
- We should be able to search using
- Workflow name
- Workflow description
- Application (we give an application s the key and all workflows
using the given app will get listed)
- Searching for WFs giving experiment details such as exp name,
exp description, exp status
- Created user
- ... any more?
7. Delete Workflows
- This is to completely delete the workflow from app catalog. We may
not need this if we can un-publish
Few questions
1. Can we have applications created only to be used in Workflows?
meaning we cannot use them individually in experiments.
2. Can we make an application available in workflows but unavailable for
individual use at a given time?
3. Currently when we search for experiments, we dont say whether we are
looking for WFs or not. but in future this may be required
4. If we are deleting applications in the system if its is been used in
a WF are we going to allow deletion?
5. In an experiment can we use more than one WF? can we use WF and
application in an experiment?
Thank You,
Best Regards,
Eroma
On Mon, Jun 9, 2014 at 12:02 AM, Sachith Withana <sw...@gmail.com>
wrote:
> Thanks Raman.
>
> You are exactly correct.
>
> I have three questions.
>
> 1. We should store the DAG only. But the problem is that DAG contains
> which applications are used, and how they are connected.( which output goes
> to which input). This is a lot of data that is already available in the
> .wxf files that we use. So basically what you are proposing is shred the
> .wxf files and store them in the App Catalog?
>
> 2. If I remember correctly, the workflows we used had web services as
> nodes (SimpleMath workflow) , and they were not available in the registry
> as applications before hand. How do you propose we handle that scenario?
>
> 3. Overall, what's the purpose of having .wxf files ( WSDLS) as workflows
> since now that we will only be using it to transfer data between the
> client( XBaya) and the server correct?
>
>
> On Thu, Jun 5, 2014 at 7:32 PM, Raminder Singh <ra...@gmail.com>
> wrote:
>
>> Sachith,
>>
>> Workflow is a set of applications configured to work together. User need
>> to define its individual applications and the next step is to create a
>> workflow to configure the applications together. We should not save
>> applications used in the workflow as part of workflow. String saved in
>> workflow is a DAG (Directed acyclic graph) and it has reference to
>> applications (currently services). Other details saved in the workflow are
>> conditional blocks (if, for, while conditions), connections (input-output)
>> and defaults inputs configured by the user. Doing this it will keep the
>> structure of workflow DAG simple and application defined can be used in
>> multiple workflows or run standalone if needed. Please let me know if you
>> have more questions
>>
>> Thanks
>> Raminder
>>
>>
>> On Jun 5, 2014, at 3:59 AM, Sachith Withana <sw...@gmail.com> wrote:
>>
>> So what you are suggesting is instead of saving a one big string, we
>> should shred it and store the nodes and their relationships separately so
>> that it can be queriable?
>>
>> But XBaya and the Workflow Interpreter would still be using the string
>> right?
>>
>>
>> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <sa...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <samindaw@gmail.com
>>>> > wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> When designing the Application Catalog for single Applications, I ran
>>>>>> into the issue of supporting workflows in the Application Catalog.
>>>>>>
>>>>>> Should we store the workflows ( workflow files) and keep ids, which
>>>>>> would be the handles for those workflows?
>>>>>>
>>>>> Thanks for bringing this up Sachith. In your opinion how do you think
>>>>> the workflow should be represented through the CPI? Workflow string or
>>>>> Thrift object with workflow metadata distinguished or Thrift object with
>>>>> all workflow data distinguished or any other way? I suppose we need to
>>>>> figure-out what should be queried in a workflow.
>>>>>
>>>>
>>>>
>>>> For the functional purposes I think it would be better to store the
>>>> workflow the way it is right now purely because we wouldn't want this to
>>>> affect the Workflow Interpreter behavior. But we can store meta-data about
>>>> the workflows to whatever the features we would want to support. Is there a
>>>> way to get the applications residing in a workflow without explicitly
>>>> passing it to get the meta-data?
>>>>
>>> Workflow nodes specifically does directly correspond to applications but
>>> service descriptors. The current workflow model object have functions to
>>> return these service descriptors.
>>>
>>>
>>>>
>>>>
>>>>>
>>>>>> We would have to keep track of the applications that are in that
>>>>>> workflow so that the deployment related data would be reused.
>>>>>>
>>>>>> Similarly to the single Applications, the sharing and other features
>>>>>> would be available for the workflows as well.
>>>>>>
>>>>>> Is there any more details that I should be concerned about when
>>>>>> implementing the aforementioned approach?
>>>>>>
>>>>> IMO we can generalize workflows as a composite application with
>>>>> special properties. For example a workflow could simply be another
>>>>> deployment for an Application interface. wdyt?
>>>>>
>>>>
>>>>
>>>> I agree. I'll send you the current design I have which generalize this.
>>>>
>>>>
>>>>>
>>>>>
>>>>>> If we are planning to add searching capability with the workflows as
>>>>>> well, then we'd have to store the applications that are used in the
>>>>>> workflows, separately in the database as well instead of storing it as a
>>>>>> whole.
>>>>>>
>>>>> You mean node information here when you say "applications" right?
>>>>>
>>>>
>>>> yep. wdyt?
>>>>
>>> I think it brings us back to whether want to save the workflow as a
>>> single blob or as separate data for the graph. IMO what we send to the
>>> registry should be a complex workflow object and we should be able to query
>>> those objects from the registry based on their properties. How registry
>>> saves this complex object (as a single blob, metadata+single blob or
>>> metadata+graph data) is immaterial for the user of the registry/app catalog.
>>>
>>>
>>>
>>>>
>>>>>
>>>>>> Any suggestions/comments on the matter is highly appreciated.
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Sachith Withana
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Sachith Withana
>>>>
>>>>
>>>
>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>>
>>
>
>
> --
> Thanks,
> Sachith Withana
>
>
--
Thank You,
Best Regards,
Eroma
Re: Workflow handling in the Application Catalog
Posted by Raminder Singh <ra...@gmail.com>.
We need to change the DAG and workflow engine according to new services. We need not to use XML representation (.WXF) of workflow going forward. We don’t have web services and WSDL binding anymore. We can come with a simpler DAG using JSON. This should work well with the web version of Xbaya also. I think GSOC 2013 students came with some JSON model and you can evaluate the and go ahead with the application catalog for workflows.
Thanks
Raminder
On Jun 9, 2014, at 12:02 AM, Sachith Withana <sw...@gmail.com> wrote:
> Thanks Raman.
>
> You are exactly correct.
>
> I have three questions.
>
> 1. We should store the DAG only. But the problem is that DAG contains which applications are used, and how they are connected.( which output goes to which input). This is a lot of data that is already available in the .wxf files that we use. So basically what you are proposing is shred the .wxf files and store them in the App Catalog?
>
> 2. If I remember correctly, the workflows we used had web services as nodes (SimpleMath workflow) , and they were not available in the registry as applications before hand. How do you propose we handle that scenario?
>
> 3. Overall, what's the purpose of having .wxf files ( WSDLS) as workflows since now that we will only be using it to transfer data between the client( XBaya) and the server correct?
>
>
> On Thu, Jun 5, 2014 at 7:32 PM, Raminder Singh <ra...@gmail.com> wrote:
> Sachith,
>
> Workflow is a set of applications configured to work together. User need to define its individual applications and the next step is to create a workflow to configure the applications together. We should not save applications used in the workflow as part of workflow. String saved in workflow is a DAG (Directed acyclic graph) and it has reference to applications (currently services). Other details saved in the workflow are conditional blocks (if, for, while conditions), connections (input-output) and defaults inputs configured by the user. Doing this it will keep the structure of workflow DAG simple and application defined can be used in multiple workflows or run standalone if needed. Please let me know if you have more questions
>
> Thanks
> Raminder
>
>
> On Jun 5, 2014, at 3:59 AM, Sachith Withana <sw...@gmail.com> wrote:
>
>> So what you are suggesting is instead of saving a one big string, we should shred it and store the nodes and their relationships separately so that it can be queriable?
>>
>> But XBaya and the Workflow Interpreter would still be using the string right?
>>
>>
>> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <sa...@gmail.com> wrote:
>>
>>
>>
>> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com> wrote:
>>
>>
>>
>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <sa...@gmail.com> wrote:
>>
>>
>>
>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com> wrote:
>> Hi all,
>>
>> When designing the Application Catalog for single Applications, I ran into the issue of supporting workflows in the Application Catalog.
>>
>> Should we store the workflows ( workflow files) and keep ids, which would be the handles for those workflows?
>> Thanks for bringing this up Sachith. In your opinion how do you think the workflow should be represented through the CPI? Workflow string or Thrift object with workflow metadata distinguished or Thrift object with all workflow data distinguished or any other way? I suppose we need to figure-out what should be queried in a workflow.
>>
>>
>> For the functional purposes I think it would be better to store the workflow the way it is right now purely because we wouldn't want this to affect the Workflow Interpreter behavior. But we can store meta-data about the workflows to whatever the features we would want to support. Is there a way to get the applications residing in a workflow without explicitly passing it to get the meta-data?
>> Workflow nodes specifically does directly correspond to applications but service descriptors. The current workflow model object have functions to return these service descriptors.
>>
>>
>>
>> We would have to keep track of the applications that are in that workflow so that the deployment related data would be reused.
>>
>> Similarly to the single Applications, the sharing and other features would be available for the workflows as well.
>>
>> Is there any more details that I should be concerned about when implementing the aforementioned approach?
>> IMO we can generalize workflows as a composite application with special properties. For example a workflow could simply be another deployment for an Application interface. wdyt?
>>
>>
>> I agree. I'll send you the current design I have which generalize this.
>>
>>
>>
>> If we are planning to add searching capability with the workflows as well, then we'd have to store the applications that are used in the workflows, separately in the database as well instead of storing it as a whole.
>> You mean node information here when you say "applications" right?
>>
>> yep. wdyt?
>> I think it brings us back to whether want to save the workflow as a single blob or as separate data for the graph. IMO what we send to the registry should be a complex workflow object and we should be able to query those objects from the registry based on their properties. How registry saves this complex object (as a single blob, metadata+single blob or metadata+graph data) is immaterial for the user of the registry/app catalog.
>>
>>
>>
>>
>> Any suggestions/comments on the matter is highly appreciated.
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>>
>>
>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>>
>>
>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>
>
>
>
> --
> Thanks,
> Sachith Withana
>
Re: Workflow handling in the Application Catalog
Posted by Sachith Withana <sw...@gmail.com>.
Thanks Raman.
You are exactly correct.
I have three questions.
1. We should store the DAG only. But the problem is that DAG contains which
applications are used, and how they are connected.( which output goes to
which input). This is a lot of data that is already available in the .wxf
files that we use. So basically what you are proposing is shred the .wxf
files and store them in the App Catalog?
2. If I remember correctly, the workflows we used had web services as nodes
(SimpleMath workflow) , and they were not available in the registry as
applications before hand. How do you propose we handle that scenario?
3. Overall, what's the purpose of having .wxf files ( WSDLS) as workflows
since now that we will only be using it to transfer data between the
client( XBaya) and the server correct?
On Thu, Jun 5, 2014 at 7:32 PM, Raminder Singh <ra...@gmail.com>
wrote:
> Sachith,
>
> Workflow is a set of applications configured to work together. User need
> to define its individual applications and the next step is to create a
> workflow to configure the applications together. We should not save
> applications used in the workflow as part of workflow. String saved in
> workflow is a DAG (Directed acyclic graph) and it has reference to
> applications (currently services). Other details saved in the workflow are
> conditional blocks (if, for, while conditions), connections (input-output)
> and defaults inputs configured by the user. Doing this it will keep the
> structure of workflow DAG simple and application defined can be used in
> multiple workflows or run standalone if needed. Please let me know if you
> have more questions
>
> Thanks
> Raminder
>
>
> On Jun 5, 2014, at 3:59 AM, Sachith Withana <sw...@gmail.com> wrote:
>
> So what you are suggesting is instead of saving a one big string, we
> should shred it and store the nodes and their relationships separately so
> that it can be queriable?
>
> But XBaya and the Workflow Interpreter would still be using the string
> right?
>
>
> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <sa...@gmail.com>
> wrote:
>
>>
>>
>>
>> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <sa...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> When designing the Application Catalog for single Applications, I ran
>>>>> into the issue of supporting workflows in the Application Catalog.
>>>>>
>>>>> Should we store the workflows ( workflow files) and keep ids, which
>>>>> would be the handles for those workflows?
>>>>>
>>>> Thanks for bringing this up Sachith. In your opinion how do you think
>>>> the workflow should be represented through the CPI? Workflow string or
>>>> Thrift object with workflow metadata distinguished or Thrift object with
>>>> all workflow data distinguished or any other way? I suppose we need to
>>>> figure-out what should be queried in a workflow.
>>>>
>>>
>>>
>>> For the functional purposes I think it would be better to store the
>>> workflow the way it is right now purely because we wouldn't want this to
>>> affect the Workflow Interpreter behavior. But we can store meta-data about
>>> the workflows to whatever the features we would want to support. Is there a
>>> way to get the applications residing in a workflow without explicitly
>>> passing it to get the meta-data?
>>>
>> Workflow nodes specifically does directly correspond to applications but
>> service descriptors. The current workflow model object have functions to
>> return these service descriptors.
>>
>>
>>>
>>>
>>>>
>>>>> We would have to keep track of the applications that are in that
>>>>> workflow so that the deployment related data would be reused.
>>>>>
>>>>> Similarly to the single Applications, the sharing and other features
>>>>> would be available for the workflows as well.
>>>>>
>>>>> Is there any more details that I should be concerned about when
>>>>> implementing the aforementioned approach?
>>>>>
>>>> IMO we can generalize workflows as a composite application with special
>>>> properties. For example a workflow could simply be another deployment for
>>>> an Application interface. wdyt?
>>>>
>>>
>>>
>>> I agree. I'll send you the current design I have which generalize this.
>>>
>>>
>>>>
>>>>
>>>>> If we are planning to add searching capability with the workflows as
>>>>> well, then we'd have to store the applications that are used in the
>>>>> workflows, separately in the database as well instead of storing it as a
>>>>> whole.
>>>>>
>>>> You mean node information here when you say "applications" right?
>>>>
>>>
>>> yep. wdyt?
>>>
>> I think it brings us back to whether want to save the workflow as a
>> single blob or as separate data for the graph. IMO what we send to the
>> registry should be a complex workflow object and we should be able to query
>> those objects from the registry based on their properties. How registry
>> saves this complex object (as a single blob, metadata+single blob or
>> metadata+graph data) is immaterial for the user of the registry/app catalog.
>>
>>
>>
>>>
>>>>
>>>>> Any suggestions/comments on the matter is highly appreciated.
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Sachith Withana
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>>
>>>
>>
>
>
> --
> Thanks,
> Sachith Withana
>
>
>
--
Thanks,
Sachith Withana
Re: Workflow handling in the Application Catalog
Posted by Raminder Singh <ra...@gmail.com>.
Sachith,
Workflow is a set of applications configured to work together. User need to define its individual applications and the next step is to create a workflow to configure the applications together. We should not save applications used in the workflow as part of workflow. String saved in workflow is a DAG (Directed acyclic graph) and it has reference to applications (currently services). Other details saved in the workflow are conditional blocks (if, for, while conditions), connections (input-output) and defaults inputs configured by the user. Doing this it will keep the structure of workflow DAG simple and application defined can be used in multiple workflows or run standalone if needed. Please let me know if you have more questions
Thanks
Raminder
On Jun 5, 2014, at 3:59 AM, Sachith Withana <sw...@gmail.com> wrote:
> So what you are suggesting is instead of saving a one big string, we should shred it and store the nodes and their relationships separately so that it can be queriable?
>
> But XBaya and the Workflow Interpreter would still be using the string right?
>
>
> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
>
>
> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com> wrote:
>
>
>
> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
>
>
> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com> wrote:
> Hi all,
>
> When designing the Application Catalog for single Applications, I ran into the issue of supporting workflows in the Application Catalog.
>
> Should we store the workflows ( workflow files) and keep ids, which would be the handles for those workflows?
> Thanks for bringing this up Sachith. In your opinion how do you think the workflow should be represented through the CPI? Workflow string or Thrift object with workflow metadata distinguished or Thrift object with all workflow data distinguished or any other way? I suppose we need to figure-out what should be queried in a workflow.
>
>
> For the functional purposes I think it would be better to store the workflow the way it is right now purely because we wouldn't want this to affect the Workflow Interpreter behavior. But we can store meta-data about the workflows to whatever the features we would want to support. Is there a way to get the applications residing in a workflow without explicitly passing it to get the meta-data?
> Workflow nodes specifically does directly correspond to applications but service descriptors. The current workflow model object have functions to return these service descriptors.
>
>
>
> We would have to keep track of the applications that are in that workflow so that the deployment related data would be reused.
>
> Similarly to the single Applications, the sharing and other features would be available for the workflows as well.
>
> Is there any more details that I should be concerned about when implementing the aforementioned approach?
> IMO we can generalize workflows as a composite application with special properties. For example a workflow could simply be another deployment for an Application interface. wdyt?
>
>
> I agree. I'll send you the current design I have which generalize this.
>
>
>
> If we are planning to add searching capability with the workflows as well, then we'd have to store the applications that are used in the workflows, separately in the database as well instead of storing it as a whole.
> You mean node information here when you say "applications" right?
>
> yep. wdyt?
> I think it brings us back to whether want to save the workflow as a single blob or as separate data for the graph. IMO what we send to the registry should be a complex workflow object and we should be able to query those objects from the registry based on their properties. How registry saves this complex object (as a single blob, metadata+single blob or metadata+graph data) is immaterial for the user of the registry/app catalog.
>
>
>
>
> Any suggestions/comments on the matter is highly appreciated.
>
> --
> Thanks,
> Sachith Withana
>
>
>
>
>
> --
> Thanks,
> Sachith Withana
>
>
>
>
>
> --
> Thanks,
> Sachith Withana
>
Re: Workflow handling in the Application Catalog
Posted by Sachith Withana <sw...@gmail.com>.
So what you are suggesting is instead of saving a one big string, we should
shred it and store the nodes and their relationships separately so that it
can be queriable?
But XBaya and the Workflow Interpreter would still be using the string
right?
On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <sa...@gmail.com>
wrote:
>
>
>
> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <sw...@gmail.com>
> wrote:
>
>>
>>
>>
>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <sa...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <sw...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> When designing the Application Catalog for single Applications, I ran
>>>> into the issue of supporting workflows in the Application Catalog.
>>>>
>>>> Should we store the workflows ( workflow files) and keep ids, which
>>>> would be the handles for those workflows?
>>>>
>>> Thanks for bringing this up Sachith. In your opinion how do you think
>>> the workflow should be represented through the CPI? Workflow string or
>>> Thrift object with workflow metadata distinguished or Thrift object with
>>> all workflow data distinguished or any other way? I suppose we need to
>>> figure-out what should be queried in a workflow.
>>>
>>
>>
>> For the functional purposes I think it would be better to store the
>> workflow the way it is right now purely because we wouldn't want this to
>> affect the Workflow Interpreter behavior. But we can store meta-data about
>> the workflows to whatever the features we would want to support. Is there a
>> way to get the applications residing in a workflow without explicitly
>> passing it to get the meta-data?
>>
> Workflow nodes specifically does directly correspond to applications but
> service descriptors. The current workflow model object have functions to
> return these service descriptors.
>
>
>>
>>
>>>
>>>> We would have to keep track of the applications that are in that
>>>> workflow so that the deployment related data would be reused.
>>>>
>>>> Similarly to the single Applications, the sharing and other features
>>>> would be available for the workflows as well.
>>>>
>>>> Is there any more details that I should be concerned about when
>>>> implementing the aforementioned approach?
>>>>
>>> IMO we can generalize workflows as a composite application with special
>>> properties. For example a workflow could simply be another deployment for
>>> an Application interface. wdyt?
>>>
>>
>>
>> I agree. I'll send you the current design I have which generalize this.
>>
>>
>>>
>>>
>>>> If we are planning to add searching capability with the workflows as
>>>> well, then we'd have to store the applications that are used in the
>>>> workflows, separately in the database as well instead of storing it as a
>>>> whole.
>>>>
>>> You mean node information here when you say "applications" right?
>>>
>>
>> yep. wdyt?
>>
> I think it brings us back to whether want to save the workflow as a single
> blob or as separate data for the graph. IMO what we send to the registry
> should be a complex workflow object and we should be able to query those
> objects from the registry based on their properties. How registry saves
> this complex object (as a single blob, metadata+single blob or
> metadata+graph data) is immaterial for the user of the registry/app catalog.
>
>
>
>>
>>>
>>>> Any suggestions/comments on the matter is highly appreciated.
>>>>
>>>> --
>>>> Thanks,
>>>> Sachith Withana
>>>>
>>>>
>>>
>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>
>>
>
--
Thanks,
Sachith Withana