Questions about Scufl2, support for it and Taverna 3

[Alex Hardisty <Ha...@cardiff.ac.uk>]

Dear developers,

I have some questions about Scufl2 and support for it that I hope someone on this list can answer for me please. I think they are generally of interest to the list.

The background is that I'm looking for a way of representing scientific workflows that is portable across different execution environments i.e., such that organisations/persons wanting to execute those workflows on their own preferred infrastructure and technologies are not tied into Taverna. I found in the description page of workflow bundles (https://taverna.incubator.apache.org/documentation/scufl2/bundle) that '...profiles define how these workflows can be realised and executed on different environments,...'. So perhaps it is possible?

My questions are:


1.       Is Scufl2 development actively progressing? How widespread is support for it?


2.       Is there a version of Taverna Workbench that produces Scufl2 workflow bundles?


3.       Allied to question (2) are further questions about: What is the status of Taverna Workbench 3.0? When will it be released? How is the community support for Apache Taverna growing?


4.       Is there any work done or in progress on realising /executing Scufl2 workflows on Apache Spark?


5.       Are there any tools/infrastructure other than the Taverna family that can interpret and execute Scufl2?


6.       If I wanted to write workflows in a manner independent of execution technology, would Scufl2 be a good choice? What alternatives are there?


Thanks in advance for your information and help.

Kind regards
--
Alex

Alex Hardisty                                                                         Alex Hardisty
Director of Informatics Projects                                       Cyfarwyddwr y Prosiect Gwybodeg
School of Computer Science and Informatics                 Yr Ysgol Cyfrifiadureg a Gwybodeg
Cardiff University, Queens Buildings                                Prifysgol Caerdydd, Adeiladau'r Frenhines
5, The Parade, Cardiff CF24 3AA                                      5, The Parade, Caerdydd CF24 3AA
United Kingdom                                                                   Y Deyrnas Unedig

tel: +44 (0)29 2087 4861                                                    ffôn : +44 (0)29 2087 4861
email: hardistyar@cardiff.ac.uk<ma...@cardiff.ac.uk>                                        ebost: hardistyar@caerdydd.ac.uk<ma...@caerdydd.ac.uk>
skype: alex.hardisty                                                             skype: alex.hardisty
orcid id: orcid.org/0000-0002-0767-4310<http://orcid.org/0000-0002-0767-4310>

Re: Questions about Scufl2, support for it and Taverna 3

[Stian Soiland-Reyes <st...@apache.org>]

On 8 December 2016 at 10:37, Alex Hardisty <Ha...@cardiff.ac.uk> wrote:
> Dear developers,

Hi!

> I have some questions about Scufl2 and support for it that I hope someone on this list can answer for me please. I think they are generally of interest to the list.

Apologies for the late reply - busy December for many, I think!


> The background is that I'm looking for a way of representing scientific workflows that is portable across different execution environments i.e., such that organisations/persons wanting to execute those workflows on their own preferred infrastructure and technologies are not tied into Taverna. I found in the description page of workflow bundles (https://taverna.incubator.apache.org/documentation/scufl2/bundle) that '...profiles define how these workflows can be realised and executed on different environments,...'. So perhaps it is possible?

> 1.       Is Scufl2 development actively progressing? How widespread is support for it?

SCUFL2 as in the Workflow Bundle format is fairly stabilized - I don't
think we want to change the format now.


The Taverna Language API (SCUFL2 API) is also stabilized - and in
latest release contains the "tavlang" tool for converting and
inspecting workflows.
https://taverna.incubator.apache.org/download/language/


I think the main thing remaining to look at are just code tidy:

* Changing RDF handling of annotations to use Commons RDF
https://issues.apache.org/jira/browse/TAVERNA-1017

* Use RO Bundle API to load/save the bundle
https://issues.apache.org/jira/browse/TAVERNA-965


The above would help improve the Taverna Language API's exposure of
rich annotations, and then we can call it 1.0 without really changing
much from the current 0.15.


> 2.       Is there a version of Taverna Workbench that produces Scufl2 workflow bundles?

The latest published release that can make SCUFL2 wfbundles would be
Taverna Workbench 3.0 alpha2 -
https://taverna.incubator.apache.org/download/pre-apache

Workflows from 2.5 can also be converted using the tavlang tool - but
then can't be opened in 2.5 anymore (unless you don't change the
scufl2 workflow, and extract its original .t2flow which is contained
in the .wfbundle zip)


> 3.       Allied to question (2) are further questions about: What is the status of Taverna Workbench 3.0? When will it be released? How is the community support for Apache Taverna growing?

The Workbench source code is now pretty much ready for a 3.1 release
following an extensive license review and updates to the build system.
https://github.com/apache/incubator-taverna-workbench


Any developer effort would be welcome to close the last loose ends,
and help push a release through!


I can't say when it will be released - it depends on how many put
effort in.  Latest effort of the community has focused on releasing
the engine, command line tool, adding docker/cwl support, as well as
the Taverna Mobile app.

Community-wise we have also worked towards graduation from the
incubator - I guess this took some energy out of the Workbench, which
to be fair was the component that needed the most work to be
"releasable" after moving to Apache.


For next year I think we have to choose between either the Taverna
Server 3.1 or the Taverna Workbench 3.1 to be focus of a release
first.


> 4.       Is there any work done or in progress on realising /executing Scufl2 workflows on Apache Spark?

Not that I am aware of.

We have an experimental backend for Hadoop support, but it has not
been looked at lately:

https://github.com/apache/incubator-taverna-engine/tree/master/taverna-execution-hadoop


> 5.       Are there any tools/infrastructure other than the Taverna family that can interpret and execute Scufl2?

Not to my knowledge.


> 6.       If I wanted to write workflows in a manner independent of execution technology, would Scufl2 be a good choice? What alternatives are there?

If your workflows would be command-line centric, I would rather look
at Common Workflow Language (CWL), which would give you a wide choice
of compatible engine, several of them cloud-centric:

http://www.commonwl.org/


We have been busy working on CWL support in Taverna as well - in two aspects:

a) Running Docker-packaged command line tools
b) Importing and using CWL-described Tools from the workbench

Support for CWL open/save of workflow definitions would work at SCUFL2
level. developed as an additional format supported by the Taverna
Language API.

There a primary goal would be supporting CWL-like command line tool
workflows only.  Supporting other Taverna activities like REST, WSDL
or interaction is also possible in CWL - but it would require more
porting effort as those service types are not currently formally
supported by other CWL engines -- it would be fairly easy to put the
SCUFL2 activity config JSON into CWL - but only Taverna would
understand it until we standardize that further.


CWL is planning a research object format for capturing a CWL workflow
- it really needs a "Workflow Bundle"-like format for capturing the
constituent parts of a workflow and move them around as a single file
that can then be archived and referenced.  I can't say now how similar
this will be to the wfbundle, but it will probably follow a similar
structure; but with CWL YAML files instead of XML files.


See for instance the CWL viewer/archiver:
https://view.commonwl.org/workflows/58457819857aba0001b0077e
which is being developed by a student at University of Manchester and
which gives you a research object download of the workflow.


While CWL is a good future goal with strong interoperability focus (as
well as being hand-editable for developers); I would think the SCUFL2
Workflow Bundle will be the primary format for Taverna workflows for a
bit longer as it supports the existing activity types out of the box
and Taverna workflows don't do just command line tools.

Being able to run CWL workflows with the Taverna Engine (through
command line, server or Java API) would help bring a Java
implementation of CWL to the market.


There might be a point where a workflow can be both a wfbundle and CWL
bundle at the same time.

Again any developer effort here is welcome!


-- 
Stian Soiland-Reyes
http://orcid.org/0000-0001-9842-9718