Re: Common workflow language

[Stian Soiland-Reyes <st...@apache.org>]

Agreed,

Here are the issues in Jira, I used the label cwl:

https://issues.apache.org/jira/issues/?jql=labels%20%3D%20cwl%20AND%20Project%20%3D%20TAVERNA


So my suggested approach is to try to use the CWL Java SDK (which
probably needs some contributions as it only appeared today on GitHub)
and parse CWL into rough JSON configuration per step in the workflow.

YAML can be parsed as JSON, so that bit shouldn't be too hard.. what
is tricky perhaps is that CWL lets you reference tool descriptions in
separate files, so CWL workflows are not very portable yet. (Unless we
propose CWL to do a Research Object bundling like we do with the
Workflow Bundles)



On 14 December 2015 at 18:12, Gale Naylor <Ga...@noventussolutions.com> wrote:
> Creating JIRA issues makes sense to me. Unless we have another way to keep
> track of proposed or potential actions.
>
> Gale
>
> On Fri, Dec 11, 2015 at 6:02 AM Ian Dunlop <ia...@manchester.ac.uk>
> wrote:
>
>> Hello,
>>
>> I know this is a crazy suggestion, but......are there some issues that
>> we can add to JIRA around CWL or jbatch. Otherwise it will be lost in
>> the email thread.
>>
>> eg
>> * CWL reader for SCUFL2
>> * Docker image for taverna activities (docker compose based?)
>> * Integrate with CWL tool registries (which I guess means getting the
>> workbench working!)
>>
>> Cheers,
>>
>> Ian
>>
>> On 10/12/2015 09:35, Stian Soiland-Reyes wrote:
>> >  I've signed up for the CWL hackathon, but I hope others would join!
>> >
>> >
>> > I find CWL interesting as it has focused on the tool descriptions, and
>> > packaging them using Docker. This is a pragmatic approach that means
>> > lots of issues on reproducibility, installation and repurposability is
>> > pretty much solved.
>> >
>> > It does however means it is hard to use non-local tools like REST
>> > services, as you would need an equivalent command line tool.  Message
>> > parsing and shims are trickier, but on the plus side you could use
>> > your favourite scripting language (as long as it works on Linux :)).
>> >
>> >
>> > Docker is also an important part of http://bioboxes.org/
>> >
>> >
>> > CWL Tool descriptions are annotated with the EDAM ontology, which
>> > includes many ELIXIR folks. ELIXIR is building a tool registry.
>> >
>> > Now I think it would be nice if we could also use such a tool registry
>> > and descriptions from Taverna.
>> >
>> >
>> > CWL focuses on data as files, in a way that is very similar to the
>> > current Tool service.
>> >
>> > I think we could update the Tool Service to handle "CWL tools" or at
>> > least handle docker images.  Of course you could do this today with
>> > just having a command line tool that is "docker run blablabla".
>> >
>> >
>> > I would like to start work on a CWL reader for SCUFL2, which would
>> > create such Tool instances in the first place.
>> >
>> >
>> > Representing any SCUFL2/Taverna workflow in CWL can be tricky, as you
>> > would need to extend the tool type to have REST, WSDL, etc/. This is
>> > an extension point in CWL though - and so it would perhaps raise the
>> > question of why use CWL with custom extensions if nobody else can run
>> > such the Taverna services. Still, the workflow structure can be
>> > repurposed, with a different activity inserted in other CWL engines.
>> > There's a danger here of "standards mutation" - like we saw a decade
>> > ago with WSDL being modified beyond recognition in Globus web services
>> > and WSRF.
>> >
>> >
>> > Another approach could be to create Docker images that can execute
>> > Taverna activities (e.g. using the wsdl-generic command line). This
>> > would be more of an interesting research angle, perhaps, as it means
>> > we would be more freely able to try to run a Taverna workflow with an
>> > alternate CWL workflow engine.
>> >
>> >
>> > Taverna workflows allow various control mechanisms like Retry,
>> > Failover, Parallelism, inplicit iterations, dot/cross product - which
>> > could be tricky to represent in CWL. I think only a subset of Taverna
>> > workflows could be run as CWL - without again adding customizations -
>> > kind of like a CWL profile.
>> >
>> >
>> > I think one aspect is that CWL is also a focus opportunity for Taverna
>> > - coordinating command line tools is simpler than trying to integrate
>> > "everything". I think also this is more of the growing space we find
>> > Taverna in now, rather than 8 years ago when Taverna aimed to be
>> > bioinformatician's best friend and we added lots of GUI and activity
>> > plugins for various bioinformatics-centric services. (however
>> > AstroTaverna is one landscape which is still open to be claimed in
>> > such a way.).  CWL is a bit more on a "hacker level" - which I think
>> > we can be more comfortable with catering for.
>> >
>> >
>> > For CWL, Taverna is a valuable proposition as we know how to handle
>> > workflow provenance, we can easily add preservation aspects (e.g.
>> > packaging all the docker images used in a run).  The Taverna Workbench
>> > has a pluggable way to add service descriptions, so we could connect
>> > to CWL tool registries and turn Taverna into a "CWL editor".
>> >
>> >
>> > On 4 December 2015 at 17:05, Alan Williams <al...@googlemail.com>
>> wrote:
>> >> On 04-Dec-15 15:28, Ian Dunlop wrote:
>> >>>
>> >>> -----BEGIN PGP SIGNED MESSAGE-----
>> >>> Hash: SHA256
>> >>>
>> >>> Hello,
>> >>>
>> >>> Does anyone have views on how Apache Taverna fits into the goals of
>> >>> the Common Workflow Language project
>> >>> https://github.com/common-workflow-language/common-workflow-language ?
>> >>> Should taverna adopt CWL instead of SCUFL2. Are there things that
>> >>> SCUFL2 or the current taverna engine offers that CWL does not?
>> >>
>> >>
>> >> Stian is probably the best person to answer this as he has been quite
>> >> involved with CWL. To the best of my knowledge, CWL started off very
>> similar
>> >> to SCUFL2 but diverged.
>> >>
>> >> There is a CWL/Taverna/Galaxy hackathon planned for May 2016 in Paris. I
>> >> think (hope) I sent around the Doodle poll for it.
>> >>
>> >>> Cheers,
>> >>>
>> >>> Ian
>> >>
>> >>
>> >> Alan
>> >>
>> >
>> >
>> >
>>
>> --
>> Ian Dunlop, eScience Lab
>> School of Computer Science
>> The University of Manchester
>> http://orcid.org/0000-0001-7066-3350
>>



-- 
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons RDF (incubating)
http://orcid.org/0000-0001-9842-9718