CWL support in Airflow

[Misha Kotliar <mi...@gmail.com>]

Hi Airflow community,

I’m Michael from Cincinnaty Children’s Hospital Medical Center, Barski-Lab.
Our lab is working on bioinfromatic workflows written in CWL. We run them on
our local server using Airflow.

Every CWL pipeline is a YAML/JSON file that describes inputs, outputs and the
relationship between the steps. Every step runs a specific tool within Docker.
Workflow inputs are provided through the separate YAML/JSON job file.
More details about CWL standard can be found here https://www.commonwl.org/ <https://www.commonwl.org/>

Unfortunately, Airflow cannot parse CWL files directly. To address this problem
we created Python package that can do it for Airflow. Actually, we developed
two packages with similar functionality.

1. CWL Airflow  -  https://github.com/Barski-lab/cwl-airflow <https://github.com/Barski-lab/cwl-airflow>2. CWL Airflow Parser  -  https://github.com/datirium/cwl-airflow-parser <https://github.com/datirium/cwl-airflow-parser>

Both of them can parse CWL file and create DAG based on the pipeline’s structure.
The major difference between this two packages is that the first one creates
new DAG for every combination of CWL workflow & job file, whereas the second
one creates DAG for every unique workflow and uses job file only to trigger
DagRun with the specific parameters. Every approach has its pros and cons.

Both programs work good on our server, and we are looking forward to integrate this
functionality into Airflow officially. For this reason we need an advice about the best
way to do this.

It would be nice to make Airflow recognize and parse *.cwl files directly from the dag
folder. Perhaps, I can somehow use Airflow Plugins for this purpose?

I would be happy to find people who share the same ideas of making Airflow compatible
with CWL standards

Best regards,
Michael