Adding Apache Airflow support to Elyra's visual pipeline editor

[Alan K Chin <ak...@us.ibm.com>]

Hello All,



Our team is building a Jupyter lab extension to enable users to construct a
ai/data pipeline workflow with notebooks and python scripts using a visual
editor. We have been exploring how to add Airflow(awesome) to our list of
pipeline/workflow
runtimes.(<https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html>).
In a nutshell, we construct a dag using the information provided by the user
in the visual editor with a help of a jinja template, however the current
experimental REST API nor the one in 2.0 will have the ability to upload one
to a dag directory. Have a work in progress branch here:
<https://github.com/elyra-ai/elyra/pull/490>



Our current implementation pushes the constructed dag to an S3 bucket and
syncs the bucket to airflow's dag dir via cron. A more official and formal
ability to sync to dags to a remote repository is something that would be very
useful and has been explored in: <https://github.com/apache/airflow/pull/3138>
as well as AIP-5 . Are there any plans to revive this effort post 2.0
stability?



Its also been suggested that current best practice(is it?) should be to take
advantage of git-syncing the dag bag to a git repo. We are exploring this
option as well.



Any comments or input would be greatly appreciated!



\--  
Alan Chin  
CODAIT, San Francisco  
Email - akchin@us.ibm.com