You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Soma S Dhavala <so...@gmail.com> on 2018/05/27 04:16:36 UTC

feature request: push/save DAG objects directly

Hi AirFlow Developer,

I am Dr. Soma Dhavala, ML architect. At EkStep <https://ekstep.org/>, a
non-profit organisation, we are developing an open source ML workbench and
we are using AirFlow as one of the core components, on the top of which are
writing an application layer. The idea is to make this tool available to
millions of users who use our SunBird <http://www.sunbird.org/about/>
platform

We compose DAGs programmatically - that is an agent auto-composes a DAG
based on a specification (i.e., DAGS are not handcrafted but are generated
based a on template), so we dont have any python script that literally
constructs a DAG object. So it was less than ideal for us to drop .py files
into the DAG folder. An ability to save/write DAG objects directly to the
DB or make them available to the DagBag would ideal.

Right now, we drop a py script that will load pickled DAG objects and makes
them available to the DagBag upon its instantiation. The work around we are
using can be seen here
https://github.com/dhavala/airflow-exp

I am wondering if you have any plans to inject DAG objects via methods
available on the "dag" class or something similar.

thanks,
-soma