You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Vincent Poulain <vi...@tinyclues.com> on 2016/08/27 14:04:10 UTC

Dynamic DAG Generation

Hello,
First, thank you for your great job. Airflow made my job easier :)

Is a dynamic generation of DAG a good practice ? As WePay explained there :
https://wecode.wepay.com/posts/airflow-wepay.

In part of a DAG, I have *n* ECS tasks need to be launched daily, each one
with specific *params*. *n & params* depending on configuration which is
daily updated.
For now, I have just one task fetching this configuration and internally
broadcast ECS task.
I am wondering if Airflow would be suitable to generate DAG dynamically
(depending on configuration) in order to split this task into those
multiple ECS task (using ECSOperator, for instance).

Thank you for your tips,