[jira] [Created] (AIRFLOW-1414) Add support for retriggering dependent workflows

["Raphael Lopez Kaufman (JIRA)" <ji...@apache.org>]

Raphael Lopez Kaufman created AIRFLOW-1414:
----------------------------------------------

             Summary: Add support for retriggering dependent workflows
                 Key: AIRFLOW-1414
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1414
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: Raphael Lopez Kaufman


Currently when using an ExternalTaskSensor sensor, to have a dag wait for the completion of a task in another dag, there is no way, when clearing the task that is depended on, to also clear the ExternalTaskSensor task (and its downstream/upstream tasks).

However, that might be important when teams have separate responsibilities and create multi-staged data pipelines. Let's say there is a team responsible for transforming logs to make them available in hadoop land. They maintain a hourly dag processing the raw logs. Other teams wait on the completion of this dag using an ExternalTaskSensor. If the logging team realizes that some logs where corrupted, not only do they have to rerun the task that processed these logs but also all the dags that use these logs to generate higher level data.

Introducing the notion of descendants can solve this problem. The descendants of dag A are the dags that have an ExternalTaskSensor pointing to a task of dag A. Then, when clearing a task in the view, an option can be added to also clear the tasks of the descendants (having the upstream, downstream, recursive, future and past flags act accordingly).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)