You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Martin Grayson (JIRA)" <ji...@apache.org> on 2016/10/11 13:02:20 UTC

[jira] [Created] (AIRFLOW-564) DagRunOperator and ExternalTaskSensor Incompatibility

Martin Grayson created AIRFLOW-564:
--------------------------------------

             Summary: DagRunOperator and ExternalTaskSensor Incompatibility
                 Key: AIRFLOW-564
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-564
             Project: Apache Airflow
          Issue Type: Task
          Components: DagRun
    Affects Versions: Airflow 1.7.1.3
            Reporter: Martin Grayson


I have an hourly Dag that acts to orchestrate other dags, triggering dag runs if a precondition is met.

The controller dag (Dag CD) uses a `TriggerDagRunOperator` operator to do this, launching 2 other dags (Dag A, Dag B) in my example (10+ in my real life). 

Within these dags they have their own dependencies, for example Dag B is dependent on Dag A's completion. This is handled using a `ExternalTaskSensor`. 

Since Dag A and B are executed dynamically with the `TriggerDagRunOperator`, they're assigned very granular execution dates, rather than dd-mm-yyyy hh:00:00 as the scheduler would assign.  Due to this, the `ExternalTaskSensor` will not succeed unless Dag A and B managed to have exactly the same execution_date (down to the second).

As far as I can tell, I'm not able to force the `ExternalTaskSensor` to poke for the correct execution date. The only possible solution would be to implement a `execution_date_fn` that looked through the metadata db to find the exact execution time. This seems massively complicated for a relatively simple requirement.

Can this scenario be modelled with these components?

I've attached a set of example dags and some screenshots, showing the issue.

Thanks,
Martin




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)