[jira] [Updated] (AIRFLOW-316) Always check DB state for Backfill Job execution

["Dan Davydov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dan Davydov updated AIRFLOW-316:
--------------------------------
    External issue URL: https://github.com/apache/incubator-airflow/pull/1654

> Always check DB state for Backfill Job execution
> ------------------------------------------------
>
>                 Key: AIRFLOW-316
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-316
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Dan Davydov
>            Assignee: Dan Davydov
>              Labels: backfill
>
> Always check DB state and not just the local state for backfill jobs for determining which task instances have not yet completed execution. This is to avoid potential race conditions with e.g. two backfill jobs running the same task instance.
> At Airbnb we have noticed a race condition with the subdag operator (which uses the backfill job) where all of the tasks in the subdag complete successfully, but the parent DAG is still waiting on a task in the subdag to succeed. Resolving this JIRA should fix the subdag operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)