[jira] [Created] (AIRFLOW-3941) Scheduler shouldn't start DagRuns that will never finish

["Andrew Stahlman (JIRA)" <ji...@apache.org>]

Andrew Stahlman created AIRFLOW-3941:
----------------------------------------

             Summary: Scheduler shouldn't start DagRuns that will never finish
                 Key: AIRFLOW-3941
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3941
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: Andrew Stahlman


*Problem:* It's possible for the scheduler to create DagRuns with 0 tasks that will be stuck in state 'running' indefinitely.

*Steps to reproduce:*
 - Define a DAG where every task has a start_date of time {{<t>}}
 - {{airflow backfill <dag> -s <t-2> -e <t-2>}}
 - Turn on the scheduler

*Actual Behavior:* The scheduler will start creating DagRun starting from {{execution_date=t-1}} in state 'running', but the DagRuns will never complete. If you check the TaskInstance Details page for any task in the DAG, the {{Dependencies Blocking Task From Getting Scheduled}} panel will contain a message like, "The execution date is 2019-01-03T00:00:00+00:00 but this is before the task's start date 2019-02-21T00:00:00+00:00."

*Ideal behavior:* Ideally, the scheduler would detect that the min(start_date) over all task_instances in the DAG is after the DagRun's execution_date, and either avoid creating such a DagRun in the first place, or create the DagRun (with 0 task instances) and immediately mark it as success.

Alternatively (or possibly in addition to the above), we could provide some opt-in protection such that if the DAG's start_date is defined, then evaluating the DAG definition will raise an Exception if the DAG's start_date < min(start_date) over all task instances.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)