[jira] [Commented] (AIRFLOW-3941) Scheduler shouldn't start DagRuns that will never finish

["t oo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837696#comment-16837696 ] 

t oo commented on AIRFLOW-3941:
-------------------------------

Also trigger dag for 2 years in future date will stay running for 2 years

> Scheduler shouldn't start DagRuns that will never finish
> --------------------------------------------------------
>
>                 Key: AIRFLOW-3941
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3941
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Andrew Stahlman
>            Priority: Major
>
> *Problem:* It's possible for the scheduler to create DagRuns with 0 tasks that will be stuck in state 'running' indefinitely.
> *Steps to reproduce:*
>  - Define a DAG where every task has a start_date of time {{<t>}}
>  - {{airflow backfill <dag> -s <t-2> -e <t-2>}}
>  - Turn on the scheduler
> *Actual Behavior:* The scheduler will start creating DagRun starting from {{execution_date=t-1}} in state 'running', but the DagRuns will never complete. If you check the TaskInstance Details page for any task in the DAG, the {{Dependencies Blocking Task From Getting Scheduled}} panel will contain a message like, "The execution date is 2019-01-03T00:00:00+00:00 but this is before the task's start date 2019-02-21T00:00:00+00:00."
> *Ideal behavior:* Ideally, the scheduler would detect that the min(start_date) over all task_instances in the DAG is after the DagRun's execution_date, and either avoid creating such a DagRun in the first place, or create the DagRun (with 0 task instances) and immediately mark it as success.
> Alternatively (or possibly in addition to the above), we could provide some opt-in protection such that if the DAG's start_date is defined, then evaluating the DAG definition will raise an Exception if the DAG's start_date < min(start_date) over all task instances.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)