[jira] [Comment Edited] (AIRFLOW-380) Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting scheduler after long)

["Sergio Kef (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940132#comment-16940132 ] 

Sergio Kef edited comment on AIRFLOW-380 at 9/28/19 6:41 PM:
-------------------------------------------------------------

I think Airflow logic is to catch up (because it follows the very simple rule while start date + interval <= now, create new task instance).

As mentioned above, one can mark all as succeeded, or even rename the dag to v2 with a new start_date.

[~vineetgoel] do you wish to keep this open?


was (Author: serkef):
I think Airflow logic is to catch up (because it follows the very simple rule while start date + interval <= now, create new task instance).

As mentioned above, one can mark all as succeeded, or even rename the dag to v2 with a new start_date.

> Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting scheduler after long)
> --------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-380
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-380
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Vineet Goel
>            Priority: Major
>
> Currently when we unpause a dag or bring the scheduler back up after long, the scheduler starts scheduler dag runs for each execution in between (for the time that has elapsed). There should be a way to cancel these "backfills" if they are not really required.
> While this is simpler to clear using the cli if the dagruns are created simultaneously, it becomes a bit hard if the max_active_runs for the dag is 1. This way, we need to clear each dag run individually which isn't ideal. One workaround is to keep start_date as datetime.now() and deleting past dag runs from the database prior to unpausing the dag. However this isn't ideal.
> Is there a way of doing that I don't know about?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)