Pause DAG leading to Hang State of DAG

[Vardan Gupta <va...@gmail.com>]

Hi Everyone,

We are using Airflow v1.9 and observing a pattern where we have few DAGs
marked in paused state and scheduled existing DAG runs are not getting
marked success/failure instead they are always in running state.

Following is the screenshot of dag table where it can be clearly seen that
DAG is in paused state.
[image: dag.png]


Following is the screenshot of task_instance table where state for two
executions of a DAG having single task has been marked as success.
[image: task_instance.png]

Following is the screenshot of dag_run table where both the executions of
DAG
[image: dag_run.png]

The moment, we unpause the DAG, it will back fill all the missed executions
along with the changing state of existing DAG runs in dag_run table. Since,
we've written a DELETE functionality in an abstracted form which works only
if the DAG is disabled & no active runs are there, we could not perform
DELETE operation as runs are in running state forever in such scenarios.

Is it a default behavior that if a DAG is disabled then no further dag run
will be kicked off along with no execution of residual part of running
instances of DAG(may be 2/3 task instances) which was unprocessed by the
time paused state was requested? If this is a bug, we can open JIRA and
also contribute.

Let us know, if additional information is required. Would love to get this
addressed.

Regards,
Vardan Gupta