[jira] [Created] (AIRFLOW-5686) Deleting DAG can leave schedules/queued tasks consuming pool slots

["Ash Berlin-Taylor (Jira)" <ji...@apache.org>]

Ash Berlin-Taylor created AIRFLOW-5686:
------------------------------------------

             Summary: Deleting DAG can leave schedules/queued tasks consuming pool slots
                 Key: AIRFLOW-5686
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5686
             Project: Apache Airflow
          Issue Type: Bug
          Components: scheduler
    Affects Versions: 1.10.5
            Reporter: Ash Berlin-Taylor
             Fix For: 1.10.7


If you delete a dag file when it had tasks in the scheduled or queued state, those tasks instances are never touched again.

With the slight tweak in Pool (making the default pool an explicit one) this now matters, and this ends up with the scheduler being "blocked" from running new tasks wiht this message:
{noformat}
Figuring out tasks to run in Pool(name=default_pool) with -9022 open slots and 45 task instances ready to be queued {noformat}
The fix should be to set any task instance in a non-terminal state (None, queued, scheduled, running, up_for_retry etc.) to "removed" inside DAG.deactivate_stale_dags (which is already called on scheduler shutdown).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)