[jira] [Commented] (AIRFLOW-5686) Deleting DAG can leave scheduld/queued tasks consuming pool slots

["Ash Berlin-Taylor (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953595#comment-16953595 ] 

Ash Berlin-Taylor commented on AIRFLOW-5686:
--------------------------------------------

If anyone else runs in to this the workaround I used to fix it was to run this SQL after upgrading:

{code:sql}
WITH inactive_dags AS (SELCT dag_id FROM dag WHERE is_active = false)
UPDATE ONLY task_instance SET state='removed' FROM inactive_dags
WHERE task_instance.dag_id=inactive_dags.dag_id
  AND (state NOT IN ('failed', 'upstream_failed', 'success') or state IS NULL);
{code}

> Deleting DAG can leave scheduld/queued tasks consuming pool slots
> -----------------------------------------------------------------
>
>                 Key: AIRFLOW-5686
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5686
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.10.5
>            Reporter: Ash Berlin-Taylor
>            Priority: Critical
>             Fix For: 1.10.7
>
>
> If you delete a dag file when it had tasks in the scheduled or queued state, those tasks instances are never touched again.
> With the slight tweak in Pool (making the default pool an explicit one) this now matters, and this ends up with the scheduler being "blocked" from running new tasks wiht this message:
> {noformat}
> Figuring out tasks to run in Pool(name=default_pool) with -9022 open slots and 45 task instances ready to be queued {noformat}
> The fix should be to set any task instance in a non-terminal state (None, queued, scheduled, running, up_for_retry etc.) to "removed" inside DAG.deactivate_stale_dags (which is already called on scheduler shutdown).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)