You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Bo Blanton (Jira)" <ji...@apache.org> on 2019/11/18 18:56:00 UTC

[jira] [Created] (AIRFLOW-6012) Kubernetes Executer very slow to Clean finished tasks

Bo Blanton created AIRFLOW-6012:
-----------------------------------

             Summary: Kubernetes Executer very slow to Clean finished tasks
                 Key: AIRFLOW-6012
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6012
             Project: Apache Airflow
          Issue Type: Improvement
          Components: scheduler
    Affects Versions: 1.10.6
            Reporter: Bo Blanton


The new clean up task in the K8 executor queries the DB for all running tasks/dags for each tasks to check for safe labels (tasks and dagid) as they may have needed to be altered for k8 safety.

For hundereds of tasks finishing, this is a very long process (lots of DB calls and a large loop).

There's an improvement we can make to to quickly exit the loop if we don't need to check for the label saftly



--
This message was sent by Atlassian Jira
(v8.3.4#803005)