[GitHub] [airflow] mik-laj commented on a change in pull request #14048: Speed up clear_task_instances by doing a single sql delete for TaskReschedule

[GitBox <gi...@apache.org>]

mik-laj commented on a change in pull request #14048:
URL: https://github.com/apache/airflow/pull/14048#discussion_r569832257



##########
File path: airflow/models/taskinstance.py
##########
@@ -166,13 +170,22 @@ def clear_task_instances(
                 ti.max_tries = max(ti.max_tries, ti.prev_attempted_tries)
             ti.state = State.NONE
             session.merge(ti)
-        # Clear all reschedules related to the ti to clear
-        session.query(TR).filter(
-            TR.dag_id == ti.dag_id,
-            TR.task_id == ti.task_id,
-            TR.execution_date == ti.execution_date,
-            TR.try_number == ti.try_number,
-        ).delete()
+
+        tr_filter.append((ti.dag_id, ti.task_id, ti.execution_date, ti.try_number))
+
+    # Clear all reschedules related to the ti to clear
+    tr_qry = session.query(TR).filter(
+        or_(
+            and_(
+                TR.dag_id == dag_id,
+                TR.task_id == task_id,
+                TR.execution_date == execution_date,
+                TR.try_number == try_number,
+            )
+            for dag_id, task_id, execution_date, try_number in tr_filter

Review comment:
       Is `tr_filter` always contain at least one element? I'm afraid that if `tr_filter` is empty, too many objects in the database will be deleted because ORM won't generate SQL query without any filters.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org