You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "xBis7 (via GitHub)" <gi...@apache.org> on 2026/08/17 21:07:19 UTC

[PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

xBis7 opened a new pull request, #71737:
URL: https://github.com/apache/airflow/pull/71737

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull request.
   Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the documentation.
   
   For user-facing UI changes, please attach before/after screenshots (or a short
   screen recording) so reviewers can assess the visual impact.
   
   Feel free to ping (in general) for the review if you do not see reaction for a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   The goal of this PR is to increase scheduler performance.
   
   ## Scheduler investigation
   
   In our production environment, sometimes schedulers become very slow under load.
   
   During my investigation, I added debug spans for every major step of the scheduler's iteration loop to help understand the issue. There is an open PR with the changes
   
   https://github.com/apache/airflow/pull/69809
   
   I ran various tests and with the help of the debug spans from the PR, I was able to identify a possible bottleneck.
   
   When we have multiple dags with a lot of tasks running in parallel, the scheduler spends most of the time scanning the task instances of the active dag runs to decide which can be queued. It actually looks for SCHEDULED tasks that can be moved to QUEUED state.
   
   Once the tasks are moved to QUEUED state, then the worker can pick them up and it's no longer up to the scheduler.
   
   The scan is consisted of 2 parts. The actual DB query that fetches the data and the SQL-alchemy part which hydrates ORM objects with the query results. Basically, it creates `taskinstance` objects and fills them with data from each row that we get back. According to the spans, the query is very fast and the entire time is spent on the object hydration part.
   
   The code distinguishes between finished and unfinished tasks but only after the query. It fetches all of them from the DB and creates objects and then splits them into 2 separate lists.
   
   ## Solution
   
   Let's assume that we have a dag run with 1000 tasks. If 100 tasks have finished, then during the scan we hydrate 100 objects for finished tasks and 900 for unfinished. Similarly, if 400 tasks have finished, then we hydrate 400 objects for finished tasks and 600 for unfinished. As the dag run progresses we get more and more finished tasks.
   
   For finished tasks, we don't really need full ORM objects because we only read 5 fields and we never modify them. Instead of creating a full `taskinstance` object, we can hydrate a more lightweight immutable object with only the 5 necessary fields.
   
   Instead of having 1 query that fetches everything, creates objects and then splits them into 2 lists, we can have 2 queries, one for unfinished tasks that hydrates full objects and one for finished tasks that hydrates lightweight read-only objects.
   
   There is a scenario with dag callbacks where we need the actual `taskinstance` object in which case we fetch it from the DB. Still cheaper than the current approach.
   
   ## Testing
   
   ### Setup
   
   * 1 scheduler
   * 3 celery workers × 18 slots = 54 execution slots
   
   ### Config
   
   * max_tis_per_query=16 (default)
   * max_dagruns_per_loop_to_schedule=20
   * parallelism, max_active_tasks_per_dag and default_pool_task_slot_count set to 4096 so that there isn't a limit other than the scheduler itself
   
   ### Workload
   
   * 35 identical dag runs all start in parallel
   * each dag has a 1st deferrable task and then 9 sequential mapped task stages
   * when the deferrable task finishes, it returns a number that determines the number of tasks that each stage expands to
   
   The deferrable task doesn't have any special meaning. It just helps to make sure that most of the mapped tasks are running in parallel.
   
   So for example, if the 1st task returns 10, the 1st stage will expand into 10 tasks. Once these 10 finish, then the 2nd expands into 10 tasks. Once, these finish then the 3rd expands into 10 tasks and so on. And this is running for 35 parallel dags. 
   
   I used 70 for the number of mapped tasks, which translates into 9stages * 70tasks * 35dags = 22050 tasks and another 35 deferred, 22085 in total.
   
   The code from `main` takes 35 minutes to run the 22000 tasks, while the code from this patch took 28 minutes. The more the load, the bigger the difference.
   
   **Here are some gathered metrics during the test. Left is the code from `main` and right is the code from this patch.**
   
   ### Time needed for each scan (the PR improvement)
   
   <img width="1980" height="614" alt="image" src="https://github.com/user-attachments/assets/1593966c-285c-45c2-b008-28e3d765240d" />
   
   Before it reached up to 94ms while after it never exceeded 20ms.
   
   ### Scheduler loop duration per iteration
   
   <img width="1978" height="608" alt="image" src="https://github.com/user-attachments/assets/d9ad6409-2bb1-4a71-8190-08f883f57fed" />
   
   Iterations became faster.
   
   ### Scheduler vs worker vs worker slot utilization
   
   <img width="3966" height="1218" alt="image" src="https://github.com/user-attachments/assets/509bd139-742a-4a6a-a07a-ae0de0d7e29d" />
   
   Scheduler is working at 100% in both cases but before it was spending most of its resources on the ORM hydration and it  wouldn't hand enough tasks to workers.
   
   I've set max slot occupancy to 54. Before the change, at any given moment we would have up to 34% of the 54 slots occupied. After, we reach 100% occupancy and that didn't occur once but for most of the time.
   
   Red line is the worker slot occupancy.
   
   ### Time waiting for the scheduler vs workers
   
   <img width="3944" height="614" alt="image" src="https://github.com/user-attachments/assets/f9a521a9-48d3-4f24-b5ff-e33970749cc3" />
   
   At the end, tasks end up waiting for workers instead of the scheduler.
   
   ### Increased number of running tasks
   
   <img width="1992" height="624" alt="image" src="https://github.com/user-attachments/assets/64d41d65-7866-423e-a955-9c18b69db2b0" />
   
   ### Waiting on the scheduler (SCHEDULED -> QUEUED) vs waiting for the workers (QUEUED -> RUNNING)
   
   <img width="3968" height="1216" alt="image" src="https://github.com/user-attachments/assets/a52846cd-32df-44f0-a294-75dcea3a5267" />
   
   Tasks wait less on the scheduler and more for a worker.
   
   ### Same number of tasks, finish faster
   
   <img width="3970" height="624" alt="image" src="https://github.com/user-attachments/assets/740afcdd-2748-4349-81aa-dbcf969af398" />
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [X] Yes (please specify the tool below)
   
   Claude code Opus 4.8
   
   <!--
   Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3871061741


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1465,10 +1512,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)

Review Comment:
   I had a better look at this.
   
   > we don't need to query the database again as it's the same task
   
   We don't query the DB at all. We are looking it up in a cached dict.
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/serialization/definitions/dag.py#L264-L267
   
   and then the `task_dict`
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/serialization/definitions/dag.py#L138
   
   > We can also bulk update it on case a lot of mapped tasks are removed but I don't think it's worth it as it does not happen a lot
   
   This is how it works. If you look at the entire code snippet
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/dagrun.py#L1519C13-L1539C18
   
   ```python
   for row in rows:
       try:
           task = dag.get_task(row.task_id)
       except TaskNotFound:
           if row.state != TaskInstanceState.REMOVED:
               orphaned.append(row.task_id)
           continue
       finished.append(FinishedTI(**row._mapping, task=task))
   if orphaned:
       self.log.error("Failed to get task for finished tis %s. Marking them as removed.", orphaned)
       session.execute(
           update(TI)
           .where(
               TI.dag_id == self.dag_id,
               TI.run_id == self.run_id,
               TI.task_id.in_(orphaned),
               TI.state != TaskInstanceState.REMOVED,
           )
           .values(state=TaskInstanceState.REMOVED)
           .execution_options(synchronize_session=False)
       )
   ```
   
   This is an iteration and for every element, we append to `orphaned` and then do a bulk update query for all of them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-5428349129

   @Nataneljpwd Thank you for the review! I've addressed all your comments and I'll look at the optimization suggestion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864032954


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -140,14 +141,41 @@
 tracer = trace.get_tracer(__name__)
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    task_id: str
+    map_index: int
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.
+FINISHED_TI_COLUMNS = tuple(getattr(TI, f.name) for f in fields(FinishedTI) if f.name != "task")

Review Comment:
   `getattr` without a default param doesn't fail silently. Instead it throws an `AttributeError` exception.
   
   https://docs.python.org/3/library/functions.html#getattr



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4311963682


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None

Review Comment:
   TaskInstanceState? (Or even a base type with less cases?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4306258412


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),
+            )
+            .order_by(TI.task_id, TI.map_index)

Review Comment:
   Nit: `fetch_task_instances` orders by `(task_id, region_id, region_index)`, but this orders by `(task_id, map_index)`. Rows of the same task that share a `region_index` across loop regions now come back in no defined order, which can matter for the `max(..., key=end_date)` tie-breaks used for the Dag callbacks. Using the same ordering would keep the output stable.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),
+            )
+            .order_by(TI.task_id, TI.map_index)

Review Comment:
   Nit: `fetch_task_instances` orders by `(task_id, region_id, region_index)`, but this orders by `(task_id, map_index)`. Rows of the same task that share a `region_index` across loop regions now come back in no defined order, which can matter for the `max(..., key=end_date)` tie-breaks used for the Dag callbacks. Using the same ordering would keep the output stable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864218652


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1333,7 +1361,7 @@ def recalculate(self) -> _UnfinishedStates:
                 callback = self.produce_dag_callback(
                     dag=dag,
                     success=False,
-                    relevant_ti=ti_causing_failure,
+                    relevant_ti=self._ensure_type_task_instance(ti_causing_failure, session=session),

Review Comment:
   `produce_dag_callback` needs an actual TaskInstance ORM object and `_ensure_type_task_instance` is used to check if the passed ti is `FinishedTI` or `TaskInstance`. If it's `FinishedTI`, we fetch the info from the DB and then hydrate a `TaskInstance` ORM object.
   
   That's why the function is called 'ensure...` because if the type is right, it's a no-op, otherwise we do the conversion.
   
   Same for the other call sites.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4293653991


##########
airflow-core/src/airflow/models/task_coordinates.py:
##########
@@ -69,14 +69,30 @@ class _ProducerRequest:
 
 
 class TaskCoordinate(Protocol):
-    """Stored task identity shared by live and retained task data."""
+    """
+    Stored task identity shared by live and retained task data.
 
-    dag_id: str
-    run_id: str
-    task_id: str
-    region_id: UUID
-    region_index: int
-    dag_version_id: UUID | None
+    Read-only on purpose: resolvers only ever read coordinates, and declaring them as
+    properties lets immutable views (such as a frozen dataclass) satisfy the protocol.
+    """
+
+    @property

Review Comment:
   When these fields are accessed through the `FinishedTI` but the code expects `TaskCoordinate`, mypy expects them to be settable but they are read-only and we end up with an error.
   
   Making them properties with the `@property` decorator solves the issue and it seems to be fine for the current code but I don't have enough context to know if keeping this as read-only will be an issue for future plans.
   
   @ashb Any thoughts on this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4377898070


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        # Both queries restrict a partial Dag to the same subset, so no row outside it is
+        # mistaken for an orphan and marked REMOVED.
+        task_ids = DagRun._get_partial_task_ids(self.dag)
+        unfinished_tis = DagRun.fetch_task_instances(
+            dag_id=self.dag_id,
+            run_id=self.run_id,
+            task_ids=task_ids,
+            state=State.unfinished,
+            session=session,
+        )
+        finished_stmt = (
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),
+            )
+            .order_by(TI.task_id, TI.region_id, TI.region_index)
+        )
+        if task_ids is not None:
+            finished_stmt = finished_stmt.where(TI.task_id.in_(task_ids))

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864687633


##########
airflow-core/src/airflow/ti_deps/dep_context.py:
##########
@@ -107,20 +107,22 @@ class DepContext:
     fresh empty dict, so they would neither read the memo nor warm it for anything else.
     """
 
-    def ensure_finished_tis(self, dag_run: DagRun, session: Session) -> list[TaskInstance]:
+    def ensure_finished_tis(self, dag_run: DagRun, session: Session) -> list[TaskInstance | FinishedTI]:
         """
         Ensure finished_tis is populated if it's currently None, which allows running tasks without dag_run.
 
          :param dag_run: The DagRun for which to find finished tasks
          :return: A list of all the finished tasks of this DAG and logical_date
         """
+        finished_tis: list[TaskInstance | FinishedTI]
         if self.finished_tis is None:
-            finished_tis = dag_run.get_task_instances(state=State.finished, session=session)
-            for ti in finished_tis:
+            fetched = dag_run.get_task_instances(state=State.finished, session=session)
+            for ti in fetched:

Review Comment:
   The query result is `list[TaskInstance]` but the method returns `list[TaskInstance | FinishedTI]`.
   
   So we have to create a new list with the union type, otherwise mypy will complain about the assignment.
   
   We could keep a single `finished_tis` assignment but we would still need the `list()` conversion and the current code
   
   ```
   finished_tis = list(fetched)
   ```
   
   is easier to read than
   
   ```
   finished_tis = list(finished_tis)
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4391464875


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()

Review Comment:
   > Moving self.last_scheduling_decision = start_dttm to after the task_instance_scheduling_decisions call keeps this flush for the case it's there for (pending in-memory TI state changes) but makes it a no-op on the scheduler path.
   
   I tried this when @vatsrahul1001 commented about the increase in the queries and it didn't work.
   
   I'll explore the other suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4391221535


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.

Review Comment:
   I wasn't aware of this edge case. This was a very good comment. Thanks @ashb. I updated it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4392123155


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()

Review Comment:
   I updated it. Now we check if there is any TI that is `new`, `dirty` or `deleted` and only then we flush.
   
   The query count went down as expected. From what I can this isn't getting triggered today.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864708106


##########
airflow-core/src/airflow/ti_deps/deps/trigger_rule_dep.py:
##########
@@ -34,6 +34,7 @@
     from sqlalchemy.orm import Session
     from sqlalchemy.sql import ColumnElement
 
+    from airflow.models.dagrun import FinishedTI

Review Comment:
   I don't have a personal opinion on this.
   
   It made sense to me to keep the FinishedTI in the place where it's used but `taskinstance` also makes sense.
   
   I'm going to leave this question open in case someone else has a preference and wants to share any thoughts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4311969393


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1495,10 +1552,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []

Review Comment:
   This should use a set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4312899012


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(

Review Comment:
   @vatsrahul1001 Because of all the new task loops code, I wanted to rerun the benchmarks. I'm going to check this as well and get back with the results.
   
   When I initially tested the code in multiple scenarios including the one that you mentioned, I didn't notice any difference at all in the timing. It was neither good nor bad, just the same. I had to increase the load a lot to see the side-effects.
   
   The underlying code has changed a lot in the last days.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4312945822


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None

Review Comment:
   I actually got that from the `TaskInstance` model
   
   https://github.com/xBis7/airflow/blob/fd7ebe8bf657366b759f4cd22e79a2d45d378f02/airflow-core/src/airflow/models/taskinstance.py#L1122
   
   I think it's better to keep it consistent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864808558


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1465,10 +1512,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)
+                except TaskNotFound:
+                    if row.state != TaskInstanceState.REMOVED:
+                        orphaned.append(row.task_id)
+                    continue
+                finished.append(FinishedTI(**row._mapping, task=task))
+            if orphaned:
+                self.log.error("Failed to get task for finished tis %s. Marking them as removed.", orphaned)
+                session.execute(
+                    update(TI)
+                    .where(
+                        TI.dag_id == self.dag_id,
+                        TI.run_id == self.run_id,
+                        TI.task_id.in_(orphaned),
+                        TI.state != TaskInstanceState.REMOVED,
+                    )
+                    .values(state=TaskInstanceState.REMOVED)
+                    .execution_options(synchronize_session=False)
+                )
+            return finished
+
+        dag = self.get_dag()
+        unfinished_tis = list(_filter_tis_and_exclude_removed(dag, unfinished_tis))

Review Comment:
   It would change how scheduling works today.
   
   Look at the diff
   
   ```diff
   - tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
   -
   - unfinished_tis = [t for t in tis if t.state in State.unfinished]
   - finished_tis = [t for t in tis if t.state in State.finished]
   ```
   
   The current code makes sure that the behavior stays the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-5947710114

   @ephraimbuddy / @uranusjr / @ashb - I'll appreciate a second look, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-5948486533

   @vatsrahul1001 has also worked on a similar improvement in the past. Rahul please take a look when you get a chance. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4313439298


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+    @property
+    def map_index(self) -> int:
+        """Mirror the ``TaskInstance.map_index`` synonym of ``region_index``."""
+        return self.region_index
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.
+FINISHED_TI_COLUMNS = tuple(getattr(TI, f.name) for f in fields(FinishedTI) if f.name != "task")
+
+
 class TISchedulingDecision(NamedTuple):
     """Type of return for DagRun.task_instance_scheduling_decisions."""
 
-    tis: list[TI]
+    tis: list[TI | FinishedTI]

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-6270494751

   Because of all the task changes lately, I re-ran the benchmark tests and results are still good.
   
   I also ran some tests with a small number of dag runs and tasks. The difference is negligible and it's actually in favor of the new code. The user won't know the difference but in the measurements, the total time is always 2-3 seconds less with the code in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864896049


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -140,14 +141,41 @@
 tracer = trace.get_tracer(__name__)
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    task_id: str
+    map_index: int
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.

Review Comment:
   To provide some context about the SSOT part, I was adding the `FinishedTI` fields by hand in a query and then I thought, if we later modify `FinishedTI`, we would need to also modify the query which can lead to bugs.
   
   So I added this variable to be used for accessing the `FinishedTI` fields.
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/dagrun.py#L1490
   
   > Also, makw it clear that the columns are taken from the TI
   
   You are referring to this part `getattr(TI,`, right?
   
   > I think It should be higher up with the field definitions and remove the part before it that just explains what's done in the line bellow
   
   In the `FinishedTI` class docstring?
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/dagrun.py#L146-L156



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -140,14 +141,41 @@
 tracer = trace.get_tracer(__name__)
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    task_id: str
+    map_index: int
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.

Review Comment:
   To provide some context about the SSOT part, I was adding the `FinishedTI` fields by hand in a query and then I thought, if we later modify `FinishedTI`, we will need to also modify the query which can lead to bugs.
   
   So I added this variable to be used for accessing the `FinishedTI` fields.
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/dagrun.py#L1490
   
   > Also, makw it clear that the columns are taken from the TI
   
   You are referring to this part `getattr(TI,`, right?
   
   > I think It should be higher up with the field definitions and remove the part before it that just explains what's done in the line bellow
   
   In the `FinishedTI` class docstring?
   
   https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/dagrun.py#L146-L156



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "Nataneljpwd (via GitHub)" <gi...@apache.org>.
Nataneljpwd commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3856794824


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -140,14 +141,41 @@
 tracer = trace.get_tracer(__name__)
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    task_id: str
+    map_index: int
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.

Review Comment:
   This comment is a little interesting I don't think that a reader would understand the implications, at first it looks like it just described what's done in code but then it states te SSOT part, I think It should be higher up with the field definitions and remove the part before it that just explains what's done in the line bellow 



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -140,14 +141,41 @@
 tracer = trace.get_tracer(__name__)
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    task_id: str
+    map_index: int
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.
+FINISHED_TI_COLUMNS = tuple(getattr(TI, f.name) for f in fields(FinishedTI) if f.name != "task")

Review Comment:
   Here if the field name does not exist for the TI, this will silently fail, maybe rather add it in the init to throw an error in that case



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1465,10 +1512,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)

Review Comment:
   Idk how lightweight this is, but if we order by task Id and map index, we may optimize it by just saving the last task and task id, if it is the same we don't need to query the database again as it's the same task
   Just an optional optimization that might speed it up even more 



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1355,15 +1383,15 @@ def recalculate(self) -> _UnfinishedStates:
             self.notify_dagrun_state_changed(msg="success")
 
             if dag.has_on_success_callback:
-                last_succeeded_ti: TI | None = max(
+                last_succeeded_ti: TI | FinishedTI | None = max(
                     (ti for ti in tis if ti.state == TaskInstanceState.SUCCESS),
                     key=lambda ti: ti.end_date or timezone.make_aware(datetime.min),
                     default=None,
                 )
                 callback = self.produce_dag_callback(
                     dag=dag,
                     success=True,
-                    relevant_ti=last_succeeded_ti,
+                    relevant_ti=self._ensure_type_task_instance(last_succeeded_ti, session=session),

Review Comment:
   Same as above, always finished ti



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1737,10 +1812,28 @@ def _expand_mapped_task_if_needed(ti: TI) -> Iterable[TI] | None:
 
         return ready_tis, changed_tis, expansion_happened
 
+    def _ensure_type_task_instance(self, ti: TI | FinishedTI | None, *, session: Session) -> TI | None:
+        """
+        Return ``ti`` as a full TaskInstance, re-fetching it when it is a FinishedTI view.
+
+        Full instances (and ``None``) pass through unchanged. Only the dag-callback path
+        needs the full row, and only when a dag_run reaches a terminal state, so the
+        re-fetch costs one query per finished dag_run rather than one per loop.
+        """
+        if not isinstance(ti, FinishedTI):
+            return ti
+        return DagRun.fetch_task_instance(
+            dag_id=self.dag_id,
+            dag_run_id=self.run_id,
+            task_id=ti.task_id,
+            map_index=ti.map_index,
+            session=session,
+        )

Review Comment:
   Is this used anywhere? As the 3 places I saw used it for no reason as it always was a finished ti



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1756,7 +1849,9 @@ def _are_premature_tis(
             dep_context.have_changed_ti_states,
         )
 
-    def _emit_true_scheduling_delay_stats_for_finished_state(self, finished_tis: list[TI]) -> None:
+    def _emit_true_scheduling_delay_stats_for_finished_state(
+        self, finished_tis: list[TI | FinishedTI]

Review Comment:
   Same as above



##########
airflow-core/src/airflow/ti_deps/dep_context.py:
##########
@@ -79,7 +79,7 @@ class DepContext:
     ignore_task_deps: bool = False
     ignore_ti_state: bool = False
     ignore_unmapped_tasks: bool = False
-    finished_tis: list[TaskInstance] | None = None
+    finished_tis: list[TaskInstance | FinishedTI] | None = None

Review Comment:
   Same as above



##########
airflow-core/src/airflow/ti_deps/dep_context.py:
##########
@@ -107,20 +107,22 @@ class DepContext:
     fresh empty dict, so they would neither read the memo nor warm it for anything else.
     """
 
-    def ensure_finished_tis(self, dag_run: DagRun, session: Session) -> list[TaskInstance]:
+    def ensure_finished_tis(self, dag_run: DagRun, session: Session) -> list[TaskInstance | FinishedTI]:
         """
         Ensure finished_tis is populated if it's currently None, which allows running tasks without dag_run.
 
          :param dag_run: The DagRun for which to find finished tasks
          :return: A list of all the finished tasks of this DAG and logical_date
         """
+        finished_tis: list[TaskInstance | FinishedTI]
         if self.finished_tis is None:
-            finished_tis = dag_run.get_task_instances(state=State.finished, session=session)
-            for ti in finished_tis:
+            fetched = dag_run.get_task_instances(state=State.finished, session=session)
+            for ti in fetched:

Review Comment:
   Why was the variable renamed? It had a better name before



##########
airflow-core/src/airflow/ti_deps/deps/trigger_rule_dep.py:
##########
@@ -58,7 +59,7 @@ class _UpstreamTIStates(NamedTuple):
     skipped_setup: int
 
     @classmethod
-    def calculate(cls, finished_upstreams: Iterator[TaskInstance]) -> _UpstreamTIStates:
+    def calculate(cls, finished_upstreams: Iterator[TaskInstance | FinishedTI]) -> _UpstreamTIStates:

Review Comment:
   Isn't it always finished tis? As above?



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -140,14 +141,41 @@
 tracer = trace.get_tracer(__name__)
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    task_id: str
+    map_index: int
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.

Review Comment:
   Also, makw it clear that the columns are taken from the TI



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1465,10 +1512,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)
+                except TaskNotFound:
+                    if row.state != TaskInstanceState.REMOVED:
+                        orphaned.append(row.task_id)
+                    continue
+                finished.append(FinishedTI(**row._mapping, task=task))
+            if orphaned:
+                self.log.error("Failed to get task for finished tis %s. Marking them as removed.", orphaned)
+                session.execute(
+                    update(TI)
+                    .where(
+                        TI.dag_id == self.dag_id,
+                        TI.run_id == self.run_id,
+                        TI.task_id.in_(orphaned),
+                        TI.state != TaskInstanceState.REMOVED,
+                    )
+                    .values(state=TaskInstanceState.REMOVED)
+                    .execution_options(synchronize_session=False)
+                )
+            return finished
+
+        dag = self.get_dag()
+        unfinished_tis = list(_filter_tis_and_exclude_removed(dag, unfinished_tis))

Review Comment:
   Can't we just not append the removed here? Or pop them on runtime?



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1355,15 +1383,15 @@ def recalculate(self) -> _UnfinishedStates:
             self.notify_dagrun_state_changed(msg="success")
 
             if dag.has_on_success_callback:
-                last_succeeded_ti: TI | None = max(
+                last_succeeded_ti: TI | FinishedTI | None = max(

Review Comment:
   Isn't it always a finished ti? If it's the last succeeded ti?



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1333,7 +1361,7 @@ def recalculate(self) -> _UnfinishedStates:
                 callback = self.produce_dag_callback(
                     dag=dag,
                     success=False,
-                    relevant_ti=ti_causing_failure,
+                    relevant_ti=self._ensure_type_task_instance(ti_causing_failure, session=session),

Review Comment:
   This ti is always failed, so always finished



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1737,10 +1812,28 @@ def _expand_mapped_task_if_needed(ti: TI) -> Iterable[TI] | None:
 
         return ready_tis, changed_tis, expansion_happened
 
+    def _ensure_type_task_instance(self, ti: TI | FinishedTI | None, *, session: Session) -> TI | None:
+        """
+        Return ``ti`` as a full TaskInstance, re-fetching it when it is a FinishedTI view.
+
+        Full instances (and ``None``) pass through unchanged. Only the dag-callback path
+        needs the full row, and only when a dag_run reaches a terminal state, so the
+        re-fetch costs one query per finished dag_run rather than one per loop.
+        """
+        if not isinstance(ti, FinishedTI):
+            return ti
+        return DagRun.fetch_task_instance(
+            dag_id=self.dag_id,
+            dag_run_id=self.run_id,
+            task_id=ti.task_id,
+            map_index=ti.map_index,
+            session=session,
+        )
+
     def _are_premature_tis(
         self,
         unfinished_tis: Sequence[TI],
-        finished_tis: list[TI],
+        finished_tis: list[TI | FinishedTI],

Review Comment:
   Can they not be finished tis? If so the type hint is wrong



##########
airflow-core/src/airflow/ti_deps/dep_context.py:
##########
@@ -107,20 +107,22 @@ class DepContext:
     fresh empty dict, so they would neither read the memo nor warm it for anything else.
     """
 
-    def ensure_finished_tis(self, dag_run: DagRun, session: Session) -> list[TaskInstance]:
+    def ensure_finished_tis(self, dag_run: DagRun, session: Session) -> list[TaskInstance | FinishedTI]:
         """
         Ensure finished_tis is populated if it's currently None, which allows running tasks without dag_run.
 
          :param dag_run: The DagRun for which to find finished tasks
          :return: A list of all the finished tasks of this DAG and logical_date
         """
+        finished_tis: list[TaskInstance | FinishedTI]

Review Comment:
   Same as above



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1629,7 +1704,7 @@ def execute_dag_callbacks(
     def _get_ready_tis(
         self,
         schedulable_tis: list[TI],
-        finished_tis: list[TI],
+        finished_tis: list[TI | FinishedTI],

Review Comment:
   How can a finished ti list contain non finished tis? 



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1465,10 +1512,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)

Review Comment:
   We can also bulk update it on case a lot of mapped tasks are removed but I don't think it's worth it as it does not happen a lot



##########
airflow-core/src/airflow/ti_deps/deps/trigger_rule_dep.py:
##########
@@ -34,6 +34,7 @@
     from sqlalchemy.orm import Session
     from sqlalchemy.sql import ColumnElement
 
+    from airflow.models.dagrun import FinishedTI

Review Comment:
   Why is it in models.dagrun? Maybe it is worth moving to the task instance model file?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4312945822


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None

Review Comment:
   I actually got that from the `TaskInstance` model
   
   https://github.com/xBis7/airflow/blob/fd7ebe8bf657366b759f4cd22e79a2d45d378f02/airflow-core/src/airflow/models/taskinstance.py#L1122
   
   I'd rather keep that consistent because the values are populated based on the `TaskInstance` and any change in the model might break this.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None

Review Comment:
   I actually got that from the `TaskInstance` model
   
   https://github.com/xBis7/airflow/blob/fd7ebe8bf657366b759f4cd22e79a2d45d378f02/airflow-core/src/airflow/models/taskinstance.py#L1122
   
   I'd rather keep it consistent because the values are populated based on the `TaskInstance` and any change in the model might break this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4359011003


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.

Review Comment:
   "Finished ones are never mutated during scheduling" doesn't hold when a mapped task's expansion shrinks. `_revise_map_indexes_if_mapped` (reached from `_get_ready_tis` for a schedulable mapped TI) bulk-UPDATEs the indexes past the new length to REMOVED. On main those finished TIs were identity-mapped ORM objects, so the UPDATE's default `synchronize_session` refreshed them, and downstream tasks evaluated later in the same `_get_ready_tis` pass saw REMOVED. A `FinishedTI` keeps the old state.
   
   Repro: `work.expand(arg=producer())` over 3 items, with `zreport` (ALL_SUCCESS) downstream. `work[1]` is SUCCESS and `work[2]` FAILED; the user clears `work[0]` with upstream, so `producer` reruns and now returns 2 items. On the next pass the revise for `work[0]` marks `work[2]` REMOVED, but `zreport` still counts it as FAILED and goes `upstream_failed`, so the run fails. On main `zreport` stays unset and waits for `work[0]`. With ONE_FAILED, `zreport` gets scheduled here and not on main.
   
   Main already gets this wrong when the downstream's task_id sorts before `work`; this change makes it happen regardless of task order.
   
   One fix: have `_revise_map_indexes_if_mapped` return the `(task_id, region_id, indexes)` it removed, and swap the matching entries in `finished_tis` (the same list object as `dep_context.finished_tis`) for `dataclasses.replace(fti, state=TaskInstanceState.REMOVED)`, with a test for the scenario above. At minimum this comment shouldn't say they're never mutated.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()

Review Comment:
   On the scheduler path the only pending change at this point is `last_scheduling_decision`, which `update_state` sets just before calling this (line 1404). So this flush turns what used to be one batched `UPDATE dag_run` at commit into one UPDATE per dag run per loop. That is the cost the "We do not flush here for performance reasons" comment at the end of `update_state` is avoiding.
   
   Actual statement counts for the `@once` 10x10 rows in `TestSchedulerJobQueriesCount` (SQLite, recorded rather than checked against the margin):
   
   - main: 208/67/60/60 (no_structure), 218/74/74/74 (linear)
   - this PR: 227/86/79/79, 237/93/93/93
   - this PR without the flush: 218/77/70/70, 228/84/84/84
   
   So about half the extra statements per loop come from this flush; the rest is the second SELECT, which is the expected cost of the split.
   
   Moving `self.last_scheduling_decision = start_dttm` to after the `task_instance_scheduling_decisions` call keeps this flush for the case it's there for (pending in-memory TI state changes) but makes it a no-op on the scheduler path. Alternatively, only flush when the session holds a new or dirty `TaskInstance`. Either way the expected counts in `test_scheduler_job.py` can then come back down; even as it stands, the linear row would pass on SQLite without being raised, because of `margin=15`.



##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -5041,6 +5056,35 @@ def my_teardown():
     }
 
 
+def test_scheduling_decisions_mark_orphaned_finished_tis_removed(dag_maker, session):
+    """A finished ti whose task no longer exists in the dag is marked REMOVED and excluded."""
+    with dag_maker(session=session):
+        EmptyOperator(task_id="stays")
+        EmptyOperator(task_id="vanishes")
+    dr = dag_maker.create_dagrun()
+    for ti in dr.get_task_instances(session=session):
+        ti.state = TaskInstanceState.SUCCESS
+    session.flush()
+
+    del dr.dag.task_dict["vanishes"]
+    decision = dr.task_instance_scheduling_decisions(session=session)
+
+    assert len(decision.finished_tis) == 1
+    finished_ti = decision.finished_tis[0]
+    assert isinstance(finished_ti, FinishedTI)
+    assert finished_ti.task_id == "stays"
+    assert finished_ti.state == TaskInstanceState.SUCCESS
+    assert finished_ti.task is dr.dag.task_dict["stays"]
+    # The view carries the row's identity and coordinates so that trigger-rule evaluation
+    # and the dag-callback re-fetch never need the full TaskInstance.
+    stays = dr.get_task_instance("stays", session=session)
+    assert (finished_ti.id, finished_ti.dag_id, finished_ti.run_id) == (stays.id, dr.dag_id, dr.run_id)
+    assert (finished_ti.region_id, finished_ti.region_index) == (stays.region_id, stays.region_index)
+    assert finished_ti.dag_version_id == stays.dag_version_id
+    session.expire_all()
+    assert dr.get_task_instance("vanishes", session=session).state == TaskInstanceState.REMOVED

Review Comment:
   This pins the orphan path but not the UPDATE's scope: dropping the `TI.dag_id`/`TI.run_id` predicates or the `TI.state != REMOVED` guard from that UPDATE leaves the suite green. A second dag run where `vanishes` is SUCCESS (asserting it stays SUCCESS), and making `vanishes` mapped with 2-3 indexes (asserting all become REMOVED), would cover it.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1495,10 +1608,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: set[str] = set()
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)
+                except TaskNotFound:
+                    if row.state != TaskInstanceState.REMOVED:
+                        orphaned.add(row.task_id)
+                    continue
+                finished.append(FinishedTI(**row._mapping, task=task))

Review Comment:
   Nit: `FINISHED_TI_COLUMNS` is built from `fields(FinishedTI)` in order, so `FinishedTI(*row, task=task)` is equivalent and skips building a dict per row. In a quick benchmark (1000 rows, SQLite) that halved the construction cost.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,95 @@ def validate_loop_decision(stop: bool | None, region_index: int, group: Serializ
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps identity equality and ``__hash__`` mirrors the key hash, the same as
+    a TaskInstance.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+    @property
+    def map_index(self) -> int:
+        """Mirror the ``TaskInstance.map_index`` synonym of ``region_index``."""
+        return self.region_index
+
+    def __hash__(self) -> int:
+        return hash((self.task_id, self.dag_id, self.run_id, self.region_index))
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.
+FINISHED_TI_COLUMNS = tuple(getattr(TI, f.name) for f in fields(FinishedTI) if f.name != "task")
+
+
+class SchedulingTI(Protocol):

Review Comment:
   Non-blocking question: `SchedulingTI` repeats the six properties of `TaskCoordinate`, which this PR made read-only so `FinishedTI` could satisfy it. Would `class SchedulingTI(TaskCoordinate, Protocol)` adding only `id`, `map_index`, `state`, `start_date`, `end_date` and `task` be better? It would have to live in `task_coordinates.py` (that module imports `dagrun` at runtime) and be imported under `TYPE_CHECKING` here.
   
   Related: `NotPreviouslySkippedDep._mapped_group_skip_decisions` (`not_previously_skipped_dep.py:214`) also receives `ensure_finished_tis()` but is still annotated `list[TaskInstance]`. mypy doesn't flag it because its caller is unannotated.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        # Both queries restrict a partial Dag to the same subset, so no row outside it is
+        # mistaken for an orphan and marked REMOVED.
+        task_ids = DagRun._get_partial_task_ids(self.dag)
+        unfinished_tis = DagRun.fetch_task_instances(
+            dag_id=self.dag_id,
+            run_id=self.run_id,
+            task_ids=task_ids,
+            state=State.unfinished,
+            session=session,
+        )
+        finished_stmt = (
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),
+            )
+            .order_by(TI.task_id, TI.region_id, TI.region_index)
+        )
+        if task_ids is not None:
+            finished_stmt = finished_stmt.where(TI.task_id.in_(task_ids))

Review Comment:
   This repeats `fetch_task_instances`' WHERE/ORDER BY by hand, and the orphan marking below relies on the two staying identical (as the comment at 1572 says). They have already drifted once in this PR: the missing partial-Dag filter in an earlier revision would have marked rows outside the subset REMOVED. Could `fetch_task_instances` take the columns to select (or a shared helper build the statement) so the filter exists in one place?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4382035019


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,95 @@ def validate_loop_decision(stop: bool | None, region_index: int, group: Serializ
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps identity equality and ``__hash__`` mirrors the key hash, the same as
+    a TaskInstance.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+    @property
+    def map_index(self) -> int:
+        """Mirror the ``TaskInstance.map_index`` synonym of ``region_index``."""
+        return self.region_index
+
+    def __hash__(self) -> int:
+        return hash((self.task_id, self.dag_id, self.run_id, self.region_index))
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.
+FINISHED_TI_COLUMNS = tuple(getattr(TI, f.name) for f in fields(FinishedTI) if f.name != "task")
+
+
+class SchedulingTI(Protocol):

Review Comment:
   Yes, it's simpler this way. I moved it under `task_coordinates` and I also updated `_mapped_group_skip_decisions`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864468948


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1737,10 +1812,28 @@ def _expand_mapped_task_if_needed(ti: TI) -> Iterable[TI] | None:
 
         return ready_tis, changed_tis, expansion_happened
 
+    def _ensure_type_task_instance(self, ti: TI | FinishedTI | None, *, session: Session) -> TI | None:
+        """
+        Return ``ti`` as a full TaskInstance, re-fetching it when it is a FinishedTI view.
+
+        Full instances (and ``None``) pass through unchanged. Only the dag-callback path
+        needs the full row, and only when a dag_run reaches a terminal state, so the
+        re-fetch costs one query per finished dag_run rather than one per loop.
+        """
+        if not isinstance(ti, FinishedTI):
+            return ti
+        return DagRun.fetch_task_instance(
+            dag_id=self.dag_id,
+            dag_run_id=self.run_id,
+            task_id=ti.task_id,
+            map_index=ti.map_index,
+            session=session,
+        )

Review Comment:
   This actually checks the type and if it's a `FinishedTI`, it converts it back to a `TaskInstance` ORM object.
   
   It doesn't make sure that we still have a `FinishedTI` but the opposite.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4313444115


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.

Review Comment:
   You are right. I fixed it



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1495,10 +1552,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: list[str] = []

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4380902491


##########
airflow-core/tests/unit/models/test_dagrun.py:
##########
@@ -5041,6 +5056,35 @@ def my_teardown():
     }
 
 
+def test_scheduling_decisions_mark_orphaned_finished_tis_removed(dag_maker, session):
+    """A finished ti whose task no longer exists in the dag is marked REMOVED and excluded."""
+    with dag_maker(session=session):
+        EmptyOperator(task_id="stays")
+        EmptyOperator(task_id="vanishes")
+    dr = dag_maker.create_dagrun()
+    for ti in dr.get_task_instances(session=session):
+        ti.state = TaskInstanceState.SUCCESS
+    session.flush()
+
+    del dr.dag.task_dict["vanishes"]
+    decision = dr.task_instance_scheduling_decisions(session=session)
+
+    assert len(decision.finished_tis) == 1
+    finished_ti = decision.finished_tis[0]
+    assert isinstance(finished_ti, FinishedTI)
+    assert finished_ti.task_id == "stays"
+    assert finished_ti.state == TaskInstanceState.SUCCESS
+    assert finished_ti.task is dr.dag.task_dict["stays"]
+    # The view carries the row's identity and coordinates so that trigger-rule evaluation
+    # and the dag-callback re-fetch never need the full TaskInstance.
+    stays = dr.get_task_instance("stays", session=session)
+    assert (finished_ti.id, finished_ti.dag_id, finished_ti.run_id) == (stays.id, dr.dag_id, dr.run_id)
+    assert (finished_ti.region_id, finished_ti.region_index) == (stays.region_id, stays.region_index)
+    assert finished_ti.dag_version_id == stays.dag_version_id
+    session.expire_all()
+    assert dr.get_task_instance("vanishes", session=session).state == TaskInstanceState.REMOVED

Review Comment:
   Added the missing tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-6454781003

   @ashb I addressed all of your comments and I also re-ran all the benchmark tests. Results are still looking good. No issues from the new changes.
   
   Please take another look when you get a chance. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4392123155


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1561,39 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()

Review Comment:
   I updated it. Now we check if there is any TI that is `new`, `dirty` or `deleted` and only then we flush.
   
   The query count went down as expected. From what I can see this isn't getting triggered today.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb merged PR #71737:
URL: https://github.com/apache/airflow/pull/71737


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r3864458639


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1355,15 +1383,15 @@ def recalculate(self) -> _UnfinishedStates:
             self.notify_dagrun_state_changed(msg="success")
 
             if dag.has_on_success_callback:
-                last_succeeded_ti: TI | None = max(
+                last_succeeded_ti: TI | FinishedTI | None = max(

Review Comment:
   In all places where we accept a union `TI | FinishedTI` and there is the same question, `since this is a FinishedTI why don't we just accept that`, if the caller is the scheduler, then yes it will be a `FinishedTI`. But there are other callers as well, e.g. the UI and in these cases we need an actual `TaskInstance`.
   
   Not to mention that enforcing all callers to use a `FinishedTI` would blow the scope of this PR and it would also make it very hard to ensure the same behavior as now. It would require extensive testing and even after that, there would be some edge case with a bug.
   
   IMO, it's not worth it because these paths aren't hot like the scheduler loop.
   
   Here are some examples, where we call these functions with an actual TI outside of the scheduler loop.
   
   1. REST API, `get_task_instance_dependencies`
   
       line 352 it calls `get_failed_dep_statuses`
       
       https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/api_fastapi/core_api/routes/public/task_instances.py#L352
       
       `get_failed_dep_statuses` calls `get_dep_statuses` for every context while iterating
       
       https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/taskinstance.py#L1180
       
       `_get_dep_statuses` for `TriggerRuleDep` calls `_evaluate_trigger_rule`
       
       https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/ti_deps/deps/trigger_rule_dep.py#L117
       
       which eventually calls `ensure_finished_tis` in multiple places and in these tis are full ORM objects.
       
       https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/ti_deps/dep_context.py#L110-L129
   
   2. CLI, `task_failed_deps`
   
         https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/cli/commands/task_command.py#L284
         
         It ends down the same path as above
   
   3. Before executing a task, `_check_and_change_state_before_execution`
   
       These 2 sets
       
       https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/taskinstance.py#L1370-L1378
       
       https://github.com/xBis7/airflow/blob/edd3176d01674ecc917ec4270a9a0b5c8fa06eb8/airflow-core/src/airflow/models/taskinstance.py#L1402-L1410
       
       don't contain FinishedTIs.
   
   
    
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-5947098891

   @shahar1 Thank you for the review! I've addressed all the comments but the threads have been open without any activity for more than a month, so I resolved them.
   
   @Nataneljpwd If any of my answers don't make sense to you or you just disagree, add a comment to the corresponding thread and I will unresolve it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4311955411


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.

Review Comment:
   This is not right, TaskInstance hash by key
   
   https://github.com/apache/airflow/blob/2be399f1418bd20cf3b72d3036e33897633df2aa/airflow-core/src/airflow/models/taskinstance.py#L1321-L1322



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4306257554


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(

Review Comment:
   Question on small runs: the query-count expectations in `test_scheduler_job.py` go from `[218, 69, 69, 69]` to `[218, 97, 89, 89]` (`@once`, `no_structure`) and from `[228, 84, 84, 84]` to `[228, 104, 104, 104]` (`linear`), about two extra queries per Dag run per loop. The benchmark covers wide mapped runs where hydration dominates. For typical small runs, especially with a remote database, the extra round trips may cost more than the hydration saves. Is there a measurement for small and medium runs, or a way to skip the second query when a run has few task instances?



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),
+            )
+            .order_by(TI.task_id, TI.map_index)

Review Comment:
   Nit: `fetch_task_instances` orders by `(task_id, region_id, region_index)`, but this orders by `(task_id, map_index)`. Rows of the same task that share a `region_index` across loop regions now come back in no defined order, which can matter for the `max(..., key=end_date)` tie-breaks used for the Dag callbacks. Using the same ordering would keep the output stable.



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),

Review Comment:
   Optional: the old path went through `get_task_instances`, which applies `DagRun._get_partial_task_ids(self.dag)` for a partial Dag. This query doesn't, so for a partial Dag, finished rows outside the subset would hit `TaskNotFound` in `_build_finished_tis` and be updated to `REMOVED`, which the old code never did. The scheduler and `dag.test` use full Dags today, so I don't think it is reachable, but adding the same `task_ids` filter here would keep the invariant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #71737:
URL: https://github.com/apache/airflow/pull/71737#issuecomment-6362238329

   Removing the backport label for now as it's not worth including this in 3.4.0. We will backport it once .0 is released.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4379480430


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1495,10 +1608,38 @@ def _filter_tis_and_exclude_removed(dag: SerializedDAG, tis: list[TI]) -> Iterab
                 else:
                     yield ti
 
-        tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
+        def _build_finished_tis(dag: SerializedDAG, rows) -> list[FinishedTI]:
+            """Attach the serialized task, dropping (and marking REMOVED) rows whose task is gone."""
+            finished: list[FinishedTI] = []
+            orphaned: set[str] = set()
+            for row in rows:
+                try:
+                    task = dag.get_task(row.task_id)
+                except TaskNotFound:
+                    if row.state != TaskInstanceState.REMOVED:
+                        orphaned.add(row.task_id)
+                    continue
+                finished.append(FinishedTI(**row._mapping, task=task))

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4313595052


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None

Review Comment:
   I don't think that there is any case where `state` can be `None` for this. The model has it because the `TaskInstance` can be `None`. We can make it more tight by changing it to `state: str`. 
   
   @uranusjr What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4315428123


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(

Review Comment:
   I measured it with 30 dags with 11 tasks each, all dependencies linear.
   
   And the difference is negligible and it's actually in favor of the new code. I run it multiple times and the new code is always 2-3 seconds faster.
   
   It makes sense because the debug spans from this PR https://github.com/apache/airflow/pull/69809 showed in my initial investigation that the query took 10% of the time and the ORM hydration 90%. The queries are very fast and python code is expensive.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4312642796


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),

Review Comment:
   Done



##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -1479,8 +1517,27 @@ def recalculate(self) -> _UnfinishedStates:
 
     @provide_session
     def task_instance_scheduling_decisions(self, *, session: Session = NEW_SESSION) -> TISchedulingDecision:
-        tis = self.get_task_instances(session=session, state=State.task_states)
-        self.log.debug("number of tis tasks for %s: %s task(s)", self, len(tis))
+        # Fetch finished task instances as lightweight read-only rows and only the
+        # unfinished ones as full ORM objects. Finished ones are never mutated during
+        # scheduling, so hydrating them is pure overhead.
+        #
+        # Flush here to avoid task instances ending up in the wrong list or in both,
+        # in case of a pending in-session state change.
+        # The airflow session maker sets autoflush=False (ref settings.configure_orm).
+        session.flush()
+        unfinished_tis = self.get_task_instances(session=session, state=State.unfinished)
+        finished_rows = session.execute(
+            select(*FINISHED_TI_COLUMNS)
+            .where(
+                TI.dag_id == self.dag_id,
+                TI.run_id == self.run_id,
+                TI.state.in_(State.finished),
+            )
+            .order_by(TI.task_id, TI.map_index)

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make finished TIs more lightweight when scanning task instances for scheduling [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71737:
URL: https://github.com/apache/airflow/pull/71737#discussion_r4311936108


##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -163,14 +164,51 @@ def validate_loop_decision(decision: str | None, region_index: int, group: Seria
         raise InvalidLoopDecision("Fixed-count loop decision does not match its iteration limit")
 
 
+@dataclass(frozen=True, slots=True, eq=False)
+class FinishedTI:
+    """
+    Immutable view of a task instance in a terminal state.
+
+    Keeps the dag_run rescan lightweight and fast: scheduling only reads these columns
+    off finished task instances, so a fully instrumented TaskInstance per row is
+    unnecessary. ``task`` is attached from the serialized dag rather than the row.
+
+    ``eq=False`` keeps the same identity-based equality and hashing a TaskInstance has.
+
+    Anything needing a full TaskInstance must re-fetch it (see ``_ensure_type_task_instance``).
+    """
+
+    id: UUID
+    dag_id: str
+    run_id: str
+    task_id: str
+    region_id: UUID
+    region_index: int
+    dag_version_id: UUID | None
+    state: str | None
+    start_date: datetime | None
+    end_date: datetime | None
+    task: Operator | None = None
+
+    @property
+    def map_index(self) -> int:
+        """Mirror the ``TaskInstance.map_index`` synonym of ``region_index``."""
+        return self.region_index
+
+
+# Get the necessary columns from the FinishedTI fields so that the class stays
+# the single source of truth. For any needed change, add or remove a field there.
+FINISHED_TI_COLUMNS = tuple(getattr(TI, f.name) for f in fields(FinishedTI) if f.name != "task")
+
+
 class TISchedulingDecision(NamedTuple):
     """Type of return for DagRun.task_instance_scheduling_decisions."""
 
-    tis: list[TI]
+    tis: list[TI | FinishedTI]

Review Comment:
   Instead of having this union everywhere, we should create a base protocol or type to represent the shape instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org