You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "LakshmiSravyaVedantham (via GitHub)" <gi...@apache.org> on 2026/02/25 20:14:58 UTC

[PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

LakshmiSravyaVedantham opened a new pull request, #62482:
URL: https://github.com/apache/airflow/pull/62482

   ## Summary
   
   Closes #62025
   
   The `get_dag_runs` REST API endpoint experiences slow response times on large Airflow deployments.
   
   ## Root Cause
   
   `eager_load_dag_run_for_validation()` used `selectinload` on `DagRun.task_instances` and `DagRun.task_instances_histories` without restricting which columns were fetched. This caused SQLAlchemy to load **all** heavyweight `TaskInstance` columns — including `executor_config` (a pickled binary blob), `hostname`, `rendered_map_index`, `context_carrier`, and many others — for every task instance across every DAG run in the result page.
   
   In practice, only `dag_version_id` (the foreign key) is needed to traverse the `TaskInstance.dag_version` association proxy used by the `DagRun.dag_versions` property, which drives `DAGRunResponse.dag_versions` serialization.
   
   ## Fix
   
   Add `load_only(TaskInstance.dag_version_id)` and `load_only(TaskInstanceHistory.dag_version_id)` to the respective `selectinload` chains. SQLAlchemy will then issue a `SELECT` for task instances that fetches only the identity columns and `dag_version_id`, rather than the full row. The subsequent `.joinedload(TaskInstance.dag_version).joinedload(DagVersion.bundle)` chain continues to work correctly.
   
   ```python
   # Before
   selectinload(DagRun.task_instances)
       .joinedload(TaskInstance.dag_version)
       .joinedload(DagVersion.bundle),
   
   # After
   selectinload(DagRun.task_instances)
       .load_only(TaskInstance.dag_version_id)
       .joinedload(TaskInstance.dag_version)
       .joinedload(DagVersion.bundle),
   ```
   
   The same pattern is applied to `task_instances_histories`.
   
   ## Impact
   
   On deployments with many task instances per DAG run, this change significantly reduces:
   - The volume of data transferred from the database per request
   - Memory pressure from deserializing large pickled `executor_config` values
   - Response latency for `GET /dags/{dag_id}/dagRuns` and the batch `POST /dags/~/dagRuns/list` endpoints
   
   ## Testing
   
   - Existing tests in `tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py` continue to pass
   - The `dag_versions` property continues to work correctly since `load_only()` restricts deferred columns, not the relationship traversal
   
   ## References
   
   - [SQLAlchemy `load_only` docs](https://docs.sqlalchemy.org/en/20/orm/queryguide/columns.html#column-deferral-using-load-only)
   - [SQLAlchemy loading strategies](https://docs.sqlalchemy.org/en/20/orm/queryguide/relationships.html)
   - Fixes: #62025


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.

pierrejeambrun closed pull request #62482: perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance
URL: https://github.com/apache/airflow/pull/62482


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.

boring-cyborg[bot] commented on PR #62482:
URL: https://github.com/apache/airflow/pull/62482#issuecomment-4012060881

   Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.

boring-cyborg[bot] commented on PR #62482:
URL: https://github.com/apache/airflow/pull/62482#issuecomment-3961779283

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.

pierrejeambrun commented on PR #62482:
URL: https://github.com/apache/airflow/pull/62482#issuecomment-4010967921

   I just pushed a commit to fix the static checks, it should look better 🤞 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #62482:
URL: https://github.com/apache/airflow/pull/62482#issuecomment-4012068964

   ### Backport successfully created: v3-1-test
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
           <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>✅</td>
           <td>v3-1-test</td>
           <td><a href="https://github.com/apache/airflow/pull/62996"><img src="https://img.shields.io/badge/PR-62996-blue" alt="PR Link"></a></td>
       </tr>
   </table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.

pierrejeambrun merged PR #62482:
URL: https://github.com/apache/airflow/pull/62482


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org