You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "fat-catTW (via GitHub)" <gi...@apache.org> on 2026/07/13 15:03:42 UTC
[PR] Read fresh Dag bundle state during refresh [airflow]
fat-catTW opened a new pull request, #69830:
URL: https://github.com/apache/airflow/pull/69830
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
## Why:
Dag processors may need to observe Dag bundle refresh state written by another processor while reusing an existing SQLAlchemy session. Loading `DagBundleModel` as an ORM entity can reuse an object already present in the session identity map, which risks reading stale `last_refreshed` or `version` values.
## Solution:
Select only the needed bundle state columns instead of loading the full ORM entity. This returns fresh row values for `last_refreshed` and `version` without relying on an existing `DagBundleModel` object in the session.
Added a regression test that first reads bundle state in one session, updates the same row from another session, and verifies the manager observes the updated values.
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- `[X]` Yes (please specify the tool below)
Generated-by: [Codex] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on code in PR #69830:
URL: https://github.com/apache/airflow/pull/69830#discussion_r3604713670
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -2712,6 +2712,30 @@ def test_get_bundle_state_returns_correct_state(self, session):
assert state == BundleState(last_refreshed=refreshed_at, version="v1")
+ def test_get_bundle_state_reads_latest_database_values(self, session):
+ bundle_name = "test_fresh_state_bundle"
+ initial_refreshed_at = timezone.datetime(2024, 1, 15, 12, 0, 0)
+ refreshed_at = timezone.datetime(2024, 1, 16, 12, 0, 0)
+ model = DagBundleModel(name=bundle_name, version="old")
+ model.last_refreshed = initial_refreshed_at
+ session.add(model)
+ session.commit()
+
+ manager = DagFileProcessorManager(max_runs=1)
+ state = manager.get_bundle_state(bundle_name, session=session)
+
+ assert state == BundleState(last_refreshed=initial_refreshed_at, version="old")
+
+ with create_session() as update_session:
+ update_model = update_session.get(DagBundleModel, bundle_name)
+ assert update_model is not None
+ update_model.last_refreshed = refreshed_at
+ update_model.version = "fresh"
+
+ state = manager.get_bundle_state(bundle_name, session=session)
Review Comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #69830:
URL: https://github.com/apache/airflow/pull/69830#issuecomment-5068245004
LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 merged PR #69830:
URL: https://github.com/apache/airflow/pull/69830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on PR #69830:
URL: https://github.com/apache/airflow/pull/69830#issuecomment-5068384549
Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on code in PR #69830:
URL: https://github.com/apache/airflow/pull/69830#discussion_r3589806447
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -815,7 +815,6 @@ def _refresh_dag_bundles(self, known_files: dict[str, set[DagFileInfo]]):
except AirflowException as e:
self.log.exception("Error initializing bundle %s: %s", bundle.name, e)
continue
- # TODO: AIP-66 test to make sure we get a fresh record from the db and it's not cached
Review Comment:
Thank you for reviewing.
Yes. The change implements this TODO.
The lookup now avoid loading a cached ORM entity, and the added regression test verifies that the manager reads updated bundle state from another session.
That is why the TODO is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69830:
URL: https://github.com/apache/airflow/pull/69830#issuecomment-5068260698
### Backport successfully created: v3-3-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/pull/70374"><img src="https://img.shields.io/badge/PR-70374-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69830:
URL: https://github.com/apache/airflow/pull/69830#discussion_r3572261006
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -815,7 +815,6 @@ def _refresh_dag_bundles(self, known_files: dict[str, set[DagFileInfo]]):
except AirflowException as e:
self.log.exception("Error initializing bundle %s: %s", bundle.name, e)
continue
- # TODO: AIP-66 test to make sure we get a fresh record from the db and it's not cached
Review Comment:
Is this part of the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Read fresh Dag bundle state during refresh [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #69830:
URL: https://github.com/apache/airflow/pull/69830#discussion_r3603037246
##########
airflow-core/tests/unit/dag_processing/test_manager.py:
##########
@@ -2712,6 +2712,30 @@ def test_get_bundle_state_returns_correct_state(self, session):
assert state == BundleState(last_refreshed=refreshed_at, version="v1")
+ def test_get_bundle_state_reads_latest_database_values(self, session):
+ bundle_name = "test_fresh_state_bundle"
+ initial_refreshed_at = timezone.datetime(2024, 1, 15, 12, 0, 0)
+ refreshed_at = timezone.datetime(2024, 1, 16, 12, 0, 0)
+ model = DagBundleModel(name=bundle_name, version="old")
+ model.last_refreshed = initial_refreshed_at
+ session.add(model)
+ session.commit()
+
+ manager = DagFileProcessorManager(max_runs=1)
+ state = manager.get_bundle_state(bundle_name, session=session)
+
+ assert state == BundleState(last_refreshed=initial_refreshed_at, version="old")
+
+ with create_session() as update_session:
+ update_model = update_session.get(DagBundleModel, bundle_name)
+ assert update_model is not None
+ update_model.last_refreshed = refreshed_at
+ update_model.version = "fresh"
+
+ state = manager.get_bundle_state(bundle_name, session=session)
Review Comment:
This test uses `create_session()` (scoped session) to simulate a second DB session, but `create_session()` defaults to `scoped=True` and will reuse the same thread-local `settings.Session` instance. That means the “update_session” may actually be the same session (and may even be closed/committed unexpectedly), so the test doesn’t reliably validate cross-session freshness. Also, on SQLite/MySQL the original session may be holding an open transaction snapshot from the first read; committing the original session before the second read makes the test deterministic while still keeping the identity map stale (`expire_on_commit=False`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org