You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ephraimbuddy (via GitHub)" <gi...@apache.org> on 2026/05/14 13:32:53 UTC
[PR] Mark Dags stale when their bundle is removed from config [airflow]
ephraimbuddy opened a new pull request, #66948:
URL: https://github.com/apache/airflow/pull/66948
When a Dag bundle is removed from the bundle config, sync_bundles_to_db flipped the bundle's active flag to False but left its Dags with is_stale=False. The processor stops parsing files for inactive bundles, so the time-based check in deactivate_stale_dags never fired for them.
deactivate_stale_dags now reads the set of active bundles from the DagBundleModel table and treats any non-stale Dag whose bundle is not active as stale, in addition to the existing last_parsed_time check for Dags in active bundles. If the bundle reappears in config later, the existing parse path resets is_stale to False per Dag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mark Dags stale when their bundle is removed from config [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #66948:
URL: https://github.com/apache/airflow/pull/66948#discussion_r3246796974
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -411,17 +406,29 @@ def deactivate_stale_dags(
):
"""Detect and deactivate DAGs which are no longer present in files."""
to_deactivate = set()
- bundle_names = {b.name for b in self._dag_bundles}
+ active_bundles = set(
+ session.scalars(select(DagBundleModel.name).where(DagBundleModel.active.is_(True))).all()
+ )
query = select(
DagModel.dag_id,
DagModel.bundle_name,
DagModel.fileloc,
DagModel.last_parsed_time,
DagModel.relative_fileloc,
- ).where(~DagModel.is_stale, DagModel.bundle_name.in_(bundle_names))
+ ).where(~DagModel.is_stale)
dags_parsed = session.execute(query)
for dag in dags_parsed:
+ # DAGs whose bundle has been removed from config (bundle no longer active) are stale —
+ # the processor has stopped parsing their files, so the time-based check below would never fire.
+ if dag.bundle_name not in active_bundles:
+ self.log.info(
+ "Deactivating DAG %s. Its bundle %s is no longer active.",
+ dag.dag_id,
Review Comment:
```suggestion
# Dags whose bundle has been removed from config (bundle no longer active) are stale —
# the processor has stopped parsing their files, so the time-based check below would never fire.
if dag.bundle_name in inactive_bundles:
self.log.info(
"Deactivating Dag %s. Its bundle %s is no longer active.",
dag.dag_id,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mark Dags stale when their bundle is removed from config [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66948:
URL: https://github.com/apache/airflow/pull/66948#issuecomment-4458903475
### Backport successfully created: v3-2-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/pull/66985"><img src="https://img.shields.io/badge/PR-66985-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mark Dags stale when their bundle is removed from config [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #66948:
URL: https://github.com/apache/airflow/pull/66948#discussion_r3246796974
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -411,17 +406,29 @@ def deactivate_stale_dags(
):
"""Detect and deactivate DAGs which are no longer present in files."""
to_deactivate = set()
- bundle_names = {b.name for b in self._dag_bundles}
+ active_bundles = set(
+ session.scalars(select(DagBundleModel.name).where(DagBundleModel.active.is_(True))).all()
+ )
query = select(
DagModel.dag_id,
DagModel.bundle_name,
DagModel.fileloc,
DagModel.last_parsed_time,
DagModel.relative_fileloc,
- ).where(~DagModel.is_stale, DagModel.bundle_name.in_(bundle_names))
+ ).where(~DagModel.is_stale)
dags_parsed = session.execute(query)
for dag in dags_parsed:
+ # DAGs whose bundle has been removed from config (bundle no longer active) are stale —
+ # the processor has stopped parsing their files, so the time-based check below would never fire.
+ if dag.bundle_name not in active_bundles:
+ self.log.info(
+ "Deactivating DAG %s. Its bundle %s is no longer active.",
+ dag.dag_id,
Review Comment:
```suggestion
# Dags whose bundle has been removed from config (bundle no longer active) are stale —
# the processor has stopped parsing their files, so the time-based check below would never fire.
if dag.bundle_name not in active_bundles:
self.log.info(
"Deactivating Dag %s. Its bundle %s is no longer active.",
dag.dag_id,
```
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -411,17 +406,29 @@ def deactivate_stale_dags(
):
"""Detect and deactivate DAGs which are no longer present in files."""
to_deactivate = set()
- bundle_names = {b.name for b in self._dag_bundles}
+ active_bundles = set(
+ session.scalars(select(DagBundleModel.name).where(DagBundleModel.active.is_(True))).all()
+ )
Review Comment:
we use "not" in the following operations. should we just make it something like this
```suggestion
inactive_bundles = set(
session.scalars(select(DagBundleModel.name).where(DagBundleModel.active.is_(False))).all()
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mark Dags stale when their bundle is removed from config [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #66948:
URL: https://github.com/apache/airflow/pull/66948#discussion_r3246985804
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -411,17 +406,29 @@ def deactivate_stale_dags(
):
"""Detect and deactivate DAGs which are no longer present in files."""
to_deactivate = set()
- bundle_names = {b.name for b in self._dag_bundles}
+ active_bundles = set(
+ session.scalars(select(DagBundleModel.name).where(DagBundleModel.active.is_(True))).all()
+ )
Review Comment:
Makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mark Dags stale when their bundle is removed from config [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy merged PR #66948:
URL: https://github.com/apache/airflow/pull/66948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org