You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/08/19 05:54:46 UTC
[PR] Report Dag cache metrics under each component's own namespace [airflow]
jason810496 opened a new pull request, #71815:
URL: https://github.com/apache/airflow/pull/71815
- depends on: #71704
- split from: #71813
Every `DBDagBag` emitted its cache counters under `api_server.dag_bag.*`, so once the scheduler gained a cache in #71704 its traffic was counted against the API server's series. An operator could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete.
Each caller now passes the namespace it reports under, and a cache built without one fails at construction rather than emitting a partial metric name mid-request. The scheduler reports under `scheduler.dag_bag.*`.
The metrics registry check matched dynamic names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller-supplied prefix; it now matches on all static parts wherever the variable sits. Supersedes #71276.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71815:
URL: https://github.com/apache/airflow/pull/71815#discussion_r3827744181
##########
airflow-core/src/airflow/api_fastapi/common/dagbag.py:
##########
@@ -40,7 +40,11 @@ def create_dag_bag() -> DBDagBag:
if cache_ttl < 0:
raise ValueError("[api] dag_cache_ttl must be greater than or equal to 0")
Review Comment:
Removed from the constructor of CachedDBDagBag in https://github.com/apache/airflow/pull/71815/commits/b19dbfad5fa255808019c49772f68f18261a74c4, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71815:
URL: https://github.com/apache/airflow/pull/71815#discussion_r3826302425
##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -122,8 +101,7 @@ def _read_dag(self, serdag: SerializedDagModel) -> SerializedDAG | None:
with self._lock:
self._dags[serdag.dag_version_id] = _CacheEntry(dag, serdag.dag_hash, time.monotonic())
cache_size = len(self._dags)
- if self._use_cache:
- stats.gauge("api_server.dag_bag.cache_size", cache_size, rate=0.1)
+ self._on_cache_size(cache_size, rate=0.1)
Review Comment:
`_on_cache_size` probably does not need to take `cache_size` at all since it’s always `len(self._dags)`? This would help eliminate the `len()` calls above when it’s not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71815:
URL: https://github.com/apache/airflow/pull/71815#discussion_r3827740245
##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -122,8 +101,7 @@ def _read_dag(self, serdag: SerializedDagModel) -> SerializedDAG | None:
with self._lock:
self._dags[serdag.dag_version_id] = _CacheEntry(dag, serdag.dag_hash, time.monotonic())
cache_size = len(self._dags)
- if self._use_cache:
- stats.gauge("api_server.dag_bag.cache_size", cache_size, rate=0.1)
+ self._on_cache_size(cache_size, rate=0.1)
Review Comment:
Good idea, addressed in https://github.com/apache/airflow/pull/71815/commits/b19dbfad5fa255808019c49772f68f18261a74c4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #71815:
URL: https://github.com/apache/airflow/pull/71815#issuecomment-5366388419
### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/32457215773'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/commit/bcf1dbf41a0593e44398c04b3d2aee7eeabe6ed5"><img src='https://img.shields.io/badge/Commit-bcf1dbf-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker bcf1dbf v3-3-test
```
This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #71815:
URL: https://github.com/apache/airflow/pull/71815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71815:
URL: https://github.com/apache/airflow/pull/71815#discussion_r3826308014
##########
airflow-core/src/airflow/api_fastapi/common/dagbag.py:
##########
@@ -40,7 +40,11 @@ def create_dag_bag() -> DBDagBag:
if cache_ttl < 0:
raise ValueError("[api] dag_cache_ttl must be greater than or equal to 0")
Review Comment:
CachedDBDagBag already contains the same checks. Since both `create_dag_bag` and `CachedDBDagBag` are Airflow-private, one of them should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Report Dag cache metrics under each component's own namespace [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #71815:
URL: https://github.com/apache/airflow/pull/71815#issuecomment-5352445267
Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the `_use_cache=True` variant a separate subclass i.e. `class CachedDBDagBag`. This way, the arguments will be cleaner (no more complicated optional arguments + checks in `__init__`, and the stats functions only need to be on the class they’re actually needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org