You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ferruzzi (via GitHub)" <gi...@apache.org> on 2026/06/05 22:30:57 UTC
[PR] Add team_name to Multi-Team metrics [airflow]
ferruzzi opened a new pull request, #68108:
URL: https://github.com/apache/airflow/pull/68108
Add `team_name` to ~35 metrics (DagRun, TaskInstance, pool slots) for multi-team Airflow deployments. When multi_team is disabled or a DAG has no team, the tag is omitted entirely (no change to existing systems).
Changes:
- Add `team_name` to `DagRun.stats_tags` and `TaskInstance.stats_tags`
- Set `_team_name` on `DagRun`/`TI` objects in 4 scheduler injection points
- Add persistent `_dag_id_to_team_name` cache on `SchedulerJobRunner`
- Add `stats_tags` property to `RuntimeTaskInstance` (Task SDK)
- Add `team_name` to pool slot gauge metrics via `pool.get_name_to_team_name_mapping`
Co-authored-by: Niko Oliveira <on...@amazon.com>
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3376055689
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -394,36 +397,35 @@ def _get_team_names_for_dag_ids(
if not dag_ids:
return {}
- try:
- # Query all team names for the given DAG IDs in a single query
- query_results = session.execute(
- select(DagModel.dag_id, Team.name)
- .join(DagBundleModel.teams) # Join Team to DagBundleModel via association table
- .join(
- DagModel, DagModel.bundle_name == DagBundleModel.name
- ) # Join DagBundleModel to DagModel
- .where(DagModel.dag_id.in_(dag_ids))
- ).all()
-
- # Create mapping from results
- dag_id_to_team_name = {dag_id: team_name for dag_id, team_name in query_results}
-
- # Ensure all requested dag_ids are in the result (with None for those not found)
- result = {dag_id: dag_id_to_team_name.get(dag_id) for dag_id in dag_ids}
+ missing = [dag_id for dag_id in dag_ids if dag_id not in self._dag_id_to_team_name]
+ if missing:
+ try:
+ # Query all team names for the given DAG IDs in a single query
+ query_results = session.execute(
Review Comment:
Sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3376007188
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -394,36 +397,35 @@ def _get_team_names_for_dag_ids(
if not dag_ids:
return {}
- try:
- # Query all team names for the given DAG IDs in a single query
- query_results = session.execute(
- select(DagModel.dag_id, Team.name)
- .join(DagBundleModel.teams) # Join Team to DagBundleModel via association table
- .join(
- DagModel, DagModel.bundle_name == DagBundleModel.name
- ) # Join DagBundleModel to DagModel
- .where(DagModel.dag_id.in_(dag_ids))
- ).all()
-
- # Create mapping from results
- dag_id_to_team_name = {dag_id: team_name for dag_id, team_name in query_results}
-
- # Ensure all requested dag_ids are in the result (with None for those not found)
- result = {dag_id: dag_id_to_team_name.get(dag_id) for dag_id in dag_ids}
+ missing = [dag_id for dag_id in dag_ids if dag_id not in self._dag_id_to_team_name]
+ if missing:
+ try:
+ # Query all team names for the given DAG IDs in a single query
+ query_results = session.execute(
Review Comment:
FWIW: I think it's useful to cache for each scheduler loop, since there are a few places during the loop where we need this association. But it should be updated each loop that any particular Dag is getting scheduled. That way we don't need to worry about TTL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi merged PR #68108:
URL: https://github.com/apache/airflow/pull/68108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3375958346
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -394,36 +397,35 @@ def _get_team_names_for_dag_ids(
if not dag_ids:
return {}
- try:
- # Query all team names for the given DAG IDs in a single query
- query_results = session.execute(
- select(DagModel.dag_id, Team.name)
- .join(DagBundleModel.teams) # Join Team to DagBundleModel via association table
- .join(
- DagModel, DagModel.bundle_name == DagBundleModel.name
- ) # Join DagBundleModel to DagModel
- .where(DagModel.dag_id.in_(dag_ids))
- ).all()
-
- # Create mapping from results
- dag_id_to_team_name = {dag_id: team_name for dag_id, team_name in query_results}
-
- # Ensure all requested dag_ids are in the result (with None for those not found)
- result = {dag_id: dag_id_to_team_name.get(dag_id) for dag_id in dag_ids}
+ missing = [dag_id for dag_id in dag_ids if dag_id not in self._dag_id_to_team_name]
+ if missing:
+ try:
+ # Query all team names for the given DAG IDs in a single query
+ query_results = session.execute(
Review Comment:
I had a chat with Niko and dropped this cache entirely. I thought I was being clever by tying it into the scheduler lifecycle (TTL was "when the scheduler restarts") but it was a bad idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3376454739
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -394,36 +397,35 @@ def _get_team_names_for_dag_ids(
if not dag_ids:
return {}
- try:
- # Query all team names for the given DAG IDs in a single query
- query_results = session.execute(
- select(DagModel.dag_id, Team.name)
- .join(DagBundleModel.teams) # Join Team to DagBundleModel via association table
- .join(
- DagModel, DagModel.bundle_name == DagBundleModel.name
- ) # Join DagBundleModel to DagModel
- .where(DagModel.dag_id.in_(dag_ids))
- ).all()
-
- # Create mapping from results
- dag_id_to_team_name = {dag_id: team_name for dag_id, team_name in query_results}
-
- # Ensure all requested dag_ids are in the result (with None for those not found)
- result = {dag_id: dag_id_to_team_name.get(dag_id) for dag_id in dag_ids}
+ missing = [dag_id for dag_id in dag_ids if dag_id not in self._dag_id_to_team_name]
+ if missing:
+ try:
+ # Query all team names for the given DAG IDs in a single query
+ query_results = session.execute(
Review Comment:
It should now be "cached for one pass" as Niko said.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3373726716
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -394,36 +397,35 @@ def _get_team_names_for_dag_ids(
if not dag_ids:
return {}
- try:
- # Query all team names for the given DAG IDs in a single query
- query_results = session.execute(
- select(DagModel.dag_id, Team.name)
- .join(DagBundleModel.teams) # Join Team to DagBundleModel via association table
- .join(
- DagModel, DagModel.bundle_name == DagBundleModel.name
- ) # Join DagBundleModel to DagModel
- .where(DagModel.dag_id.in_(dag_ids))
- ).all()
-
- # Create mapping from results
- dag_id_to_team_name = {dag_id: team_name for dag_id, team_name in query_results}
-
- # Ensure all requested dag_ids are in the result (with None for those not found)
- result = {dag_id: dag_id_to_team_name.get(dag_id) for dag_id in dag_ids}
+ missing = [dag_id for dag_id in dag_ids if dag_id not in self._dag_id_to_team_name]
+ if missing:
+ try:
+ # Query all team names for the given DAG IDs in a single query
+ query_results = session.execute(
Review Comment:
1. What is the TTL of the cache?
2. What happens when a Dag has no team? Do we still cache this result or we keep querying? I tend to think that `query_results` is empty in that case, which means keep querying the team name for this Dag at every single heartbeat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3376048907
##########
airflow-core/src/airflow/models/dagrun.py:
##########
@@ -501,7 +500,11 @@ def check_version_id_exists_in_dr(self, dag_version_id: UUID, *, session: Sessio
@property
def stats_tags(self) -> dict[str, str]:
- return prune_dict({"dag_id": self.dag_id, "run_type": self.run_type})
+ tags = prune_dict({"dag_id": self.dag_id, "run_type": self.run_type})
+ team_name = getattr(self, "_team_name", None)
+ if team_name:
+ tags["team_name"] = team_name
+ return tags
Review Comment:
nit (but also personal preference so feel free to leave it as is if you prefer your version)
```suggestion
tags = prune_dict({"dag_id": self.dag_id, "run_type": self.run_type, "team_name": getattr(self, "_team_name", None)})
```
If you apply it, apply it elsewhere as well (e.g. `airflow-core/src/airflow/models/taskinstance.py`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add team_name to Multi-Team metrics [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #68108:
URL: https://github.com/apache/airflow/pull/68108#discussion_r3383789282
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -385,9 +386,11 @@ def _get_team_names_for_dag_ids(
self, dag_ids: Collection[str], session: Session
) -> dict[str, str | None]:
"""
- Batch query to resolve team names for multiple DAG IDs using the DAG > Bundle > Team relationship chain.
+ Resolve team names for DAG IDs via the DAG → Bundle → Team relationship.
Review Comment:
Nit: I actually prefer `>` over `→`, it's more universally renderable and supported.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1714,6 +1729,7 @@ def _run_scheduler_loop(self) -> None:
idle_count = 0
for loop_count in itertools.count(start=1):
+ self._dag_id_to_team_name = {}
Review Comment:
Worth adding a comment explaining why this is reset each loop, or at least that it is very intentional and important to keep this way.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -716,6 +718,9 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session) -
len(unique_dag_ids),
list(unique_dag_ids),
)
+ for ti in task_instances_to_examine:
+ if team := dag_id_to_team_name.get(ti.dag_id):
+ ti._team_name = team
Review Comment:
Maybe worth noting in a comment for posterity that we have to do this because the community refused to allow us to update the DB schema for TIs to include team_name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org