You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rcampos87 (via GitHub)" <gi...@apache.org> on 2025/05/08 06:50:41 UTC
[I] Otel Metrics missing dag_ids [airflow]
rcampos87 opened a new issue, #50334:
URL: https://github.com/apache/airflow/issues/50334
### Apache Airflow version
2.10.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have setup opentelemetry collector with my airflow 2.10.5 in kubernetes and I have currently 60 DAGs running often, and while I was setting up dashboards to view airflow data I noticed some otel metrics contains only 1 dag_id, always differing on which, but its never representing all my dag_ids.
For example,
```
airflow_dagrun_dependency_check
airflow_dagrun_duration_failed
airflow_dagrun_duration_success
airflow_dagrun_first_task_scheduling_delay
airflow_dagrun_schedule_delay
airflow_local_task_job_task_exit_total
airflow_operator_failures_total
airflow_scheduler_tasks_killed_externally_total
airflow_task_instance_created_total
airflow_task_queued_duration
airflow_ti_failures_total
airflow_ti_finish_total
airflow_ti_start_total
```
In some of them I see labels for only a couple dag_ids, while in others I see only 1. Its almost like metrics sent by airflow are overriding dag_id label.
### What you think should happen instead?
I should be able to see all my dag_ids as labels for metrics when querying prometheus
### How to reproduce
Setup airflow 2.10.5 with otel collector in alloy
[config.alloy.txt](https://github.com/user-attachments/files/20098070/config.alloy.txt)
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Otel Metrics missing dag_ids [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50334:
URL: https://github.com/apache/airflow/issues/50334#issuecomment-5031444113
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Otel Metrics missing dag_ids [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #50334: Otel Metrics missing dag_ids
URL: https://github.com/apache/airflow/issues/50334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Otel Metrics missing dag_ids [airflow]
Posted by "shan-zeeshan786 (via GitHub)" <gi...@apache.org>.
shan-zeeshan786 commented on issue #50334:
URL: https://github.com/apache/airflow/issues/50334#issuecomment-2984502910
We are also observing an inconsistency in how OpenTelemetry metrics are being emitted in our Airflow setup. Specifically, metrics that include tagging (such as dag_id) are only being emitted for a single DAG, despite having multiple DAGs running.
For example, consider the following two metrics:
1. **dagrun.dependency-check.<dag_id>**
2. **dagrun.dependency-check** (with dag_id as a label)
We are receiving the metric:
**otelairflow_dagrun_dependency_check**{dag_id="hello_world_dag", job="Airflow", otel_scope_name="airflow.metrics.otel_logger", run_type="scheduled"} 0.0127
However, this tagged metric is only being emitted for hello_world_dag. We are not seeing the same metric emitted for our other DAGs.
On the other hand, the per-DAG metric (dagrun.dependency-check.<dag_id>) is being correctly emitted for all DAGs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Otel Metrics missing dag_ids [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50334:
URL: https://github.com/apache/airflow/issues/50334#issuecomment-4749981574
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org