You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rcampos87 (via GitHub)" <gi...@apache.org> on 2025/05/08 06:50:41 UTC

[I] Otel Metrics missing dag_ids [airflow]

rcampos87 opened a new issue, #50334:
URL: https://github.com/apache/airflow/issues/50334

   ### Apache Airflow version
   
   2.10.5
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   I have setup opentelemetry collector with my airflow 2.10.5 in kubernetes and I have currently 60 DAGs running often, and while I was setting up dashboards to view airflow data I noticed some otel metrics contains only 1 dag_id, always differing on which, but its never representing all my dag_ids. 
   
   For example, 
   ```
   airflow_dagrun_dependency_check
   airflow_dagrun_duration_failed
   airflow_dagrun_duration_success
   airflow_dagrun_first_task_scheduling_delay
   airflow_dagrun_schedule_delay
   airflow_local_task_job_task_exit_total
   airflow_operator_failures_total
   airflow_scheduler_tasks_killed_externally_total
   airflow_task_instance_created_total
   airflow_task_queued_duration
   airflow_ti_failures_total
   airflow_ti_finish_total
   airflow_ti_start_total
   ```
   In some of them I see labels for only a couple dag_ids, while in others I see only 1. Its almost like metrics sent by airflow are overriding dag_id label.
   
   ### What you think should happen instead?
   
   I should be able to see all my dag_ids as labels for metrics when querying prometheus
   
   ### How to reproduce
   
   Setup airflow 2.10.5 with otel collector in alloy
   
   [config.alloy.txt](https://github.com/user-attachments/files/20098070/config.alloy.txt)
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Otel Metrics missing dag_ids [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50334:
URL: https://github.com/apache/airflow/issues/50334#issuecomment-5031444113

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Otel Metrics missing dag_ids [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #50334: Otel Metrics missing dag_ids
URL: https://github.com/apache/airflow/issues/50334


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Otel Metrics missing dag_ids [airflow]

Posted by "shan-zeeshan786 (via GitHub)" <gi...@apache.org>.
shan-zeeshan786 commented on issue #50334:
URL: https://github.com/apache/airflow/issues/50334#issuecomment-2984502910

   We are also observing an inconsistency in how OpenTelemetry metrics are being emitted in our Airflow setup. Specifically, metrics that include tagging (such as dag_id) are only being emitted for a single DAG, despite having multiple DAGs running.
   
   For example, consider the following two metrics:
   1. **dagrun.dependency-check.<dag_id>**
   2. **dagrun.dependency-check** (with dag_id as a label)
   
   We are receiving the metric:
   **otelairflow_dagrun_dependency_check**{dag_id="hello_world_dag", job="Airflow", otel_scope_name="airflow.metrics.otel_logger", run_type="scheduled"} 0.0127
   
   However, this tagged metric is only being emitted for hello_world_dag. We are not seeing the same metric emitted for our other DAGs.
   
   On the other hand, the per-DAG metric (dagrun.dependency-check.<dag_id>) is being correctly emitted for all DAGs.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Otel Metrics missing dag_ids [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #50334:
URL: https://github.com/apache/airflow/issues/50334#issuecomment-4749981574

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org