You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "AutomationDev85 (via GitHub)" <gi...@apache.org> on 2025/02/06 10:19:09 UTC

[PR] OTEL metric export: Switch ObservableGauge to sync Gauge metric export [airflow]

AutomationDev85 opened a new pull request, #46510:
URL: https://github.com/apache/airflow/pull/46510

   # Overview
   Hi Airflow community,
   
   I´m currently trying to get Airflow metrics exported via OTEL -> Prometheus into Grafana Dashboards. I wanted to use the label advantage of OTEL to create nice Dashboards. 
   
   During the implementation I found an issue and will use metric airflow_dag_processing_last_duration to explain the details:
   
   This metric is exported in 2 different ways in the Airflow code, to support statsd way and otel way with labels.
   Stats.timing(f"dag_processing.last_duration.{file_name}", stat.last_duration)
   Stats.timing("dag_processing.last_duration", stat.last_duration, tags={"file_name": file_name})
   
   The following prometheus example contains metric export for 2 dags (dag1, dag2). But the metric with the label (Otel) does only export the metric for 1 dag. Metric export with file_name in the metric name is exported for 2 dags.
   
   # HELP airflow_dag_processing_last_duration
   # TYPE airflow_dag_processing_last_duration gauge
   airflow_dag_processing_last_duration{file_name="dag1",job="Airflow"} 0.293856
   
   # HELP airflow_dag_processing_last_duration_dag1
   # TYPE airflow_dag_processing_last_duration_dag1  gauge
   airflow_dag_processing_last_duration_dag1 {job="Airflow"} 0.293856
   # HELP airflow_dag_processing_last_duration_blabla2
   # TYPE airflow_dag_processing_last_duration_blabla2 gauge
   airflow_dag_processing_last_duration_dag2{job="Airflow"} 0.343803
   
   I would expect that the metric is also exported like this:
   airflow_dag_processing_last_duration{file_name="dag2",job="Airflow"} 0.343803
   
   
   So I debugged some time and found out that this issue is only related to the gauge export. If the metric is a counter the label export works fine.
   
   The issue is that the gauge value is created as an ObserveableGauge and with that OTEL uses a callback to collect the metric. For this OTEL python lib creates intruments to handle the metric. The down side of this is that if a second Observable instrument with the same metric name is created, OTEL will only create one instrument because it checks for the metric name. 
   This results in the issue that only the callback for the first registered metric will be executed and all other metrics with different label but same name will be ignored.
   My idea is now to switch to an syncronos gauge export of the metric like it is used for the counter export. 
   
   I´m not sure why the ObserveableGauge was used but I did not found a solution to fix the issue without switching to sync gauge export.
   Also not sure about any down side of using sync gauge, like maybe runtime, but for the counter metric export sync was also used. Anyone has more know how in that area and give feedback for this?
   
   # Details of changes:
   * Use of normal sync Gauge instead of ObservedGauge.
   * Moved logic to handle gauge into InternalGauge class.
   
   Looking forward to fix this issue and the feedback from your side!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OTEL metric export: Switch ObservableGauge to sync Gauge metric export [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #46510:
URL: https://github.com/apache/airflow/pull/46510


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OTEL metric export: Switch ObservableGauge to sync Gauge metric export [airflow]

Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #46510:
URL: https://github.com/apache/airflow/pull/46510#issuecomment-2643971955

   > Do you have a link to the issue thread were the issues are described which are mentioned by you?
   
   You found the one that I was thinking about, bu if you [search open Issues for "otel"](https://github.com/apache/airflow/issues?q=is%3Aissue%20state%3Aopen%20otel) there are around 20 or so, if you are interested in helping.    `sla_miss` is removed in 3.0 so that one might not be worth fixing now, and the one about duplicate config options needs to be discussed, but there are some that might be interesting if you are inclined.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OTEL metric export: Switch ObservableGauge to sync Gauge metric export [airflow]

Posted by "AutomationDev85 (via GitHub)" <gi...@apache.org>.
AutomationDev85 commented on PR #46510:
URL: https://github.com/apache/airflow/pull/46510#issuecomment-2642389107

   @ferruzzi  Thanks for the feedback. I tested it:
   ![image](https://github.com/user-attachments/assets/c462d18a-331b-4a15-b8c6-7bc592104bc3)
   
   Here you see that after  switching to my changes the metric includes all DAGs and not only for one.
   I will also try to check the counter issue because I ran into some strange behavior. 
   
   Do you have a link to the issue thread were the issues are described which are mentioned by you?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org