You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "KulykDmytro (via GitHub)" <gi...@apache.org> on 2023/02/24 09:22:41 UTC
[GitHub] [airflow] KulykDmytro opened a new issue, #29737: Metric dag_processing.total_parse_time is exposed incorrectly
KulykDmytro opened a new issue, #29737:
URL: https://github.com/apache/airflow/issues/29737
### Apache Airflow version
2.5.1
### What happened
Following metric `dag_processing.total_parse_time` exposing smth not relevant to overall processing time
```
DAG File Processing Stats
File Path PID Runtime # DAGs # Errors Last Runtime Last Run
------------------------------------------------- ----- --------- -------- ---------- -------------- -------------------
/opt/airflow/dags/player_time_features.py 1 0 0.89s 2023-02-24T09:18:59
/opt/airflow/dags/maxmind_uploader.py 1 0 0.81s 2023-02-24T09:19:29
/opt/airflow/dags/player_team_similarities.py 1 0 1.34s 2023-02-24T09:18:31
/opt/airflow/dags/aws_sanitize.py 1 0 3.58s 2023-02-24T09:18:58
/opt/airflow/dags/ad_backoffice.py 1 0 1.05s 2023-02-24T09:19:28
/opt/airflow/dags/gdpr.py 1 0 0.46s 2023-02-24T09:18:39
/opt/airflow/dags/aws_regional_split_backfill.py 1 0 2.55s 2023-02-24T09:19:12
/opt/airflow/dags/reconciliation_daily.py 1 0 1.04s 2023-02-24T09:19:16
/opt/airflow/dags/maintenance.py 1 0 0.24s 2023-02-24T09:17:51
/opt/airflow/dags/athena_repair_partitions.py 1 0 0.81s 2023-02-24T09:19:28
/opt/airflow/dags/trending_events.py 1 0 1.74s 2023-02-24T09:19:07
/opt/airflow/dags/casino_games_similarities.py 1 0 1.17s 2023-02-24T09:18:18
/opt/airflow/dags/player_segmentation.py 1 0 1.44s 2023-02-24T09:17:41
/opt/airflow/dags/test_copy.py 1 0 0.71s 2023-02-24T09:19:01
/opt/airflow/dags/datahub_ingest.py 1 0 0.90s 2023-02-24T09:19:05
/opt/airflow/dags/gs_exporter.py 1 0 2.65s 2023-02-24T09:17:37
/opt/airflow/dags/canary.py 1 0 1.75s 2023-02-24T09:19:04
/opt/airflow/dags/sfsc_load.py 1 0 1.06s 2023-02-24T09:19:15
/opt/airflow/dags/led_recognition_processed.py 1 0 2.28s 2023-02-24T09:19:09
/opt/airflow/dags/casino_games_recommendations.py 1 0 1.77s 2023-02-24T09:18:59
/opt/airflow/dags/vip_export.py 1 0 1.29s 2023-02-24T09:18:29
/opt/airflow/dags/autotest.py 2 0 0.63s 2023-02-24T09:19:30
/opt/airflow/dags/anomaly_detection.py 2 0 3.67s 2023-02-24T09:17:37
/opt/airflow/dags/native_rls.py 3 0 1.14s 2023-02-24T09:19:26
/opt/airflow/dags/dbt.py 4 0 1.23s 2023-02-24T09:18:14
/opt/airflow/dags/front_data.py 4 0 8.55s 2023-02-24T09:18:10
/opt/airflow/dags/gs_loader.py 4 0 1.37s 2023-02-24T09:19:25
/opt/airflow/dags/backfill_lambda.py 6 0 1.91s 2023-02-24T09:19:14
/opt/airflow/dags/tableau_refresher.py 7 0 1.85s 2023-02-24T09:17:39
/opt/airflow/dags/extract_load.py 8 0 1.30s 2023-02-24T09:18:49
/opt/airflow/dags/validator.py 30 0 1.91s 2023-02-24T09:19:24
/opt/airflow/dags/aws_transforms.py 1768 34.15s 56 0 151.91s 2023-02-24T09:17:01
```
### What you think should happen instead
_No response_
### How to reproduce
process dags by dag-processor
observe exposed metric: dag_processing.total_parse_time
### Operating System
n/a
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
AWS EKS
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29737:
URL: https://github.com/apache/airflow/issues/29737#issuecomment-1513963179
I think this is something to be addressed with the Open-Telemetry implementation @feruzzi @howardyoo - possibly dropping or replacing this metric with some aggregated one.
I think we should not keep it as an issue, now when it is documented, I doubt we will do anything with it but it should be looked at with the OTEL work.
converting to a discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly
URL: https://github.com/apache/airflow/issues/29737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on issue #29737: Metric dag_processing.total_parse_time is exposed incorrectly
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on issue #29737:
URL: https://github.com/apache/airflow/issues/29737#issuecomment-1499650247
I made a change recently (#30290) to edit the doc about this metric. The number of dag (files) parsed can vary from one run to the next, explaining (some of) the volatility.
You could try dividing one metric by the other, but since they are not emitted at the same time, it can still yield to bad results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org