You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/03 16:09:42 UTC
[GitHub] [airflow] john-jac opened a new issue #14583: Document Airflow Metrics Optimal Values
john-jac opened a new issue #14583:
URL: https://github.com/apache/airflow/issues/14583
**Description**
Users of Airflow appreciate the [metrics](https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/metrics.html) that Airflow emits, however are often confused as to what values indicate a healthy system versus a system that might be exhibiting problems.
This feature request is to add value ranges to the metrics documentation as to what values are appropriate, as well as troubleshooting information as to how to address when those metrics fall outside recommended values.
**Use case / motivation**
For example, looking at `dag_processing.total_parse_time` under [Gauges](https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/metrics.html?highlight=metrics#gauges), what are appropriate values here? As a user, how would I know that my scheduler is under-powered, or that I have too many/too complex DAGs, or that some other issue exists?
**Are you willing to submit a PR?**
Yes
**Related Issues**
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #14583: Document Airflow Metrics Optimal Values
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #14583:
URL: https://github.com/apache/airflow/issues/14583#issuecomment-813015721
I'm not sure if there is one (or even range) universal values to be considered healthy that match all.
For example `dag_processing.total_parse_time | Seconds taken to scan and import all DAG files once`
What is the appropriate value that can be recommended? Consider an Airflow instance with 3 dags and instance with 3000 Dags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] john-jac commented on issue #14583: Document Airflow Metrics Optimal Values
Posted by GitBox <gi...@apache.org>.
john-jac commented on issue #14583:
URL: https://github.com/apache/airflow/issues/14583#issuecomment-823574467
> I'm not sure if there is one (or even range) universal values to be considered healthy that match all.
> For example `dag_processing.total_parse_time | Seconds taken to scan and import all DAG files once`
>
> What is the appropriate value that can be recommended? Consider an Airflow instance with 3 dags and instance with 3000 Dags.
What would be helpful is recommended guidelines, regardless of the number of DAGs (as the number of DAGs don't linearly equate to parse time given other variables like DAG complexity). For example, if Total Parse Time > x minutes, how does that effect task execution? Does that mean that a user should add more resources to the scheduler? If so, should they add memory/cpu/both?
Another option would be mapping values like "if you have metric x, then you should have a configuration value of x". Going back to the parse time example, if I have a parse time of 2 minutes then I should set processor_poll_interval to X and min_file_process_interval to Y
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14583: Document Airflow Metrics Optimal Values
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14583:
URL: https://github.com/apache/airflow/issues/14583#issuecomment-789828611
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org