You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/07 20:21:06 UTC
[GitHub] [airflow] rachelwigell opened a new issue #18069: Move auto-refresh interval to config variable
rachelwigell opened a new issue #18069:
URL: https://github.com/apache/airflow/issues/18069
### Description
Auto-refresh is an awesome new feature! Right now it occurs every [3 seconds](https://github.com/astronomer/airflow/blob/main/airflow/www/static/js/tree.js#L463), and this interval is not configurable.
### Use case/motivation
On a DAG with many tasks and many active DAG runs, auto-refresh can be a significant strain on the webserver. We observed our production webserver reach 100% CPU today when we had about 30 running DAG runs of a DAG containing about 50 tasks. Being able to increase the interval to guard against this would be helpful.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachelwigell commented on issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
rachelwigell commented on issue #18069:
URL: https://github.com/apache/airflow/issues/18069#issuecomment-916330614
Hmm okay. We were able to observe pretty clearly that when we had tree view open, with auto-refresh on, looking back over 365 DAG runs, of which about 30 were in the running state, our webserver ground to a halt and CPU was at 100%. When we closed that tab it recovered. I can do some experimentation in my dev environment tonight to see if I can nail down what exactly strains the CPU so much. It could be that even with just one request there was too much data to parse out of the request results for the webserver CPU to handle doing every 3 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18069:
URL: https://github.com/apache/airflow/issues/18069#issuecomment-914602138
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18069:
URL: https://github.com/apache/airflow/issues/18069#issuecomment-916187641
The config option is fine, but the number of Tasks in a DAG _shouldn't_ have any bearing on the number of requests made (it should be a single request per use on a Graph or Tree View page).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18069:
URL: https://github.com/apache/airflow/issues/18069#issuecomment-916864057
Yeah, absolutely no problem with your PR, I was just a bit surprised by the effects!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachelwigell commented on issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
rachelwigell commented on issue #18069:
URL: https://github.com/apache/airflow/issues/18069#issuecomment-916577592
Yeah I haven't quite been able to locally replicate the same severe effects on CPU we saw a few days ago. I do see that we are indeed making just one request to populate the tree view regardless of whether I have many active tasks or DAG runs. Of course, that request has to do more work if there are lots of tasks to iterate over, but the n's here don't seem like enough to tank a server. Regardless, I'll move forward with my PR as I think folks still might like being able to customize this setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #18069:
URL: https://github.com/apache/airflow/issues/18069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rachelwigell edited a comment on issue #18069: Move auto-refresh interval to config variable
Posted by GitBox <gi...@apache.org>.
rachelwigell edited a comment on issue #18069:
URL: https://github.com/apache/airflow/issues/18069#issuecomment-916330614
Hmm okay. We were able to observe pretty clearly that when we had tree view open, with auto-refresh on, looking back over 365 DAG runs, of which about 30 were in the running state, our webserver ground to a halt and CPU was at 100%. When we closed that tab it recovered. I can do some experimentation in my dev environment tonight to see if I can nail down what exactly strains the CPU so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org