You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dirrao (via GitHub)" <gi...@apache.org> on 2023/06/16 10:45:21 UTC
[GitHub] [airflow] dirrao opened a new issue, #31958: Improve clear_not_launched_queued_tasks function call latency
dirrao opened a new issue, #31958:
URL: https://github.com/apache/airflow/issues/31958
### Description
The function clear_not_launched_queued_tasks takes time when there are more queued tasks (a few hundred). The reason for latency is due to list_namespaced_pod kube API triggered for each queued task. It leads to scheduler heartbeat delay. Improve this function by calling list_namespaced_pod function using pagination and getting all the required pods with fewer calls.
### Use case/motivation
As we run the airflow at a large scale, we have found that the clear_not_launched_queued_tasks function might take a few minutes (> 5 minutes). These will delay the heartbeat of the scheduler and leads to the scheduler instance restarting/killed. To avoid this issue, use a list namespaced pod with pagination and get all the worker pods with fewer calls.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #31958: Improve clear_not_launched_queued_tasks function call latency
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #31958:
URL: https://github.com/apache/airflow/issues/31958#issuecomment-1691503233
@dirrao are you working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve clear_not_launched_queued_tasks function call latency [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on issue #31958:
URL: https://github.com/apache/airflow/issues/31958#issuecomment-2901971794
Hi all, the performance issue addressed here appears to have been fixed by PR #34985, which also closed the related ticket #34877.
Since both issues describe the same problem with `clear_not_launched_queued_tasks()`, so I think we could mark this one as a duplicate of #34877 and close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31958: Improve clear_not_launched_queued_tasks function call latency
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31958:
URL: https://github.com/apache/airflow/issues/31958#issuecomment-1594565444
Same as the other issue - feel free to add those metrics. Assigned you. cc: @ferruzzi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org