You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dirrao (via GitHub)" <gi...@apache.org> on 2023/11/16 08:09:02 UTC
[I] Leak in Kubernetes Executor Running Tasks SLOT Count [airflow]
dirrao opened a new issue, #35675:
URL: https://github.com/apache/airflow/issues/35675
### Apache Airflow version
main (development)
### What happened
Schedulers are racing for pod adoption when there is a delay in schedulers' heartbeats. However, the schedulers are alive but not dead their heartbeat is delayed due to network timeout or heavy processing, etc. This leads to a leak in the executor.running_tasks slots. Eventually, the schedulers are not able to launch the pods due to executor.running_tasks=parallelism.
### What you think should happen instead
We should remove the entry from the Kubernetes executor running queue when we worker pod deleted / moved to another scheduler.
### How to reproduce
Reduce the scheduler_health_check_threshold=5 and orphaned_tasks_check_interval=10 values in the airflow config file
Launch the airflow with two schedulers and try to schedule multiple DAGs with backfill for every 1/5 mins.
### Operating System
CentOS 6
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes=7.9.0
### Deployment
Other Docker-based deployment
### Deployment details
Terraform
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Leak in Kubernetes Executor Running Tasks Slot Count [airflow]
Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on issue #35675:
URL: https://github.com/apache/airflow/issues/35675#issuecomment-1842586190
We had the same problem and only noticed it from Airflow 2.7.3, but it was there before.
Maybe this helps for debugging:
On the left of the red line we had Airflow 2.6.X
On the right side is the update to 2.7.3.
Each line is one scheduler.
<img width="1522" alt="image" src="https://github.com/apache/airflow/assets/12560027/c6cd22e4-cbfd-445f-b0dd-5f19bbae0982">
Before the update, scheduler open slots continuously went down to even below zero sometimes, but only for some executors.
After the update this is much more distributed between all the executors.
**Our Workaround**
What helped us is limit the parallelism via pools, and just set `AIRFLOW__CORE__PARALLELISM` to an arbitrary high number. Also regular deployment restarts helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Leak in Kubernetes Executor Running Tasks Slot Count [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #35675:
URL: https://github.com/apache/airflow/issues/35675#issuecomment-1842591230
@c-thiel
Yes. This workaround works and we are following the same. I have identified the leaks and am going to open the MR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Leak in Kubernetes Executor Running Tasks Slot Count [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #35675: Leak in Kubernetes Executor Running Tasks Slot Count
URL: https://github.com/apache/airflow/issues/35675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org