You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "crabio (via GitHub)" <gi...@apache.org> on 2024/05/31 10:35:53 UTC
[I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
crabio opened a new issue, #36478:
URL: https://github.com/apache/airflow/issues/36478
### Apache Airflow version
2.8.0
### If "Other Airflow 2 version" selected, which one?
2.7.3
### What happened?
I found race condition between 2 schedulers.
Scheduler 1 starts task pod
task finished
Scheduler 2 processes task finish and kill pod
Scheduler 1 thinks that task is still running, but Scheduler 2 updates tasks status in the database.
### What you think should happen instead?
Multiple schedulers should process properly concurrent work with Kubernetes Executor
### How to reproduce
1. Deploy Airflow via Community Helm chart with Kubernetes Executor
2. Add more than 50 tasks in 2-3 DAGs
### Operating System
Docker based on apache/airflow:2.7.3
### Versions of Apache Airflow Providers
apache-airflow == 2.7.3
dbt-core == 1.6.6
dbt-snowflake == 1.6.4
apache-airflow-providers-snowflake
apache-airflow[statsd]
facebook-business == 16.0.2
google-ads == 21.1.0
twitter-ads == 11.0.0
acryl-datahub-airflow-plugin
acryl-datahub[dbt]
checksumdir
filelock
openpyxl
cronsim
apache-airflow-providers-cncf-kubernetes==7.8.0
apache-airflow-providers-apache-kafka == 1.2.0
kubernetes
snowplow_analytics_sdk
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### Anything else?
logs:
[Untitled discover search.csv](https://github.com/apache/airflow/files/13318290/Untitled.discover.search.csv)
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-2044283620
Hi!
I have tested this bug on the Airflow 2.8.4 and it still present...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio closed issue #36478: Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks
URL: https://github.com/apache/airflow/issues/36478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1884755897
> @dirrao Yes, thank you!
>
> I rollback our cluster to 1 Scheduler and 64 parallelism to have the same capacity :) About restart - I found that it could be made in the Airflow < 2.x.x, because scheduler run duration removed from the configuration. I can do it manually or via Kubernetes job :) But it seems not so good idea
You can deduce the number of iterations based on scheduler loop time metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1871513362
Discussed in https://github.com/apache/airflow/discussions/35426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1909132579
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy closed issue #36478: Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks
URL: https://github.com/apache/airflow/issues/36478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #36478: Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks
URL: https://github.com/apache/airflow/issues/36478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1922529692
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1878294865
Hello @dirrao
I upgraded my Airflow package `apache-airflow-providers-cncf-kubernetes==7.13.0` and increased Schedulers replicas to 2 and in our Dev environment all was fine. But on prod, where we have more load - scheduler starts leak again.
<img width="1059" alt="image" src="https://github.com/apache/airflow/assets/40871973/7d551e01-7df0-4fab-b11e-f7eb1a7a0020">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-2141737200
duplicated in https://github.com/apache/airflow/issues/36998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1878329845
@crabio,
Thanks for updating us on this. This requires triaging. Meanwhile, you can bump up the parallelism configuration to a higher number to beat the leak. Or Restart the scheduler after a certain number of iterations to rest these values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-2044286035
@ephraimbuddy @dirrao may you reopen the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-2044294535
duplicated in https://github.com/apache/airflow/issues/36998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1873982378
@dirrao Seems it works well! thank you
I will watch for some days for fix and reopen issue if anything will be wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1871717832
Hi @crabio,
The K8s executor open_slots are full over the time. This is due to the adoption of completed pods by schedulers. We have found a leak in the open slots. It will be accumulated over time and no slots are available to schedule the tasks. This is fixed in the https://github.com/apache/airflow/pull/36240. Multiple fixes are available in the CNCF Kubernetes provider 7.12.0. You can try and let us know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1872228151
Thank you (@dirrao) very much!
I will try and be back soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks [airflow]
Posted by "crabio (via GitHub)" <gi...@apache.org>.
crabio commented on issue #36478:
URL: https://github.com/apache/airflow/issues/36478#issuecomment-1878480186
@dirrao Yes, thank you!
I rollback our cluster to 1 Scheduler and 64 parallelism to have the same capacity :)
About restart - I found that it could be made in the Airflow < 2.x.x, because scheduler run duration removed from the configuration.
I can do it manually or via Kubernetes job :)
But it seems not so good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org