You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "benrifkind (via GitHub)" <gi...@apache.org> on 2024/10/28 18:25:24 UTC
[I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
benrifkind opened a new issue, #43448:
URL: https://github.com/apache/airflow/issues/43448
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I am running Airflow 2.10.2 on an AWS EKS cluster with the CeleryExecutor. I ran into an issue where the Airflow workers were dying because they were OOMing. The workers had triggered many KubernetesPodOperator tasks which were in the process of running when the workers died. The workers would restart but the Airflow scheduler would get stuck in a bad state thinking it had no slots available even when the tasks that were triggered by the now dead worker had completed. New tasks would fail to get scheduled and I would see repeated logs like
```
[2024-10-23T18:37:10.786+0000] {base_executor.py:292} INFO - Executor parallelism limit reached. 0 open slots.
```
Looking at the logs, it seems like the scheduler would realize that these are zombie tasks and would set them to failed/up for retry but it still would not free up slots for it to schedule new tasks
```
ERROR - Detected zombie job:
```
### What you think should happen instead?
If the scheduler realizes that the tasks are zombie tasks and sets them to failed/up_for_retry, it should also free up the slots that those tasks are taking up on the scheduler
### How to reproduce
Deploy Airflow on kubernetes with the AIRFLOW__CORE__PARALLELISM set to a lower number and one worker whose concurrency is greater than or equal to AIRFLOW__CORE__PARALLELISM. Then create a DAG that runs the max number of KubernetesPodOperator tasks concurrently. Wait until the tasks have started and then delete the worker with
```
kubectl delete pod airflow-worker-0 --grace-period=0 --force
```
The task pods will continue running and succeed. And the scheduler will register that these tasks are zombies and mark them as failed. But it will not open up slots for these tasks and so it will get stuck. Restarting the scheduler will unstick it.
Note also that we don't have to max out the number of tasks that the worker is running relative to AIRFLOW__CORE__PARALLELISM to create this issue. Whatever number of tasks that the worker is running when it fails will indefinitely take up scheduler slots. I checked this by killing the worker while it was running less than `AIRFLOW__CORE__PARALLELISM` multiple times. This eventually recreates the issue once all the scheduler slots are taken up.
Here's a configuration and a DAG that I used to create the issue
```
AIRFLOW__CORE__PARALLELISM: 4
AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD: 120
AIRFLOW__CELERY__WORKER_CONCURRENCY: 4
```
```python
from datetime import datetime
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from kubernetes.client.models import V1ResourceRequirements
dag = DAG(
dag_id="high_task_dag_1",
schedule_interval="* * * * *",
start_date=datetime(2024, 10, 1),
max_active_runs=1,
concurrency=4,
catchup=False,
)
empty_operator = EmptyOperator(task_id="start", dag=dag)
for i in range(10):
(
KubernetesPodOperator(
dag=dag,
task_id=f"task_{i}",
cmds=["sleep"],
arguments=["30"],
image="busybox:latest",
container_resources=V1ResourceRequirements(
requests={"cpu": "0.1", "memory": "10Mi"},
limits={"cpu": "0.1", "memory": "10Mi"},
),
service_account_name="salesman",
)
>> empty_operator
)
```
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.28.0
apache-airflow-providers-celery==3.8.1
apache-airflow-providers-cncf-kubernetes==8.4.1
apache-airflow-providers-common-compat==1.2.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.16.0
apache-airflow-providers-docker==3.13.0
apache-airflow-providers-elasticsearch==5.5.0
apache-airflow-providers-fab==1.3.0
apache-airflow-providers-ftp==3.11.0
apache-airflow-providers-google==10.22.0
apache-airflow-providers-grpc==3.6.0
apache-airflow-providers-hashicorp==3.8.0
apache-airflow-providers-http==4.13.0
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-microsoft-azure==10.4.0
apache-airflow-providers-mysql==5.7.0
apache-airflow-providers-odbc==4.7.0
apache-airflow-providers-openlineage==1.11.0
apache-airflow-providers-postgres==5.12.0
apache-airflow-providers-redis==3.8.0
apache-airflow-providers-sendgrid==3.6.0
apache-airflow-providers-sftp==4.11.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-snowflake==5.7.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1
### Deployment
Other 3rd-party Helm chart
### Deployment details
I am using this helm chart https://github.com/airflow-helm/charts/tree/main/charts/airflow
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43448:
URL: https://github.com/apache/airflow/issues/43448#issuecomment-3046156305
> I just had the same issue. Thanks god it was today on Monday. [@potiuk](https://github.com/potiuk) are you sure updating the minor version is the fix for it? I just restarted my schedulers and it did run ok! but of course it can happen again at any time.
No. I have no certainty. But upgrading is good idea anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham closed issue #43448: Airflow scheduler thinks slots are not available when worker dies running tasks
URL: https://github.com/apache/airflow/issues/43448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43448:
URL: https://github.com/apache/airflow/issues/43448#issuecomment-2458306629
> @potiuk Upgrading seems to have worked. I can't recreate this issue on 2.10.3. Thanks!
Thanks! That's encouraging !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43448:
URL: https://github.com/apache/airflow/issues/43448#issuecomment-2457637003
Can you please upgrade and report back @benrifkind ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "benrifkind (via GitHub)" <gi...@apache.org>.
benrifkind commented on issue #43448:
URL: https://github.com/apache/airflow/issues/43448#issuecomment-2457829013
@potiuk Upgrading seems to have worked. I can't recreate this issue on 2.10.3. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #43448:
URL: https://github.com/apache/airflow/issues/43448#issuecomment-2457380525
#42932, coming in 2.10.3, should fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow scheduler thinks slots are not available when worker dies running tasks [airflow]
Posted by "netogerbi (via GitHub)" <gi...@apache.org>.
netogerbi commented on issue #43448:
URL: https://github.com/apache/airflow/issues/43448#issuecomment-3046152675
I just had the same issue. Thanks god it was today on Monday. @potiuk are you sure updating the minor version is the fix for it? I just restarted my schedulers and it did run ok! but of course it can happen again at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org