You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Bisk1 (via GitHub)" <gi...@apache.org> on 2023/08/23 15:42:17 UTC
[GitHub] [airflow] Bisk1 opened a new issue, #33661: Zombie tasks in RESTARTING state are not cleaned
Bisk1 opened a new issue, #33661:
URL: https://github.com/apache/airflow/issues/33661
### Apache Airflow version
2.7.0
### What happened
Recently we added some automation to clearing Airflow tasks so we use this feature a lot. It also often happens when tasks are in RUNNING state, which means that they go into RESTARTING state. We noticed that a lot of those tasks get stuck in RESTARTING state. Our Airflow infrastructure runs in an environment where any process can get suddenly killed without graceful shutdown.
We run Airflow on GKE but I managed to reproduce this behaviour on local environment with SequentialExecutor. See steps to reproduced below for details.
### What you think should happen instead
Tasks should get cleaned after scheduler restart and eventually get scheduled and executed.
### How to reproduce
After some code investigation, I reproduced this kind of behaviour on local environment and it seems that RESTARTING tasks are only properly handled if the original restarting task is gracefully shut down so it can mark task as UP_FOR_RETRY or at least there is a healthy scheduler to do it if they fail for any other reason. The problem is with the following scenario:
1. Task is initially in RUNNING state.
2. Scheduler process dies suddenly.
3. The task process also dies suddenly.
4. Clear command is issues so the task moves to RESTARTING state.
5. From now on, even if we restart scheduler, the task will never get scheduled or change its state. It needs to have its state manually fixed, e.g. by clearing it again.
A recording of steps to reproduce on local environment:
https://vimeo.com/857192666?share=copy
### Operating System
MacOS Ventura 13.4.1
### Versions of Apache Airflow Providers
N/A
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
N/A
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33661: Zombie tasks in RESTARTING state are not cleaned
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33661:
URL: https://github.com/apache/airflow/issues/33661#issuecomment-1690198116
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33661: Zombie tasks in RESTARTING state are not cleaned
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33661:
URL: https://github.com/apache/airflow/issues/33661#issuecomment-1692345607
Yep. Likely that should do the trick. Opening PR and dscussion there where we can drag attention of others might be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33661: Zombie tasks in RESTARTING state are not cleaned
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33661:
URL: https://github.com/apache/airflow/issues/33661#issuecomment-1692274809
it cannot be moved to scheduler, because of the support of various executors we have - namely Celery executor. Schduler does NOT control nor even really cares which workers are running the tasks. Celery executors work with QUEUE based mechanism, where Scheduler puts task to the queue, and workers pull tasks from the queue and once task is acknowledged further communication happens via the database - there is no other communication channel from scheduler to running tasks - it cannot simply order specifc tasks running on specific celery worker to restart.
Changing it woudl be a major change in how the whole architecture of scheduler -> executor -> worker workes. You would have to introduce a communication channel where scheduler could somehow signal specific celery worker to kill and restart task that runs on it. Currently it works in the way that there is a separate forked process that heartbeats the database and checks if the task state has been changed externally and kills child task running in order to restart it.
I am afraid the only option is to add a possibility of adopting tasks that are staying for too long in restarting state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bisk1 commented on issue #33661: Zombie tasks in RESTARTING state are not cleaned
Posted by "Bisk1 (via GitHub)" <gi...@apache.org>.
Bisk1 commented on issue #33661:
URL: https://github.com/apache/airflow/issues/33661#issuecomment-1690201811
I felt like this kind of situation should be handled by adopt_or_reset_orphaned_tasks() part of scheduler, but apparently that piece of code is meant to handle tasks that were orphaned by the scheduler process. In current implementation, the task process is responsible for a task in RESTARTING state, not the scheduler process.
I think that the responsibility for handling tasks in RESTARTING state should be shifted to scheduler. Then scheduler would be able to handle orphaned RESTARTING tasks. The task process itself would only handle SIGTERM to interrupt the restarted task (on best-effort basis).
My only concern is that it would cause race condition - what if the scheduler attempts to reschedule restarted task before the old task process handled SIGTERM - we could have 2 process with the same task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bisk1 commented on issue #33661: Zombie tasks in RESTARTING state are not cleaned
Posted by "Bisk1 (via GitHub)" <gi...@apache.org>.
Bisk1 commented on issue #33661:
URL: https://github.com/apache/airflow/issues/33661#issuecomment-1692295680
@potiuk ok, thanks for explanation.
I looked at the code again and now I can think we handle it with simpler change, just adding RESTARTING state to the list of resettable states that are used by scheduler when its `adopt_or_reset_orphaned_tasks().` RUNNING tasks are also handled this way and it doesn't seem to cause any problems with race conditions because by then they are either dead or can be successfully adopted - same case with RESTARTING tasks I assume.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #33661: Zombie tasks in RESTARTING state are not cleaned
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33661: Zombie tasks in RESTARTING state are not cleaned
URL: https://github.com/apache/airflow/issues/33661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org