You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pankajkoti (via GitHub)" <gi...@apache.org> on 2023/11/23 20:00:24 UTC
[PR] Enhance docs for zombie tasks [airflow]
pankajkoti opened a new pull request, #35825:
URL: https://github.com/apache/airflow/pull/35825
closes: #35698
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404227678
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
Review Comment:
yes, okay, removed the code block. The description is essentially explaining the where clauses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404011284
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
Review Comment:
We shouldn't go into details explaining the code. I'm okay with pointing out the code location but not with the code explanation because the code can change at any time and we might not remember to update this explanation. Similarly, the reproduction steps for zombie tasks are not necessary for a concept doc IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #35825:
URL: https://github.com/apache/airflow/pull/35825#issuecomment-1824867590
cc: @vatsrahul1001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404113452
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
+
+2. **Job State and Heartbeat Check**
+
+ Zombie tasks are identified if the associated job is not in the RUNNING state or if the latest heartbeat of the job is
+ earlier than the calculated time threshold (limit_dttm). The heartbeat is a mechanism to indicate that a task or job is
+ still alive and running.
+
+.. code-block::
+
+ .where(
+ or_(
+ Job.state != JobState.RUNNING,
+ Job.latest_heartbeat < limit_dttm,
+ )
+ )
+
+3. **Job Type**
+
+ The job associated with the task must be of type "LocalTaskJob."
+
+.. code-block::
+
+ .where(Job.job_type == "LocalTaskJob")
+
+4. **Queued by Job ID**
+
+ Only tasks queued by the same job that is currently being processed are considered.
+
+.. code-block::
+
+ .where(TI.queued_by_job_id == self.job.id)
+
+These conditions collectively help identify running tasks that may be zombies based on their state, associated job
+state, heartbeat status, job type, and the specific job that queued them. If a task meets these criteria, it is
+considered a potential zombie, and further actions, such as logging and sending a callback request, are taken.
+
+Reproducing zombie tasks locally
Review Comment:
I like the example of reproducing how it works - this is a good way of explaining things by showing people how they can experiment with it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404115850
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
Review Comment:
I think it's better to explain these it in SQL terms - translating this to WHERE statement would be more widely understood by the users
Not many of our users know ORM/sqlalchemy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404227678
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
Review Comment:
yes, okay, removed the code block. The existing description is essentially explaining the where clauses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404111057
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
Review Comment:
Yeah agree with @ephraimbuddy that showing airflow code internals in this case is a bit too much . This description is really valuable because it explains WHAT happens and WHEN (and this part is cool) . But showing the details HOW is not very useful. Users always can go and check it and when it will change it will get confusing a bit for them I think.
I'd just remove the implementation and leave all the rest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #35825:
URL: https://github.com/apache/airflow/pull/35825#issuecomment-1824864322
Preview of the locally built docs:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti closed pull request #35825: Enhance docs for zombie tasks
URL: https://github.com/apache/airflow/pull/35825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404218105
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
+
+2. **Job State and Heartbeat Check**
+
+ Zombie tasks are identified if the associated job is not in the RUNNING state or if the latest heartbeat of the job is
+ earlier than the calculated time threshold (limit_dttm). The heartbeat is a mechanism to indicate that a task or job is
+ still alive and running.
+
+.. code-block::
+
+ .where(
+ or_(
+ Job.state != JobState.RUNNING,
+ Job.latest_heartbeat < limit_dttm,
+ )
+ )
+
+3. **Job Type**
+
+ The job associated with the task must be of type "LocalTaskJob."
+
+.. code-block::
+
+ .where(Job.job_type == "LocalTaskJob")
+
+4. **Queued by Job ID**
+
+ Only tasks queued by the same job that is currently being processed are considered.
+
+.. code-block::
+
+ .where(TI.queued_by_job_id == self.job.id)
+
+These conditions collectively help identify running tasks that may be zombies based on their state, associated job
+state, heartbeat status, job type, and the specific job that queued them. If a task meets these criteria, it is
+considered a potential zombie, and further actions, such as logging and sending a callback request, are taken.
+
+Reproducing zombie tasks locally
Review Comment:
yes, this was @vatsrahul1001's smart way of reproducing :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404111057
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
Review Comment:
Yeah agree with @ephraimbuddy that showing airflow code internals in this case is a bit too much . This description is really valuable because it explains WHT happens and WHEN (and this part is cool) . But showing the details is not very useful. Users always can go and check it and when it will change it will get confusing a bit for them I think.
I'd just remove the implementation and leave all the rest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404086190
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
Review Comment:
okay. I initially wrote in the ticket description that it would nice to explain how exactly zombies are detected and also steps for reproducing them https://github.com/apache/airflow/issues/35698
and hence added here thinking it might be helpful :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #35825:
URL: https://github.com/apache/airflow/pull/35825#discussion_r1404163077
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
+
+2. **Job State and Heartbeat Check**
+
+ Zombie tasks are identified if the associated job is not in the RUNNING state or if the latest heartbeat of the job is
+ earlier than the calculated time threshold (limit_dttm). The heartbeat is a mechanism to indicate that a task or job is
+ still alive and running.
+
+.. code-block::
+
+ .where(
+ or_(
+ Job.state != JobState.RUNNING,
+ Job.latest_heartbeat < limit_dttm,
+ )
+ )
+
+3. **Job Type**
+
+ The job associated with the task must be of type "LocalTaskJob."
+
+.. code-block::
+
+ .where(Job.job_type == "LocalTaskJob")
+
+4. **Queued by Job ID**
+
+ Only tasks queued by the same job that is currently being processed are considered.
+
+.. code-block::
+
+ .where(TI.queued_by_job_id == self.job.id)
+
+These conditions collectively help identify running tasks that may be zombies based on their state, associated job
+state, heartbeat status, job type, and the specific job that queued them. If a task meets these criteria, it is
+considered a potential zombie, and further actions, such as logging and sending a callback request, are taken.
+
+Reproducing zombie tasks locally
Review Comment:
This looks way smarter than how I reproduce the zombie task 🙌
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
+
+2. **Job State and Heartbeat Check**
+
+ Zombie tasks are identified if the associated job is not in the RUNNING state or if the latest heartbeat of the job is
+ earlier than the calculated time threshold (limit_dttm). The heartbeat is a mechanism to indicate that a task or job is
+ still alive and running.
+
+.. code-block::
+
+ .where(
+ or_(
+ Job.state != JobState.RUNNING,
+ Job.latest_heartbeat < limit_dttm,
+ )
+ )
+
+3. **Job Type**
+
+ The job associated with the task must be of type "LocalTaskJob."
+
+.. code-block::
+
+ .where(Job.job_type == "LocalTaskJob")
+
+4. **Queued by Job ID**
+
+ Only tasks queued by the same job that is currently being processed are considered.
+
+.. code-block::
+
+ .where(TI.queued_by_job_id == self.job.id)
+
+These conditions collectively help identify running tasks that may be zombies based on their state, associated job
+state, heartbeat status, job type, and the specific job that queued them. If a task meets these criteria, it is
+considered a potential zombie, and further actions, such as logging and sending a callback request, are taken.
+
+Reproducing zombie tasks locally
Review Comment:
This looks way smarter than how I reproduce the zombie task 🙌
##########
docs/apache-airflow/core-concepts/tasks.rst:
##########
@@ -243,9 +243,103 @@ Zombie/Undead Tasks
No system runs perfectly, and task instances are expected to die once in a while. Airflow detects two kinds of task/process mismatch:
-* *Zombie tasks* are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
+* *Zombie tasks* are ``TaskInstances`` stuck in a ``running`` state despite their associated jobs being inactive
+ (e.g. their process didn't send a recent heartbeat as it got killed, or the machine died). Airflow will find these
+ periodically, clean them up, and either fail or retry the task depending on its settings.
+
+* *Undead tasks* are tasks that are *not* supposed to be running but are, often caused when you manually edit Task
+ Instances via the UI. Airflow will find them periodically and terminate them.
+
+
+Below is the code snippet from the Airflow scheduler that runs periodically to detect zombie/undead tasks.
+
+.. exampleinclude:: /../../airflow/jobs/scheduler_job_runner.py
+ :language: python
+ :start-after: [START find_zombies]
+ :end-before: [END find_zombies]
+
+
+The explanation of the criteria used in the above snippet to detect zombie tasks is as below:
+
+1. **Task Instance State**
+
+ Only task instances in the RUNNING state are considered potential zombies.
+
+.. code-block::
+
+ .where(TI.state == TaskInstanceState.RUNNING)
Review Comment:
If that is the case, I guess we could remove all the codes here and replace them with description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance docs for zombie tasks [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti merged PR #35825:
URL: https://github.com/apache/airflow/pull/35825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org