You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2025/09/12 23:28:37 UTC
[PR] Be a little clearer when task likely OOM error [airflow]
dstandish opened a new pull request, #55602:
URL: https://github.com/apache/airflow/pull/55602
When a task process is terminated with exit code -9 (sigkill) then there's a good chance it's due to out of memory error (OOM). We can be a little more direct about indicating that in the logs. And I reorganize the way we present that in the docs a bit.
closes #55599
cc @collinmcnulty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Be a little clearer when task likely OOM error [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #55602:
URL: https://github.com/apache/airflow/pull/55602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Be a little clearer when task likely OOM error [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #55602:
URL: https://github.com/apache/airflow/pull/55602#discussion_r2349596630
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -36,16 +36,26 @@ Below are some example scenarios that could cause a task's state to change by a
- A user marked the task as successful or failed in the Airflow UI.
- An external script or process used the :doc:`Airflow REST API <stable-rest-api-ref>` to change the state of a task.
-TaskRunner killed
------------------
+Process terminated by signal
+----------------------------
Sometimes, Airflow or some adjacent system will kill a task instance's ``TaskRunner``, causing the task instance to fail.
-Here are some examples that could cause such an event:
+Below we discuss a few common cases.
-- A Dag run timeout, specified by ``dagrun_timeout`` in the Dag's definition.
-- An Airflow worker running out of memory
- - Usually, Airflow workers that run out of memory receive a SIGKILL, and the scheduler will fail the corresponding task instance for not having a heartbeat. However, in some scenarios, Airflow kills the task before that happens.
+Dag run timeout
+"""""""""""""""
+
+A dag run timeout can be specified by ``dagrun_timeout`` in the dag's definition.
+The task process would likely be killed with SIGTERM (exit code -15).
+
+Out of memory error (OOM)
+"""""""""""""""""""""""""
+
+When a task process consumes too much memory for a worker, the best case scenario is it is killed
+with SIGTERM (exit code -9). Depending on configuration and infrastructure, it is also
Review Comment:
https://github.com/apache/airflow/pull/55690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Be a little clearer when task likely OOM error [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #55602:
URL: https://github.com/apache/airflow/pull/55602#discussion_r2349591691
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -36,16 +36,26 @@ Below are some example scenarios that could cause a task's state to change by a
- A user marked the task as successful or failed in the Airflow UI.
- An external script or process used the :doc:`Airflow REST API <stable-rest-api-ref>` to change the state of a task.
-TaskRunner killed
------------------
+Process terminated by signal
+----------------------------
Sometimes, Airflow or some adjacent system will kill a task instance's ``TaskRunner``, causing the task instance to fail.
-Here are some examples that could cause such an event:
+Below we discuss a few common cases.
-- A Dag run timeout, specified by ``dagrun_timeout`` in the Dag's definition.
-- An Airflow worker running out of memory
- - Usually, Airflow workers that run out of memory receive a SIGKILL, and the scheduler will fail the corresponding task instance for not having a heartbeat. However, in some scenarios, Airflow kills the task before that happens.
+Dag run timeout
+"""""""""""""""
+
+A dag run timeout can be specified by ``dagrun_timeout`` in the dag's definition.
+The task process would likely be killed with SIGTERM (exit code -15).
+
+Out of memory error (OOM)
+"""""""""""""""""""""""""
+
+When a task process consumes too much memory for a worker, the best case scenario is it is killed
+with SIGTERM (exit code -9). Depending on configuration and infrastructure, it is also
Review Comment:
i will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Be a little clearer when task likely OOM error [airflow]
Posted by "collinmcnulty (via GitHub)" <gi...@apache.org>.
collinmcnulty commented on code in PR #55602:
URL: https://github.com/apache/airflow/pull/55602#discussion_r2349328566
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -36,16 +36,26 @@ Below are some example scenarios that could cause a task's state to change by a
- A user marked the task as successful or failed in the Airflow UI.
- An external script or process used the :doc:`Airflow REST API <stable-rest-api-ref>` to change the state of a task.
-TaskRunner killed
------------------
+Process terminated by signal
+----------------------------
Sometimes, Airflow or some adjacent system will kill a task instance's ``TaskRunner``, causing the task instance to fail.
-Here are some examples that could cause such an event:
+Below we discuss a few common cases.
-- A Dag run timeout, specified by ``dagrun_timeout`` in the Dag's definition.
-- An Airflow worker running out of memory
- - Usually, Airflow workers that run out of memory receive a SIGKILL, and the scheduler will fail the corresponding task instance for not having a heartbeat. However, in some scenarios, Airflow kills the task before that happens.
+Dag run timeout
+"""""""""""""""
+
+A dag run timeout can be specified by ``dagrun_timeout`` in the dag's definition.
+The task process would likely be killed with SIGTERM (exit code -15).
+
+Out of memory error (OOM)
+"""""""""""""""""""""""""
+
+When a task process consumes too much memory for a worker, the best case scenario is it is killed
+with SIGTERM (exit code -9). Depending on configuration and infrastructure, it is also
Review Comment:
-9 is SIGKILL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org