You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/08 17:56:07 UTC

[GitHub] [airflow] lindsable opened a new pull request #18095: Kubernetes Executor Bug Fix: Set task state to failed when pod is DELETED while running

lindsable opened a new pull request #18095:
URL: https://github.com/apache/airflow/pull/18095

closes: #17693

As is described in the linked issue there is a bug in the Kubernetes Job Watcher that occurs when a node with a running worker pod is removed from the cluster. If the worker pod doesn't completed before the node is removed, it is orphaned and force deleted by the garbage collector. This is communicated by the api with a status='RUNNING' but an event with type='DELETED'

Because in the if statement the Job Watcher doesn't check the event type, the last information we get from the pod is that is it running. The running scheduler never gets any information about the pod and shows it as stuck in a queued state. This situation is fixed when the scheduler/executor restarts and [this](https://github.com/apache/airflow/blob/ff64fe84857a58c4f6e47ec3338b576125c4223f/airflow/executors/kubernetes_executor.py#L433) function is run.

A mitigation for this bug is to set the execution_timeout for the dag so the run will be stopped or restarting the scheduler on an interval.

---

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ephraimbuddy merged pull request #18095: Kubernetes Executor Bug Fix: Set task state to failed when pod is DELETED while running

Posted by GitBox <gi...@apache.org>.

ephraimbuddy merged pull request #18095:
URL: https://github.com/apache/airflow/pull/18095


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #18095: Kubernetes Executor Bug Fix: Set task state to failed when pod is DELETED while running

Posted by GitBox <gi...@apache.org>.

ephraimbuddy commented on pull request #18095:
URL: https://github.com/apache/airflow/pull/18095#issuecomment-917405437


   @lindsable please rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] lindsable commented on pull request #18095: Kubernetes Executor Bug Fix: Set task state to failed when pod is DELETED while running

Posted by GitBox <gi...@apache.org>.

lindsable commented on pull request #18095:
URL: https://github.com/apache/airflow/pull/18095#issuecomment-917426093


   @ephraimbuddy done!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #18095: Kubernetes Executor Bug Fix: Set task state to failed when pod is DELETED while running

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on pull request #18095:
URL: https://github.com/apache/airflow/pull/18095#issuecomment-917242783


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org