[jira] [Resolved] (SPARK-55075) Track executor pod creation errors with ExecutorFailureTracker

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-55075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-55075.
-----------------------------------
    Fix Version/s: 4.2.0
       Resolution: Fixed

Issue resolved by pull request 53840
[https://github.com/apache/spark/pull/53840]

> Track executor pod creation errors with ExecutorFailureTracker
> --------------------------------------------------------------
>
>                 Key: SPARK-55075
>                 URL: https://issues.apache.org/jira/browse/SPARK-55075
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Kubernetes
>    Affects Versions: 4.2.0
>            Reporter: Parth Chandra
>            Assignee: Parth Chandra
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0
>
>
> On K8s, we may encounter failures in creating pods which are frequently recoverable errors, but sometimes may be unrecoverable.
> Spark currently handles pod creation failures as recoverable and will keep attempting to create new executors which is not ideal if the failure cannot be recovered from.
> Additionally, in some cases, in some cases the same error may be either recoverable or unrecoverable. For instance an ImagePullErr may be caused by a temporary error in accessing the container registry (recoverable) or may be because of a missing image (unrecoverable). 
> One way to handle this is to retry pod creation with backoff when we do get to know that pod creation has failed. In addition, if the number of failures exceeds a threshold then fail the application so that the user knows that there is a problem in the cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org