You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "lan1994 (via GitHub)" <gi...@apache.org> on 2026/07/24 05:21:01 UTC

[I] [K8S][CORE] Heartbeat-timeout executor replacement may be classified as ExecutorKilled and skip task failure counting [spark]

lan1994 opened a new issue, #57494:
URL: https://github.com/apache/spark/issues/57494

   **Version**
   spark 3.5+ (k8s)
   
   **Description**
   I observed a behavior difference between Spark on Kubernetes and YARN for executor heartbeat-timeout scenarios.
   In heartbeat timeout handling, HeartbeatReceiver triggers both:
   1. sc.killAndReplaceExecutor(executorId) (which uses countFailures=true)
   2. RemoveExecutor(executorId, ExecutorProcessLost(timeout))
   However, in Kubernetes backend, doKillExecutors() historically removes executors eagerly with ExecutorKilled.
   This can win the race against the later timeout-based RemoveExecutor(...ExecutorProcessLost...), so the final loss reason becomes ExecutorKilled.
   Since ExecutorKilled is treated as non-app-caused failure (exitCausedByApp=false), task failures may not be counted towards spark.task.maxFailures, and fast-fail is less likely to happen.
   
   **Impact**
   
   1. Heartbeat-timeout task failures may bypass failure counting.
   2. Jobs can retry for a long time instead of fast-failing.
   3. Behavior diverges from YARN under similar failure conditions.
   
   **Suggested fix direction**
   In K8s doKillExecutors, preserve countFailures semantics from killExecutors:
   Keep ExecutorKilled for explicit driver-initiated non-counting kills.
   For replacement kills (countFailures=true), avoid eagerly converting to ExecutorKilled; keep a non-ExecutorKilled reason (or allow timeout reason to land) so task-failure counting semantics are preserved.
   If this direction is acceptable to maintainers, I can contribute a PR with a minimal fix and corresponding unit tests.
   
   
   **Additional question:** 
   Could maintainers advise how to obtain an Apache JIRA account for new contributors?
   
   <img width="583" height="279" alt="Image" src="https://github.com/user-attachments/assets/06ad189c-2cf7-42bc-8e42-38d12e135ef5" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] [K8S][CORE] Heartbeat-timeout executor replacement may be classified as ExecutorKilled and skip task failure counting [spark]

Posted by "lan1994 (via GitHub)" <gi...@apache.org>.
lan1994 commented on issue #57494:
URL: https://github.com/apache/spark/issues/57494#issuecomment-5179623294

   @cloud-fan  If you get a chance, could you take a look at this issue? I'd really appreciate it. Also, if you could send me a Jira invite, that would be amazing—thanks a ton!
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] [K8S][CORE] Heartbeat-timeout executor replacement may be classified as ExecutorKilled and skip task failure counting [spark]

Posted by "github-actions[bot] (via GitHub) via reviews" <re...@spark.apache.org>.
github-actions[bot] closed issue #57494: [K8S][CORE] Heartbeat-timeout executor replacement may be classified as ExecutorKilled and skip task failure counting
URL: https://github.com/apache/spark/issues/57494


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org