You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/04 00:41:00 UTC
[jira] [Updated] (SPARK-59972) Dynamic allocation can stall after an unfinished task is killed
[ https://issues.apache.org/jira/browse/SPARK-59972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-59972:
-----------------------------------
Labels: pull-request-available (was: )
> Dynamic allocation can stall after an unfinished task is killed
> ---------------------------------------------------------------
>
> Key: SPARK-59972
> URL: https://issues.apache.org/jira/browse/SPARK-59972
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.3.0
> Reporter: MJ Deng
> Priority: Major
> Labels: pull-request-available
>
> ExecutorAllocationManager does not count an unfinished regular task as pending again when its running attempt is explicitly killed.
> A possible event sequence is:
> # A task starts on an available executor while the executor target is zero.
> # Starting the task drains the scheduler queue and clears the scheduler backlog timer.
> # SparkContext.killTaskAttempt kills the task before it succeeds.
> # The scheduler requeues the unfinished task.
> # ExecutorAllocationListener.onTaskEnd ignores TaskKilled, so it does not mark the task as pending or restart the backlog timer.
> # If the available executor is subsequently lost before the task relaunches, dynamic allocation does not request a replacement executor and the stage can stall.
> TaskKilled cannot always be treated as pending because a regular attempt may be killed after a speculative copy has already succeeded, as covered by SPARK-30511.
> The listener should therefore count a killed regular task as pending and restart the backlog timer only when:
> * the stage attempt is still active;
> * the killed attempt is non-speculative;
> * the task index has not already succeeded; and
> * the task index was known to be running.
> Killed speculative attempts and task-end events from completed stages should remain
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org