[jira] [Created] (SPARK-52090) FetchFailedException causing stage failures when gracefully decommissioning executors

["Sam Wheating (Jira)" <ji...@apache.org>]

Sam Wheating created SPARK-52090:
------------------------------------

             Summary: FetchFailedException causing stage failures when gracefully decommissioning executors
                 Key: SPARK-52090
                 URL: https://issues.apache.org/jira/browse/SPARK-52090
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes
    Affects Versions: 3.5.5
            Reporter: Sam Wheating


We are running Spark 3.5.5 on Kubernetes, with the following configuration:


|spark.decommission.enabled|true|
|spark.storage.decommission.enabled|true|

As well as a 600s terminationGracePeriod on the executor pods themselves.

When an executor is removed (for example, using `kubectl delete pod`), I can see the graceful decommissioning working as expected. The pod stays in a terminating state for a few minutes while it finishes its running tasks and migrates shuffle data to other executors. See relevant logs below:

 
{code:java}
25/05/12 18:58:43 INFO BlockManager: Starting block manager decommissioning process...
25/05/12 18:58:43 INFO CoarseGrainedExecutorBackend: Will exit when finished decommissioning
25/05/12 18:58:43 INFO BlockManagerDecommissioner: Starting block migration
25/05/12 18:58:43 INFO BlockManagerDecommissioner: Attempting to migrate all RDD blocks
25/05/12 18:58:43 INFO BlockManagerDecommissioner: Attempting to migrate all shuffle blocks
25/05/12 18:58:43 INFO BlockManagerDecommissioner: Start refreshing migratable shuffle blocks
...
25/05/12 19:00:42 INFO BlockManagerDecommissioner: Migrated migrate_shuffle_18_15196 (size: 356.4 MiB) to BlockManagerId(26, 10.101.0.143, 36127, None) in 3750 ms
...
25/05/12 19:01:13 INFO BlockManagerDecommissioner: Finished current round refreshing migratable shuffle blocks, waiting for 30000ms before the next round refreshing.
25/05/12 19:01:14 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
25/05/12 19:01:14 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
25/05/12 19:01:14 INFO CoarseGrainedExecutorBackend: No running tasks, all blocks migrated, stopping.
25/05/12 19:01:14 INFO CoarseGrainedExecutorBackend: Executor self-exiting due to : Finished decommissioning
25/05/12 19:01:14 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown{code}

However, as soon as the executor finishes decommissioning we tasks fail due to an ExecutorDeadException:
{code:java}
25/05/12 19:01:17 WARN TaskSetManager: Lost task 6.0 in stage 63.0 (TID 16410) (10.97.126.203 executor 14): FetchFailed(BlockManagerId(25, 10.101.0.142, 36601, None), shuffleId=18, mapIndex=3129, mapId=14579, reduceId=6, message=2025-05-12 19:01:18.896	at org.apache.spark.errors.SparkCoreErrors$.fetchFailedError(SparkCoreErrors.scala:437) 
...
Caused by: org.apache.spark.ExecutorDeadException: [INTERNAL_ERROR_NETWORK] The relative remote executor(Id: 25), which maintains the block data to fetch is dead.{code}

And then the driver logs the stage failure:
{code:java}
25/05/12 19:01:17 INFO DAGScheduler: ResultStage 63 (parquet at SparkJob.scala:499) failed in 94.460 s due to org.apache.spark.shuffle.FetchFailedException {code}

Based on the documentation, the loss of an executor should be recoverable without restarting the whole stage, assuming storage decommissioning is enabled, but it seems like this isn't the case in practice.

When running larger sparkApplications on kubernetes, we often see multiple executor evictions within a single execution (as we are aggresively scaling our clusters), so having multiple stage restarts is significantly slowing down our jobs.

Please let me know if we're missing any other configuration, or if there's any other information I can provide. Thanks!

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org