You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Szehon Ho (Jira)" <ji...@apache.org> on 2026/09/27 07:23:00 UTC
[jira] [Assigned] (SPARK-59138) Preserve reliably-stored shuffle map outputs on executor loss
[ https://issues.apache.org/jira/browse/SPARK-59138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Szehon Ho reassigned SPARK-59138:
---------------------------------
Assignee: Apache Spark
> Preserve reliably-stored shuffle map outputs on executor loss
> -------------------------------------------------------------
>
> Key: SPARK-59138
> URL: https://issues.apache.org/jira/browse/SPARK-59138
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.3
> Reporter: Venkata krishnan Sowrirajan
> Assignee: Apache Spark
> Priority: Major
> Labels: pull-request-available
>
> On executor loss, DAGScheduler unregisters all map outputs on the lost executor and forces the map stage to recompute. When a shuffle's output is reliably stored off-executor (e.g. a remote shuffle service such as Celeborn), the data survives the executor, so this recompute is wasteful.
> Today reliability is expressed only through ShuffleDriverComponents.supportsReliableStorage(), a single application-wide flag. That cannot represent a mixed setup where one shuffle lives on the remote service while another falls back to local disk on the same executor (e.g. Celeborn AUTO fallback). As a result Spark either recomputes reliable shuffles unnecessarily, or would skip recompute for local-disk fallback shuffles whose output was genuinely lost.
> Make reliability per-shuffle:
> - Add ShuffleHandle.isReliablyStored (default false), overridable by a ShuffleManager that routes a shuffle to reliable storage.
> - Store the bit per shuffle in MapOutputTracker and skip only reliably-stored shuffles when unregistering outputs on executor/worker loss. Genuine fetch failures continue to unregister everything.
> - Honor the per-shuffle bit in TaskSetManager's re-run gate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org