You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Enrico Minack (Jira)" <ji...@apache.org> on 2025/07/07 20:54:00 UTC

[jira] [Commented] (SPARK-52506) Migrate to fallback storage only

    [ https://issues.apache.org/jira/browse/SPARK-52506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18003622#comment-18003622 ] 

Enrico Minack commented on SPARK-52506:
---------------------------------------

This is fixed in [https://github.com/apache/spark/pull/51201.]

> Migrate to fallback storage only
> --------------------------------
>
>                 Key: SPARK-52506
>                 URL: https://issues.apache.org/jira/browse/SPARK-52506
>             Project: Spark
>          Issue Type: Sub-task
>          Components: k8s, Kubernetes
>    Affects Versions: 4.1.0
>            Reporter: Enrico Minack
>            Priority: Major
>
> Using the storage decommissioning feature with fallback storage on Kubernetes allows to migrate shuffle data from executors onto an external cloud storage. Currently, this fallback storage can only be used in conjunction with migrating shuffle data to other executors. However, those executors are also subject of eventual decommissioning, so shuffle data are potentially migrated multiple times.
> The user should have the option to migrate shuffle data at most once, and that is to the fallback storage only.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org