[jira] [Updated] (SPARK-44526) Porting k8s PVC reuse logic to spark standalone

["Faiz Halde (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-44526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Faiz Halde updated SPARK-44526:
-------------------------------
    Affects Version/s: 3.5.0
                           (was: 3.4.1)

> Porting k8s PVC reuse logic to spark standalone
> -----------------------------------------------
>
>                 Key: SPARK-44526
>                 URL: https://issues.apache.org/jira/browse/SPARK-44526
>             Project: Spark
>          Issue Type: New Feature
>          Components: Shuffle, Spark Core
>    Affects Versions: 3.5.0
>            Reporter: Faiz Halde
>            Priority: Major
>
> Hi,
> This ticket is meant to understand the work that would be involved in porting the k8s PVC reuse feature onto the spark standalone cluster manager which reuses the shuffle files present locally in the disk
> We are a heavy user of spot instances and we suffer from spot terminations impacting our long running jobs
> The logic in `KubernetesLocalDiskShuffleExecutorComponents` itself is not that much. However when I tried this on the `LocalDiskShuffleExecutorComponents` it was not a successful experiment which suggests there is more to recovering shuffle files
> I'd like to understand what will be the work involved for this. We'll be more than happy to contribute



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org