[jira] [Resolved] (SPARK-52509) Fallback storage accumulates removed shuffle data

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-52509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-52509.
-----------------------------------
    Fix Version/s: 4.1.0
       Resolution: Fixed

Issue resolved by pull request 51199
[https://github.com/apache/spark/pull/51199]

> Fallback storage accumulates removed shuffle data
> -------------------------------------------------
>
>                 Key: SPARK-52509
>                 URL: https://issues.apache.org/jira/browse/SPARK-52509
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Kubernetes
>    Affects Versions: 4.1.0
>            Reporter: Enrico Minack
>            Assignee: Enrico Minack
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> The fallback storage removes migrated shuffle data on Spark context shutdown. Ideally, it should remove individual shuffles once they get removed from the Spark context. Otherwise, all shuffle data ever migrated to the fallback storage accumulate on the remote storage until the Spark context shuts down. For long running jobs with a lot of decommissioning, this can be orders of magnitude more data than what is actively been used / usable / referenced.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org