[jira] [Commented] (SPARK-52509) Fallback storage accumulates removed shuffle data

["Subham Singhal (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-52509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17998372#comment-17998372 ] 

Subham Singhal commented on SPARK-52509:
----------------------------------------

[~enricomi] Do we have to create a custom Spark event(UnregisterShuffle) and add a listener which will remove shuffleId from fallback storage?

> Fallback storage accumulates removed shuffle data
> -------------------------------------------------
>
>                 Key: SPARK-52509
>                 URL: https://issues.apache.org/jira/browse/SPARK-52509
>             Project: Spark
>          Issue Type: Sub-task
>          Components: k8s, Kubernetes
>    Affects Versions: 4.1.0
>            Reporter: Enrico Minack
>            Priority: Major
>
> The fallback storage removes migrated shuffle data on Spark context shutdown. Ideally, it should remove individual shuffles once they get removed from the Spark context. Otherwise, all shuffle data ever migrated to the fallback storage accumulate on the remote storage until the Spark context shuts down. For long running jobs with a lot of decommissioning, this can be orders of magnitude more data than what is actively been used / usable / referenced.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org