[jira] [Resolved] (SPARK-35546) Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

["Ye Zhou (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ye Zhou resolved SPARK-35546.
-----------------------------
       Fix Version/s: 3.2.0
    Target Version/s: 3.2.0
          Resolution: Fixed

Issue resolved by pull request 33078 and merged into Branch 3.2 and Master

https://github.com/apache/spark/pull/33078

> Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way 
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-35546
>                 URL: https://issues.apache.org/jira/browse/SPARK-35546
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Shuffle
>    Affects Versions: 3.1.0
>            Reporter: Ye Zhou
>            Priority: Major
>             Fix For: 3.2.0
>
>
> In the current implementation of RemoteBlockPushResolver, two ConcurrentHashmap are used to store #1 applicationId -> mergedShuffleLocalDirPath #2 applicationId+attemptId+shuffleID -> mergedShuffleParitionInfo. As there are four types of messages: ExecutorRegister, PushBlocks, FinalizeShuffleMerge and ApplicationRemove, will trigger different types of operations within these two hashmaps, it is required to maintain strong consistency about the informations stored in these two hashmaps. Otherwise, either there will be data corruption/correctness issues or memory leak in shuffle server. 
> We should come up with systematic way to resolve this, other than spot fixing the potential issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org