You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Min Shen (Jira)" <ji...@apache.org> on 2021/06/09 00:27:00 UTC

[jira] [Commented] (SPARK-35426) When addMergerLocation exceed the maxRetainedMergerLocations , we should remove the merger based on merged shuffle data size.

    [ https://issues.apache.org/jira/browse/SPARK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359673#comment-17359673 ] 

Min Shen commented on SPARK-35426:
----------------------------------

[~zhuqi], the retained mergers are meant for choosing merger locations for shuffles.

When we remove the merger from the retained list, the merged shuffle data on the corresponding shuffle service is not removed.

The driver would still track these locations inside MapOutputTracker if the removed merger holds shuffle data for shuffles that haven't been cleaned yet.

Are you suggesting that, we should remove mergers with the largest amount of merged shuffle data, so that the remaining mergers have potentially more disk space to store new merged shuffle data?

> When addMergerLocation exceed the maxRetainedMergerLocations , we should remove the merger based on merged shuffle data size.
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-35426
>                 URL: https://issues.apache.org/jira/browse/SPARK-35426
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 3.2.0
>            Reporter: Qi Zhu
>            Priority: Major
>
> Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size. 
> The oldest merger may have big merged shuffle data size, it will not be a good choice to do so.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org