You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Qi Zhu (Jira)" <ji...@apache.org> on 2021/08/17 09:52:00 UTC
[jira] [Updated] (SPARK-35426) When addMergerLocation exceed the
maxRetainedMergerLocations , we should remove the merger based on merged
shuffle data size.
[ https://issues.apache.org/jira/browse/SPARK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Qi Zhu updated SPARK-35426:
---------------------------
Description:
Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size.
was:
Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size.
The oldest merger may have big merged shuffle data size, it will not be a good choice to do so.
> When addMergerLocation exceed the maxRetainedMergerLocations , we should remove the merger based on merged shuffle data size.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-35426
> URL: https://issues.apache.org/jira/browse/SPARK-35426
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Qi Zhu
> Priority: Major
>
> Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org