You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2021/06/28 20:57:00 UTC
[jira] [Resolved] (SPARK-35910) Update remoteBlockBytes based on
merged block info
[ https://issues.apache.org/jira/browse/SPARK-35910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-35910.
-----------------------------------
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 33109
[https://github.com/apache/spark/pull/33109]
> Update remoteBlockBytes based on merged block info
> --------------------------------------------------
>
> Key: SPARK-35910
> URL: https://issues.apache.org/jira/browse/SPARK-35910
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 3.2.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Fix For: 3.2.0
>
>
> Currently, we calculate the `remoteBlockBytes` based on the original block info list. If the original reducer size is big but the actual reducer size is small due to automatically partition coalescing of AQE, the reducer will take more time to calculate `remoteBlockBytes`. We can reduce this cost via remote requests which contain merged block info lists.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org