You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2018/05/07 06:35:00 UTC
[jira] [Resolved] (SPARK-24160) ShuffleBlockFetcherIterator should
fail if it receives zero-size blocks
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-24160.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 21219
[https://github.com/apache/spark/pull/21219]
> ShuffleBlockFetcherIterator should fail if it receives zero-size blocks
> -----------------------------------------------------------------------
>
> Key: SPARK-24160
> URL: https://issues.apache.org/jira/browse/SPARK-24160
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.3.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
> Fix For: 2.4.0
>
>
> In the shuffle layer, we guarantee that zero-size blocks will never be requested (a block containing zero records is always 0 bytes in size and is marked as empty such that it will never be legitimately requested by executors). However, we failed to take advantage of this in the shuffle-read path: the existing code did not explicitly check whether blocks are non-zero-size.
>
> We should add `buf.size != 0` checks to ShuffleBlockFetcherIterator to take advantage of this invariant and prevent potential data loss / corruption issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org