You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2018/02/09 14:45:10 UTC
spark git commit: [SPARK-23358][CORE] When the number of partitions
is greater than 2^28, it will result in an error result
Repository: spark
Updated Branches:
refs/heads/master 4b4ee2601 -> f77270b88
[SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in an error result
## What changes were proposed in this pull request?
In the `checkIndexAndDataFile`,the `blocks` is the ` Int` type, when it is greater than 2^28, `blocks*8` will overflow, and this will result in an error result.
In fact, `blocks` is actually the number of partitions.
## How was this patch tested?
Manual test
Author: liuxian <li...@zte.com.cn>
Closes #20544 from 10110346/overflow.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f77270b8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f77270b8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f77270b8
Branch: refs/heads/master
Commit: f77270b8811bbd8956d0c08fa556265d2c5ee20e
Parents: 4b4ee26
Author: liuxian <li...@zte.com.cn>
Authored: Fri Feb 9 08:45:06 2018 -0600
Committer: Sean Owen <so...@cloudera.com>
Committed: Fri Feb 9 08:45:06 2018 -0600
----------------------------------------------------------------------
.../scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/f77270b8/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
index c5f3f6e..d88b25c 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
@@ -84,7 +84,7 @@ private[spark] class IndexShuffleBlockResolver(
*/
private def checkIndexAndDataFile(index: File, data: File, blocks: Int): Array[Long] = {
// the index file should have `block + 1` longs as offset.
- if (index.length() != (blocks + 1) * 8) {
+ if (index.length() != (blocks + 1) * 8L) {
return null
}
val lengths = new Array[Long](blocks)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org