[GitHub] spark pull request #20184: [SPARK-22987][Core] UnsafeExternalSorter cases OO...

[kiszk <gi...@git.apache.org>]

Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20184#discussion_r208993136
  
    --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java ---
    @@ -116,13 +138,18 @@ public void loadNext() throws IOException {
         if (taskContext != null) {
           taskContext.killTaskIfInterrupted();
         }
    -    recordLength = din.readInt();
    -    keyPrefix = din.readLong();
    -    if (recordLength > arr.length) {
    -      arr = new byte[recordLength];
    +    // check if the reader is closed to prevent reopen the in and din.
    +    if (!hasNext()) {
    +      throw new IndexOutOfBoundsException("Can not load next item when UnsafeSorterSpillReader is closed.");
    +    }
    +    recordLength = getDin().readInt();
    +    keyPrefix = getDin().readLong();
    +    int arrLength = Math.max(1024 * 1024, recordLength);
    +    if (arrLength > arr.length) {
    +      arr = new byte[arrLength];
           baseObject = arr;
         }
    -    ByteStreams.readFully(in, arr, 0, recordLength);
    +    ByteStreams.readFully(getIn(), arr, 0, recordLength);
    --- End diff --
    
    Is it fine if `recordLength` is greater than `1024 * 1024`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org