[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

[squito <gi...@git.apache.org>]

Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21593#discussion_r210094158
  
    --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java ---
    @@ -137,30 +137,15 @@ protected void deallocate() {
       }
     
       private int copyByteBuf(ByteBuf buf, WritableByteChannel target) throws IOException {
    -    ByteBuffer buffer = buf.nioBuffer();
    -    int written = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
    -      target.write(buffer) : writeNioBuffer(target, buffer);
    +    // SPARK-24578: cap the sub-region's size of returned nio buffer to improve the performance
    +    // for the case that the passed-in buffer has too many components.
    +    int length = Math.min(buf.readableBytes(), NIO_BUFFER_LIMIT);
    +    ByteBuffer buffer = buf.nioBuffer(buf.readerIndex(), length);
    --- End diff --
    
    for anybody watching this, eventually SPARK-25115 was opened (currently has a PR)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org