[GitHub] spark pull request #20461: [SPARK-23289][CORE]OneForOneBlockFetcher.Download...

[zsxwing <gi...@git.apache.org>]

Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20461#discussion_r165225508
  
    --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ---
    @@ -171,7 +171,9 @@ private void failRemainingBlocks(String[] failedBlockIds, Throwable e) {
     
         @Override
         public void onData(String streamId, ByteBuffer buf) throws IOException {
    -      channel.write(buf);
    +      while (buf.hasRemaining()) {
    +        channel.write(buf);
    --- End diff --
    
    I don't hit this issue and it may be just because this feature is disabled by default.
    
    Just to be safe as [FileDispatcherImpl](http://www.grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileChannelImpl.java#210) calls `write/pwrite` system call but doesn't check the written size. Seems the written size may be less than the buffer size: https://stackoverflow.com/questions/32683086/handling-incomplete-write-calls


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org