[GitHub] [incubator-doris] morningman commented on a change in pull request #6308: [Bug] Fix bug that broker can't load OSS/S3A files

[GitBox <gi...@apache.org>]

morningman commented on a change in pull request #6308:
URL: https://github.com/apache/incubator-doris/pull/6308#discussion_r677337957



##########
File path: fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
##########
@@ -561,22 +561,25 @@ public ByteBuffer pread(TBrokerFD fd, long offset, long length) {
                             currentStreamOffset, offset);
                 }
             }
-            ByteBuffer buf;
+            // Avoid using the ByteBuffer based read for Hadoop because some FSDataInputStream
+            // implementations are not ByteBufferReadable,
+            // See https://issues.apache.org/jira/browse/HADOOP-14603
+            byte[] buf;
             if (length > readBufferSize) {
-                buf = ByteBuffer.allocate(readBufferSize);
+                buf = new byte[readBufferSize];

Review comment:
       Could you help to test if the `readBufferSize` is larger than 128kb, can this `pread()` method read enough bytes that we want?
   The reason why we use `bytebuffer` is that when using `byte array`, it seems that it can not read more than 128kb.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org