[GitHub] [parquet-mr] steveloughran commented on a diff in pull request #983: WiP: parquet to use openfile api and some other performance enhancements

[GitBox <gi...@apache.org>]

steveloughran commented on code in PR #983:
URL: https://github.com/apache/parquet-mr/pull/983#discussion_r941640533


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/HadoopInputFile.java:
##########
@@ -66,7 +68,13 @@ public long getLength() {
 
   @Override
   public SeekableInputStream newStream() throws IOException {
-    return HadoopStreams.wrap(fs.open(stat.getPath()));
+    // specify random access always
+    return HadoopStreams.wrap(
+      awaitFuture(fs.openFile(stat.getPath())

Review Comment:
   like I said, WiP 😄 
   
   the shim library I'm doing does retrofit this to older versions, but I'm doing this to see what opportunities there are with basic adoption of 3.3.x apis. (https://github.com/steveloughran/fs-api-shim)
   
   Mukund intends to add vectored stream reads to that shim lib too falling back to the positioned read operations, for versions <= 3.3.4



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org