You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by GitBox <gi...@apache.org> on 2022/10/12 11:15:36 UTC

[GitHub] [orc] HarshitGupta11 opened a new pull request, #1276: ORC-1251:Support Vectored IO in ORC [Draft]

HarshitGupta11 opened a new pull request, #1276:
URL: https://github.com/apache/orc/pull/1276

   Support the new fast Vectored Read API in ORC for parallel reads. 
   The patch was tested internally along with S3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "williamhyun (via GitHub)" <gi...@apache.org>.
williamhyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1869172553

   Let me take over this for ORC 2.0.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] steveloughran commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1554918583

   @HarshitGupta11 can you rebase this to orc main and against release hadoop 3.3.5.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1848783089

   Let me close this PR since this commit has been stale over one year already.
   
   ![Screenshot 2023-12-09 at 3 43 56 PM](https://github.com/apache/orc/assets/9700541/d2c537e7-79ce-4858-bce3-2572ac01c8fc)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1868852105

   @mukund-thakur @HarshitGupta11 Do you have any plan to revive this work?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] jerqi commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1614033863

   > Support the new fast Vectored Read API in ORC for parallel reads. The patch was tested internally along with S3
   
   Could you share some performance benchmark tests? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] steveloughran commented on a diff in pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #1276:
URL: https://github.com/apache/orc/pull/1276#discussion_r1199185617


##########
java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java:
##########
@@ -488,6 +504,65 @@ private static BufferChunk findSingleRead(BufferChunk first, long minSeekSize) {
     return last;
   }
 
+  /**
+   * Convert from DiskRangeList to FileRange
+   * @param range
+   * @return list of fileRange
+   */
+  static List<FileRange> getFileRangeFrom(BufferChunkList range) {
+    List<FileRange> fRange = new ArrayList<>();
+    BufferChunk cr = range.get();
+    while (cr.next != null) {
+      long len = cr.getLength();
+      long off = cr.getOffset();
+      FileRange currentRange = FileRange.createFileRange(off, (int) len);
+      fRange.add(currentRange);
+      cr = (BufferChunk) cr.next;
+    }
+    return fRange;
+  }
+
+  /**
+   * Read the list of ranges from the file.
+   * @param range the disk ranges within the stripe to read
+   * @return the bytes read for each disk range, which is the same length as
+   *    ranges
+   * @throws IOException
+   */
+  static void readDiskRangesVectored(FSDataInputStream fileInputStream,
+      BufferChunkList range, boolean doForceDirect) throws IOException {
+    if (range == null)
+      return;
+    BufferChunkList rootRange = range;
+    //Convert DiskRange to FileRange here
+    List<FileRange> fRanges = getFileRangeFrom(range);
+
+    try {
+      IntFunction<ByteBuffer> allocate = doForceDirect ? ByteBuffer::allocateDirect : ByteBuffer::allocate;
+
+      fileInputStream.readVectored(fRanges, allocate);
+
+      int index = 0;
+      range = rootRange;
+      BufferChunk current = range.get();
+      while (current != null) {
+        if (current.hasData()) {
+          current = (BufferChunk) current.next;
+          index ++;
+        }
+        ByteBuffer data = fRanges.get(index).getData().get();
+        current.setChunk(data);
+        //replace the range data with the buffer chunk returned after reading
+        current = (BufferChunk) current.next;
+      }
+    } catch (InterruptedException e) {
+      Thread.currentThread().interrupt();
+      throw new IOException(e);
+    } catch (ExecutionException e) {

Review Comment:
   org.apache.hadoop.util.functional.FutureIO has some methods to help extract useful IOEs from ExecutionExceptions -see unwrapInnerException()



##########
java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java:
##########
@@ -488,6 +504,65 @@ private static BufferChunk findSingleRead(BufferChunk first, long minSeekSize) {
     return last;
   }
 
+  /**
+   * Convert from DiskRangeList to FileRange
+   * @param range
+   * @return list of fileRange
+   */
+  static List<FileRange> getFileRangeFrom(BufferChunkList range) {
+    List<FileRange> fRange = new ArrayList<>();
+    BufferChunk cr = range.get();
+    while (cr.next != null) {
+      long len = cr.getLength();
+      long off = cr.getOffset();
+      FileRange currentRange = FileRange.createFileRange(off, (int) len);
+      fRange.add(currentRange);
+      cr = (BufferChunk) cr.next;
+    }
+    return fRange;
+  }
+
+  /**
+   * Read the list of ranges from the file.
+   * @param range the disk ranges within the stripe to read
+   * @return the bytes read for each disk range, which is the same length as
+   *    ranges
+   * @throws IOException
+   */
+  static void readDiskRangesVectored(FSDataInputStream fileInputStream,
+      BufferChunkList range, boolean doForceDirect) throws IOException {
+    if (range == null)
+      return;
+    BufferChunkList rootRange = range;
+    //Convert DiskRange to FileRange here
+    List<FileRange> fRanges = getFileRangeFrom(range);
+
+    try {
+      IntFunction<ByteBuffer> allocate = doForceDirect ? ByteBuffer::allocateDirect : ByteBuffer::allocate;
+
+      fileInputStream.readVectored(fRanges, allocate);
+
+      int index = 0;
+      range = rootRange;
+      BufferChunk current = range.get();
+      while (current != null) {

Review Comment:
   because FileRange has a reference field, you can put a reference back to the specific BufferChunk instances for each range. Then waiting for results becomes a matter of
   1. waiting for the ranged reads to complete
   2. resolving the references for each range
   3. set their Chunk bytebuffers to the ranges read.



##########
java/pom.xml:
##########
@@ -70,8 +70,8 @@
     <test.tmp.dir>${project.build.directory}/testing-tmp</test.tmp.dir>
     <example.dir>${project.basedir}/../../examples</example.dir>
 
-    <min.hadoop.version>2.7.3</min.hadoop.version>
-    <hadoop.version>2.7.3</hadoop.version>
+    <min.hadoop.version>3.3.9-SNAPSHOT</min.hadoop.version>

Review Comment:
   set to 3.3.5



##########
java/shims/pom.xml:
##########
@@ -59,6 +59,12 @@
       <artifactId>junit-jupiter-api</artifactId>
       <scope>test</scope>
     </dependency>
+      <dependency>
+          <groupId>org.apache.hadoop</groupId>
+          <artifactId>hadoop-hdfs-client</artifactId>
+          <version>3.3.9-SNAPSHOT</version>

Review Comment:
   is this really needed? If it is it's a failure in the api. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] steveloughran commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1614462653

   slides from berlin buzzword last week
   https://docs.google.com/presentation/d/1U5QRN4etbM7gkbnGO3OW4sCfUZx9LqJN/
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] mukund-thakur commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1282895312

   Hi @dongjoon-hyun 
   Thanks for the review. Yeah, I understand we can't upgrade the Hadoop version directly just like this. This PR is to show how the integration can be done with the upcoming Vectored IO API  https://issues.apache.org/jira/browse/HADOOP-18103 in 
   upcoming Hadoop 3.3.5 release and discuss the best approach to get the changes committed. 
   
   Hi @omalley ,
   This is the orc patch I was talking about. Please take a look whenever you get time. Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
URL: https://github.com/apache/orc/pull/1276


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1553951061

   FYI, Apache ORC 2.0 Milestone is created and aims to use Hadoop 3 dependency to access all Hadoop 2 and 3 clusters.
   - https://github.com/apache/orc/milestone/20


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1283271045

   Got it. According to the request, I'll leave this PR to @omalley .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1869347714

   Thank you, @williamhyun !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "williamhyun (via GitHub)" <gi...@apache.org>.
williamhyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1869177276

   I made a PR, please take a look. 
   https://github.com/apache/orc/pull/1708
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1868626935

   As of today, this item is removed from Apache ORC 2.0.0 milestone.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1837284413

   Gentle ping.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org