You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by GitBox <gi...@apache.org> on 2022/10/12 11:15:36 UTC
[GitHub] [orc] HarshitGupta11 opened a new pull request, #1276: ORC-1251:Support Vectored IO in ORC [Draft]
HarshitGupta11 opened a new pull request, #1276:
URL: https://github.com/apache/orc/pull/1276
Support the new fast Vectored Read API in ORC for parallel reads.
The patch was tested internally along with S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "williamhyun (via GitHub)" <gi...@apache.org>.
williamhyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1869172553
Let me take over this for ORC 2.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] steveloughran commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1554918583
@HarshitGupta11 can you rebase this to orc main and against release hadoop 3.3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1848783089
Let me close this PR since this commit has been stale over one year already.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1868852105
@mukund-thakur @HarshitGupta11 Do you have any plan to revive this work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] jerqi commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1614033863
> Support the new fast Vectored Read API in ORC for parallel reads. The patch was tested internally along with S3
Could you share some performance benchmark tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] steveloughran commented on a diff in pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #1276:
URL: https://github.com/apache/orc/pull/1276#discussion_r1199185617
##########
java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java:
##########
@@ -488,6 +504,65 @@ private static BufferChunk findSingleRead(BufferChunk first, long minSeekSize) {
return last;
}
+ /**
+ * Convert from DiskRangeList to FileRange
+ * @param range
+ * @return list of fileRange
+ */
+ static List<FileRange> getFileRangeFrom(BufferChunkList range) {
+ List<FileRange> fRange = new ArrayList<>();
+ BufferChunk cr = range.get();
+ while (cr.next != null) {
+ long len = cr.getLength();
+ long off = cr.getOffset();
+ FileRange currentRange = FileRange.createFileRange(off, (int) len);
+ fRange.add(currentRange);
+ cr = (BufferChunk) cr.next;
+ }
+ return fRange;
+ }
+
+ /**
+ * Read the list of ranges from the file.
+ * @param range the disk ranges within the stripe to read
+ * @return the bytes read for each disk range, which is the same length as
+ * ranges
+ * @throws IOException
+ */
+ static void readDiskRangesVectored(FSDataInputStream fileInputStream,
+ BufferChunkList range, boolean doForceDirect) throws IOException {
+ if (range == null)
+ return;
+ BufferChunkList rootRange = range;
+ //Convert DiskRange to FileRange here
+ List<FileRange> fRanges = getFileRangeFrom(range);
+
+ try {
+ IntFunction<ByteBuffer> allocate = doForceDirect ? ByteBuffer::allocateDirect : ByteBuffer::allocate;
+
+ fileInputStream.readVectored(fRanges, allocate);
+
+ int index = 0;
+ range = rootRange;
+ BufferChunk current = range.get();
+ while (current != null) {
+ if (current.hasData()) {
+ current = (BufferChunk) current.next;
+ index ++;
+ }
+ ByteBuffer data = fRanges.get(index).getData().get();
+ current.setChunk(data);
+ //replace the range data with the buffer chunk returned after reading
+ current = (BufferChunk) current.next;
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IOException(e);
+ } catch (ExecutionException e) {
Review Comment:
org.apache.hadoop.util.functional.FutureIO has some methods to help extract useful IOEs from ExecutionExceptions -see unwrapInnerException()
##########
java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java:
##########
@@ -488,6 +504,65 @@ private static BufferChunk findSingleRead(BufferChunk first, long minSeekSize) {
return last;
}
+ /**
+ * Convert from DiskRangeList to FileRange
+ * @param range
+ * @return list of fileRange
+ */
+ static List<FileRange> getFileRangeFrom(BufferChunkList range) {
+ List<FileRange> fRange = new ArrayList<>();
+ BufferChunk cr = range.get();
+ while (cr.next != null) {
+ long len = cr.getLength();
+ long off = cr.getOffset();
+ FileRange currentRange = FileRange.createFileRange(off, (int) len);
+ fRange.add(currentRange);
+ cr = (BufferChunk) cr.next;
+ }
+ return fRange;
+ }
+
+ /**
+ * Read the list of ranges from the file.
+ * @param range the disk ranges within the stripe to read
+ * @return the bytes read for each disk range, which is the same length as
+ * ranges
+ * @throws IOException
+ */
+ static void readDiskRangesVectored(FSDataInputStream fileInputStream,
+ BufferChunkList range, boolean doForceDirect) throws IOException {
+ if (range == null)
+ return;
+ BufferChunkList rootRange = range;
+ //Convert DiskRange to FileRange here
+ List<FileRange> fRanges = getFileRangeFrom(range);
+
+ try {
+ IntFunction<ByteBuffer> allocate = doForceDirect ? ByteBuffer::allocateDirect : ByteBuffer::allocate;
+
+ fileInputStream.readVectored(fRanges, allocate);
+
+ int index = 0;
+ range = rootRange;
+ BufferChunk current = range.get();
+ while (current != null) {
Review Comment:
because FileRange has a reference field, you can put a reference back to the specific BufferChunk instances for each range. Then waiting for results becomes a matter of
1. waiting for the ranged reads to complete
2. resolving the references for each range
3. set their Chunk bytebuffers to the ranges read.
##########
java/pom.xml:
##########
@@ -70,8 +70,8 @@
<test.tmp.dir>${project.build.directory}/testing-tmp</test.tmp.dir>
<example.dir>${project.basedir}/../../examples</example.dir>
- <min.hadoop.version>2.7.3</min.hadoop.version>
- <hadoop.version>2.7.3</hadoop.version>
+ <min.hadoop.version>3.3.9-SNAPSHOT</min.hadoop.version>
Review Comment:
set to 3.3.5
##########
java/shims/pom.xml:
##########
@@ -59,6 +59,12 @@
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs-client</artifactId>
+ <version>3.3.9-SNAPSHOT</version>
Review Comment:
is this really needed? If it is it's a failure in the api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] steveloughran commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1614462653
slides from berlin buzzword last week
https://docs.google.com/presentation/d/1U5QRN4etbM7gkbnGO3OW4sCfUZx9LqJN/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] mukund-thakur commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1282895312
Hi @dongjoon-hyun
Thanks for the review. Yeah, I understand we can't upgrade the Hadoop version directly just like this. This PR is to show how the integration can be done with the upcoming Vectored IO API https://issues.apache.org/jira/browse/HADOOP-18103 in
upcoming Hadoop 3.3.5 release and discuss the best approach to get the changes committed.
Hi @omalley ,
This is the orc patch I was talking about. Please take a look whenever you get time. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
URL: https://github.com/apache/orc/pull/1276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1553951061
FYI, Apache ORC 2.0 Milestone is created and aims to use Hadoop 3 dependency to access all Hadoop 2 and 3 clusters.
- https://github.com/apache/orc/milestone/20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #1276: ORC-1251:Support Vectored IO in ORC [Draft]
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1283271045
Got it. According to the request, I'll leave this PR to @omalley .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1869347714
Thank you, @williamhyun !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "williamhyun (via GitHub)" <gi...@apache.org>.
williamhyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1869177276
I made a PR, please take a look.
https://github.com/apache/orc/pull/1708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1868626935
As of today, this item is removed from Apache ORC 2.0.0 milestone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1251:Support Vectored IO in ORC [Draft] [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1276:
URL: https://github.com/apache/orc/pull/1276#issuecomment-1837284413
Gentle ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org