You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "nastra (via GitHub)" <gi...@apache.org> on 2026/06/19 13:09:31 UTC
[PR] GH-3558: Properly close buffers for vectored IO [parquet-java]
nastra opened a new pull request, #3620:
URL: https://github.com/apache/parquet-java/pull/3620
<!--
Thanks for opening a pull request!
If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing
Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose
and format pull request title as below:
GH-${GITHUB_ISSUE_ID}: ${SUMMARY}
or simply use the title below if it is a minor issue:
MINOR: ${SUMMARY}
-->
### Rationale for this change
This closes allocated buffers for vectorized IO. Related issues are https://github.com/apache/parquet-java/issues/3356 and https://github.com/apache/parquet-java/issues/3358.
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3558: Bump hadoop to 3.4.3 and properly close buffers for vectored IO [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3620:
URL: https://github.com/apache/parquet-java/pull/3620#discussion_r3448898610
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1361,8 +1362,32 @@ private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder b
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
+ // Wrap the allocator to track buffers allocated during the vectored read.
+ // Hadoop's vectored IO may merge ranges and return slices of merged buffers,
+ // so the buffers returned via CompletableFuture may differ from those originally
+ // allocated. We track the originals here to ensure they are properly released.
+ ByteBufferAllocator baseAllocator = options.getAllocator();
+ List<ByteBuffer> allocatedBuffers = new ArrayList<>();
+ ByteBufferAllocator trackingAllocator = new ByteBufferAllocator() {
Review Comment:
I don't like adding a wrapper here just to fix something in Hadoop. @steveloughran suggested to set the config to avoid checksums on the Hadoop side: https://github.com/apache/parquet-java/issues/3356#issuecomment-4089172411 This doesn't add any value to Parquet anyway. This is more or less what I suggested in https://github.com/apache/parquet-java/pull/3559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3558: Bump hadoop to 3.4.3 and properly close buffers for vectored IO [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on PR #3620:
URL: https://github.com/apache/parquet-java/pull/3620#issuecomment-4759873248
Wondering if my approach for the same issue in https://github.com/apache/parquet-java/pull/3579/changes#diff-8da24c84aef62e6e836d073938f7843d289785baaeddf446f3afeae6d4ef4b10R1368
is slightly more robust. Also for reference in my change when I free the resources it really does not apply them because it is up to Hadoop to do it :(
https://github.com/apache/hadoop/pull/8511
If anyone here knows someone who can help us get this reviewed/merged on the Hadoop side that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3558: Bump hadoop to 3.4.3 and properly close buffers for vectored IO [parquet-java]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #3620:
URL: https://github.com/apache/parquet-java/pull/3620#discussion_r3451650910
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1361,8 +1362,32 @@ private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder b
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
+ // Wrap the allocator to track buffers allocated during the vectored read.
+ // Hadoop's vectored IO may merge ranges and return slices of merged buffers,
+ // so the buffers returned via CompletableFuture may differ from those originally
+ // allocated. We track the originals here to ensure they are properly released.
+ ByteBufferAllocator baseAllocator = options.getAllocator();
+ List<ByteBuffer> allocatedBuffers = new ArrayList<>();
+ ByteBufferAllocator trackingAllocator = new ByteBufferAllocator() {
Review Comment:
makes sense, let's go with your PR then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3558: Bump hadoop to 3.4.3 and properly close buffers for vectored IO [parquet-java]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #3620:
URL: https://github.com/apache/parquet-java/pull/3620#issuecomment-4752033914
/cc @Fokko @wgtmac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3558: Bump hadoop to 3.4.3 and properly close buffers for vectored IO [parquet-java]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed pull request #3620: GH-3558: Bump hadoop to 3.4.3 and properly close buffers for vectored IO
URL: https://github.com/apache/parquet-java/pull/3620
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org