You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "annimesh2809 (via GitHub)" <gi...@apache.org> on 2025/11/13 17:24:11 UTC
[I] TestParquetReader fails with hadoop 3.4.2 [parquet-java]
annimesh2809 opened a new issue, #3356:
URL: https://github.com/apache/parquet-java/issues/3356
### Describe the bug, including details regarding any error messages, version, and platform.
A couple of test cases of `TestParquetReader` suite started failing with errors like:
```
- testRangeFiltering[0] *** FAILED ***
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException: 24 ByteBuffer object(s) is/are remained unreleased after closing this allocator.
at org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:161)
at org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
```
with hadoop 3.4.2.
The leaks are happening when reading using vectored IO because we never pass the buffers to the releaser.
### Component(s)
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] TestParquetReader fails with hadoop 3.4.2 [parquet-java]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #3356:
URL: https://github.com/apache/parquet-java/issues/3356#issuecomment-4444412710
I actually do want to sit down with claude and get the fundamental issue fixed, which is ChecksumFileSystem.readVectored
for 3.4.2+ you can actually turn off checksum validation here
```
fs.file.checksum.verify = false
```
see https://issues.apache.org/jira/browse/HADOOP-18296
This is tangibly faster for vector iO, as all requests should be mapped to jni scatter gather. There's no checksumming for any file:// reads, but if you trust your SSD you are safe, at least for unit tests.
we do this for our tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] TestParquetReader fails with hadoop 3.4.2 [parquet-java]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #3356:
URL: https://github.com/apache/parquet-java/issues/3356#issuecomment-4097143871
Let me fix this and remind me if I don't submit a PR: there's an internal PR we have which addresses this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] TestParquetReader fails with hadoop 3.4.2 [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #3356:
URL: https://github.com/apache/parquet-java/issues/3356#issuecomment-4437390762
@steveloughran I noticed this issue after creating the PR, WDYT of https://github.com/apache/parquet-java/pull/3559/?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] TestParquetReader fails with hadoop 3.4.2 [parquet-java]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #3356:
URL: https://github.com/apache/parquet-java/issues/3356#issuecomment-4089172411
problem with the vector IO buffer merging and the checksum blocks of local fs.
See HADOOP-18296: setting `fs.file.checksum.verify` to false will stop this by handing down to the raw local code. (bonus benefit: vector reads handoff to java nio scatter/gather and you get maximum perfomance).
1. parquet needs a PR for this
2. someone who can understand the relevant bits of `ChecksumFileSystem` could see if it could be clever here and return the original buffers
This doesn't actually surface in production because the ranges of parquet/ORC files is too big, though I think we've also been considering removing range merging from the s3a code. The issue there is correctly handling partial failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org