You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/08/16 02:43:56 UTC
[PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
sunchao opened a new pull request, #3726:
URL: https://github.com/apache/parquet-java/pull/3726
## Why are the changes needed?
Apache Parquet's Hadoop vectored-read path can fall back to ordinary reads after
asynchronous filesystem requests have already been submitted or partially
consumed. Retrying against the same partially populated chunk builder can
duplicate selected page data and return incorrect filtered rows. Outstanding
sibling requests can also continue using a stream while fallback or cleanup
begins.
The same path requests one buffer for an entire contiguous projected range
instead of splitting requests at `parquet.read.allocation.size`, which can cause
unexpectedly large allocations.
Vectored I/O is enabled by default in Apache Parquet 1.16 and later, so these
issues are not limited to applications that explicitly opt in.
## What changes were proposed in this PR?
- Split contiguous requested filesystem ranges at the configured allocation
limit while preserving the existing logical page and column plan.
- Keep ordinary fallback for failures detected while preparing vectored ranges;
convert failures after the vectored submission call is attempted into
`IOException` rather than replaying reads against a partially populated
builder.
- Await already-published sibling read futures within the existing shared
timeout before propagating an asynchronous failure.
- Verify dictionary pages and V1/V2 data-page checksums incrementally across
split buffers instead of allocating another oversized contiguous buffer.
- Add focused vectored-reader and checksum regression coverage.
The current patch still has four known review points that need follow-up:
- Filesystems that align ranges for checksums can allocate more than the
requested range size, so splitting requests alone does not guarantee that
every backend allocation respects the configured cap.
- A backend that rejects vectored I/O before submitting work should retain safe
ordinary-read fallback; the current submission boundary cannot distinguish
that case from a partially submitted failure.
- Filesystems such as S3A can block while synchronously submitting more ranges
than their in-flight limit permits, before the existing read timeout applies.
- A backend can start reads and then fail before publishing their futures to
the wrapper, leaving those reads invisible to the current cleanup path.
## How was this PR tested?
Previously recorded results for this commit on Java 17:
```bash
mvn -pl parquet-hadoop \
-Dtest=TestParquetFileReaderVectoredIO,TestDataPageChecksums \
-Dsurefire.failIfNoSpecifiedTests=false \
test
```
The focused suites reported **37 passing tests**, including **19 vectored-reader
tests** and **18 checksum tests**.
```bash
mvn -pl parquet-hadoop test
mvn -pl parquet-hadoop spotless:check
```
The complete module reported **748 tests, zero failures, zero errors, and 24
existing Hadoop-capability skips**. Apache RAT approved all **256** scanned
licenses, and Spotless passed. The four review points above are not covered by
those recorded passing tests.
Closes #3719.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "dossett (via GitHub)" <gi...@apache.org>.
dossett commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3797672691
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous failure can occur after some reads have been scheduled,
+ // so falling back to normal IO is unsafe once this call has been entered.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
Review Comment:
An `UnsupportedOperationException` will now be rethrown as an `IOExeption`. If I understand correctly, on `master` this will now kill the read entirely instead of resulting in a warning and a fallback to non-vectored IO. If so, that seems like a meaningful change.
Maybe I'm missing something, or maybe this just wouldn't happen in practice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#issuecomment-6312046742
I just merged it. Thanks everyone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4089442748
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1358,32 +1355,150 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit bounds requested range lengths. Filesystems may merge or
+ * checksum-align ranges and allocate larger buffers; decoders may also require
+ * larger contiguous buffers. This is not a total memory limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ if (vectoredReadExecutor == null) {
+ vectoredReadExecutor = VectoredReadOperation.newExecutor();
Review Comment:
A blocked backend call still uses one worker per reader with no global bound. Is this acceptable for workloads with many concurrent readers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3800579324
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous failure can occur after some reads have been scheduled,
+ // so falling back to normal IO is unsafe once this call has been entered.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
Review Comment:
Yes, this is a deliberate and meaningful behavior change. Even a synchronous `UnsupportedOperationException` can follow partial submission, and the Hadoop bridge only copies the futures back to Parquet after submission returns. During result consumption, the chunk builder may also already contain data. Falling back in either situation can race outstanding reads or replay chunks and return incorrect rows.
Ordinary I/O still handles disabled/unavailable vectored I/O, and preparation-time `IllegalArgumentException`/`UnsupportedOperationException` still permits fallback. I clarified that boundary in [ad623d6](https://github.com/apache/parquet-java/pull/3726/commits/ad623d6f4a44934ee2ccb98ab28bfee7242b0885) and corrected the availability documentation: the Hadoop bridge checks runtime API/allocator support, not per-stream `hasCapability`. The tests cover ordinary-read selection plus synchronous, asynchronous, and partial-submission failures without fallback.
I do not have an observed production rejection from an otherwise supported backend. A genuinely pre-submission rejection should be recoverable, but the current interface does not reliably distinguish it from partial submission. Restoring broader fallback would need an explicit no-work-submitted guarantee; I have kept the conservative behavior for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3959123314
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1364,153 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ if (vectoredReadExecutor == null) {
+ vectoredReadExecutor = Executors.newSingleThreadExecutor(task -> {
Review Comment:
Each reader keeps one worker thread after its first vectored read. Could this be expensive when many readers are open?
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous failure can occur after some reads have been scheduled,
+ // so falling back to normal IO is unsafe once this call has been entered.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
Review Comment:
I think fail-fast is safer here. We cannot reliably tell a rejection before submission from a partial submission. This may change behavior for some backends, but I am okay with this trade-off.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1364,153 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
Review Comment:
Hadoop checksum reads can merge ranges and allocate larger buffers than `maximumAllocation`. Is the allocation limit only best effort for such filesystems?
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1951,10 +2096,12 @@ protected PageHeader readPageHeader(BlockCipher.Decryptor blockDecryptor, byte[]
* Calculate checksum of input bytes, throw decoding exception if it does not match the provided
* reference crc
*/
- private void verifyCrc(int referenceCrc, BytesInput bytes, String exceptionMsg) {
+ private void verifyCrc(int referenceCrc, String exceptionMsg, BytesInput... inputs) throws IOException {
crc.reset();
- try (ByteBufferReleaser releaser = crcAllocator.getReleaser()) {
- crc.update(bytes.toByteBuffer(releaser));
+ for (BytesInput input : inputs) {
+ for (ByteBuffer buffer : input.toInputStream().remainingBuffers()) {
Review Comment:
Is `crcAllocator` still needed? I only find its initialization and close, but no use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#issuecomment-6324015504
Thanks for the review! @wgtmac @gszadovszky @divjotarora @dossett !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4318390344
##########
parquet-hadoop/README.md:
##########
@@ -558,6 +558,26 @@ if not found then the library will use the classic non-vectored reads: it is saf
---
+**JVM system property:** `parquet.hadoop.vectored.io.threads`
+
+**Description:** Maximum number of shared vectored-read workers. Set a positive integer
Review Comment:
Agreed — added this in effbc38496207ddcb4c9a0b3204c4ff8c9d1e023.
Callers can now wrap an application-owned pool with a shared admission limit:
```java
VectoredReadExecutor reads = new VectoredReadExecutor(engineIoPool, 8);
ParquetReadOptions options = ParquetReadOptions.builder()
.withVectoredReadExecutor(reads)
.build();
```
The option is also available through `HadoopReadOptions` and `ParquetReader.Builder`; copied options retain the same wrapper. Leaving it unset keeps the existing JVM shared pool as the default. Parquet never shuts down the supplied pool, and accepted operations retain capacity until submission, the ownership decision, and failure cleanup finish.
The supplied executor must dispatch asynchronously, accept or reject rather than silently drop work, and drain accepted tasks during shutdown. Inline execution (including saturated `CallerRunsPolicy`) is rejected before filesystem submission, leaving the reader usable for retry.
Added regressions for both options builders and copies, reader closure and error cleanup without pool shutdown, inline rejection, permit recovery, and retrying the same reader. Validation passed: 877 module tests (26 Hadoop-capability skips), 17 integration tests, and 118 focused Hadoop 3.4.1 tests with no skips, plus API compatibility, dependency analysis, Spotless, and RAT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4178558095
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1358,32 +1355,150 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit bounds requested range lengths. Filesystems may merge or
+ * checksum-align ranges and allocate larger buffers; decoders may also require
+ * larger contiguous buffers. This is not a total memory limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ if (vectoredReadExecutor == null) {
+ vectoredReadExecutor = VectoredReadOperation.newExecutor();
Review Comment:
Agreed: idle expiry did not bound workers held by blocked submissions, and retries could accumulate them. Fixed in [8ea4302](https://github.com/apache/parquet-java/pull/3726/commits/8ea4302a61254de697f66ef20e5edc75729bbf69) with a shared bounded executor and deadline-aware admission. The default is 64 workers, configurable before first use through the JVM system property `parquet.hadoop.vectored.io.threads`.
A worker retains capacity until the backend call actually exits, the caller transfers or abandons buffer ownership, and failure cleanup finishes. Cleanup runs on that same worker after submission, so cancellation cannot release capacity early or let cleanup overtake the backend. Admission shares the existing 300-second deadline; if capacity is unavailable, the read fails before submission and its stream remains reader-owned. Readers no longer shut down the shared executor.
Tests cover two blocked submissions followed by repeated admission timeouts without extra backend calls/workers, recovery after real completion, blocked stream close, interruption before admission with reader reuse, and abort before the worker starts. The 110-test Hadoop 3.4.1 run also passed with the production shared pool capped at two workers; full default-Hadoop module passed 869 tests with 26 capability skips. The README documents saturation behavior and the limits: stuck calls can consume capacity indefinitely, and filesystem-owned threads/total memory are outside this cap. No live S3 or throughput validation was performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#issuecomment-5332346914
cc @wgtmac can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3887225406
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
+ }
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous rejection can follow partial submission. The Hadoop bridge
+ // publishes futures only after submission returns, so missing futures do not prove
+ // that no reads started. Once this call is entered, normal-read fallback is unsafe.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
+ // Consumption may also have populated the builder. Do not replay those chunks.
+ IOException failure =
+ new IOException("Vectored read failed after asynchronous reads may have been submitted", e);
+ awaitRemainingVectoredReads(ranges, readStart, failure);
+ throw failure;
+ } catch (IOException | RuntimeException e) {
+ awaitRemainingVectoredReads(ranges, readStart, e);
+ throw e;
+ }
+ }
+
+ /**
+ * Wait for submitted reads with published futures to finish before their stream can be
+ * closed. Cancelling result futures does not stop all Hadoop backends from continuing IO.
+ */
+ private void awaitRemainingVectoredReads(List<ParquetFileRange> ranges, long readStart, Throwable failure) {
+ if (Thread.currentThread().isInterrupted()
+ || failure instanceof InterruptedIOException && failure.getCause() instanceof InterruptedException) {
Review Comment:
Adjusted in [f82eaa42](https://github.com/apache/parquet-java/pull/3726/commits/f82eaa428528849cf019c99410fe879c4d641875). An interrupted wait still propagates promptly as `InterruptedIOException`; it does not block draining siblings. The failed reader is invalidated and its original stream is handed to deferred cleanup, which cannot overtake a submission call still running after cancellation.
Cleanup may close a stream with asynchronous reads still pending. Close or cancellation is not treated as proof that I/O stopped: buffer release requires every range to have a published, completed, non-cancelled future. Unknown completion retains the batch's allocations.
Added tests for a submission that ignores interruption and for a pending read that completes after stream closure, checking that its buffer is not released early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "dossett (via GitHub)" <gi...@apache.org>.
dossett commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3804681928
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous failure can occur after some reads have been scheduled,
+ // so falling back to normal IO is unsafe once this call has been entered.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
Review Comment:
Thank you for the thoughtful reply. I'll defer to others more knowledgeable than myself about the tradeoffs here, but I've learned a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3800577091
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
Review Comment:
Thanks, addressed in [ad623d6](https://github.com/apache/parquet-java/pull/3726/commits/ad623d6f4a44934ee2ccb98ab28bfee7242b0885). `HadoopInputFile` already reads its cached `FileStatus`, but other `InputFile` implementations need not do so. I now cache the length lazily for vectored-range validation, so it adds at most one length lookup per reader instead of one per row-group read. Ordinary reads, including those with a supplied footer, do not acquire an extra lookup.
Added regression tests for repeated filtered/unfiltered reads and both footer paths. All 40 focused tests pass; the full `parquet-hadoop` module reports 751 tests, zero failures or errors, and 24 Hadoop-capability skips. Spotless and RAT also pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4102364760
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1358,32 +1355,150 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit bounds requested range lengths. Filesystems may merge or
+ * checksum-align ranges and allocate larger buffers; decoders may also require
+ * larger contiguous buffers. This is not a total memory limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ if (vectoredReadExecutor == null) {
+ vectoredReadExecutor = VectoredReadOperation.newExecutor();
+ }
+ VectoredReadOperation operation = new VectoredReadOperation(
+ f,
+ ranges,
+ options.getAllocator(),
+ vectoredReadExecutor,
+ HADOOP_VECTORED_READ_TIMEOUT_SECONDS,
+ TimeUnit.SECONDS);
+ try {
+ // Even a synchronous rejection can follow partial submission. The Hadoop bridge
+ // may expose futures for reads which were never scheduled. Once submission is
+ // attempted, neither missing futures nor a synchronous error permit replay.
+ operation.awaitSubmission();
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex)
+ .readFromVectoredRanges(ranges.subList(firstRange, endRange), builder, operation);
+ firstRange = endRange;
+ }
+ operation.transferTo(builder.releaser);
+ } catch (TimeoutException e) {
+ IOException failure = new IOException("Timed out submitting vectored reads", e);
+ abortVectoredRead(operation, failure);
+ throw failure;
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
+ // Consumption may also have populated the builder. Do not replay those chunks.
+ IOException failure =
+ new IOException("Vectored read failed after asynchronous reads may have been submitted", e);
+ if (operation.submissionSucceeded()) {
+ awaitRemainingVectoredReads(ranges, operation, failure);
+ }
+ abortVectoredRead(operation, failure);
Review Comment:
`awaitRemainingVectoredReads` calls `FutureIO.awaitFuture`, which can throw `Error` ([link](https://github.com/apache/parquet-java/blob/5d3d6f3eb37f48cb46f28d4f040b2dc2ecfb6828/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/wrapped/io/FutureIO.java#L131)). The `Error` is not explicitly handled by `awaitRemainingVectoredReads`, so in that case, `abortVectoredRead` never gets called. Is this desirable? If not, should we use `try/finally` to guarantee that `abortVectoredRead` always runs? Same question for the `catch` block below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3887224876
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -2343,32 +2396,39 @@ private void setReadMetrics(long startNs, long len) {
}
/**
- * Populate data in a parquet file range from a vectored range; will block for up
- * to {@link #HADOOP_VECTORED_READ_TIMEOUT_SECONDS} seconds.
- * @param currRange range to populated.
+ * Populate data in a parquet file range from one or more bounded vectored ranges; together
+ * they may block for up to {@link #HADOOP_VECTORED_READ_TIMEOUT_SECONDS} seconds.
+ * @param ranges bounded ranges containing this part.
* @param builder used to build chunk list to read the pages for the different columns.
* @throws IOException if there is an error while reading from the stream, including a timeout.
*/
- public void readFromVectoredRange(ParquetFileRange currRange, ChunkListBuilder builder) throws IOException {
- ByteBuffer buffer;
+ public void readFromVectoredRanges(List<ParquetFileRange> ranges, ChunkListBuilder builder) throws IOException {
+ List<ByteBuffer> buffers = new ArrayList<>(ranges.size());
+ ParquetFileRange currentRange = null;
final long timeoutSeconds = HADOOP_VECTORED_READ_TIMEOUT_SECONDS;
+ final long timeoutNanos = TimeUnit.SECONDS.toNanos(timeoutSeconds);
long readStart = System.nanoTime();
try {
- LOG.debug(
- "Waiting for vectored read to finish for range {} with timeout {} seconds",
- currRange,
- timeoutSeconds);
- buffer = FutureIO.awaitFuture(currRange.getDataReadFuture(), timeoutSeconds, TimeUnit.SECONDS);
- setReadMetrics(readStart, currRange.getLength());
+ for (ParquetFileRange range : ranges) {
+ currentRange = range;
+ LOG.debug(
+ "Waiting for vectored read to finish for range {} with timeout {} seconds",
+ range,
+ timeoutSeconds);
+ long remainingNanos = Math.max(timeoutNanos - (System.nanoTime() - readStart), 0L);
+ buffers.add(FutureIO.awaitFuture(range.getDataReadFuture(), remainingNanos, TimeUnit.NANOSECONDS));
Review Comment:
Fixed in [f82eaa42](https://github.com/apache/parquet-java/pull/3726/commits/f82eaa428528849cf019c99410fe879c4d641875). Vectored reads now track the original allocations and transfer them to the row group's releaser after successful completion. This includes extra checksum buffers and buffers whose returned results are slices, so the allocator receives the original objects it created. The row group is also closed on read/parsing failures.
Added ownership tests for filtered and unfiltered reads, slices, checksum allocations, and parse failures. Real Hadoop 3.4.1 raw/checksum filesystem probes decoded all 128 rows with balanced allocations/releases and a clean tracking allocator.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
+ }
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous rejection can follow partial submission. The Hadoop bridge
+ // publishes futures only after submission returns, so missing futures do not prove
+ // that no reads started. Once this call is entered, normal-read fallback is unsafe.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
+ // Consumption may also have populated the builder. Do not replay those chunks.
+ IOException failure =
+ new IOException("Vectored read failed after asynchronous reads may have been submitted", e);
+ awaitRemainingVectoredReads(ranges, readStart, failure);
Review Comment:
Addressed in [f82eaa42](https://github.com/apache/parquet-java/pull/3726/commits/f82eaa428528849cf019c99410fe879c4d641875). The bridge now publishes available futures in a `finally` block, without masking the original submission failure. A failed submission still fails the read without replaying ordinary I/O.
Stream closure is queued behind the actual submission call. Buffer release requires every range to have a published, completed, non-cancelled future. Hadoop can publish futures for work it never schedules, so cleanup does not block indefinitely waiting for those; the batch retains its remaining allocations if completion cannot be established.
Added regressions for partial submission and a future-publication failure that must not replace the submission exception.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
+ }
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
Review Comment:
Fixed in [f82eaa42](https://github.com/apache/parquet-java/pull/3726/commits/f82eaa428528849cf019c99410fe879c4d641875). Submission now runs on a reader-owned worker, and one 300-second budget per vectored-read batch starts before submission and is shared with every range wait. It is no longer reset for each part.
On timeout, the reader is invalidated and cleanup is queued on the same worker, so cancelling the submission `Future` cannot let cleanup overtake a backend call that is still running. The caller can return even if the backend ignores interruption; its resources remain owned by deferred cleanup, which may itself be delayed indefinitely.
Added tests for blocked submission, the shared deadline, and submission that continues after cancellation. These are local regression tests; I have not validated this against live S3.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
Review Comment:
Fixed in [f82eaa42](https://github.com/apache/parquet-java/pull/3726/commits/f82eaa428528849cf019c99410fe879c4d641875). The length lookup now has a narrow pre-submission fallback: a metadata `IOException` disables vectored I/O for that reader and continues with ordinary reads on the already-open stream. This avoids repeated failed lookups on later row groups. Interruption still propagates, and failures after submission still cannot trigger ordinary-read replay.
Added regression tests with and without a supplied footer, plus an interrupted-lookup case. Both fallback regressions fail on the original commit and pass with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#issuecomment-5810073474
@wgtmac I'd be interested in reviewing, but I'm a bit swamped right now. Can get to it tomorrow if that's OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#issuecomment-5809265707
Do you want to take a look as well? @gszadovszky @Fokko @divjotarora
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4084776553
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1364,153 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
Review Comment:
Yes: the strict bound is on the length of each range Parquet requests, not on every buffer the filesystem allocates. Checksum alignment and range merging can produce larger allocations. This is now explicit in the configuration README, `ParquetReadOptions` API documentation, and reader documentation in [5d3d6f3e](https://github.com/apache/parquet-java/commit/5d3d6f3eb37f48cb46f28d4f040b2dc2ecfb6828). Footer/decoder allocations and total memory are outside this setting as well.
Strengthened the merged-buffer regression to verify that requested ranges stay within 128 bytes while the backend allocates a larger merged buffer, all rows decode correctly, and the original merged/checksum buffers are released. The PR description now states this limitation rather than implying a hard filesystem allocation cap.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1951,10 +2096,12 @@ protected PageHeader readPageHeader(BlockCipher.Decryptor blockDecryptor, byte[]
* Calculate checksum of input bytes, throw decoding exception if it does not match the provided
* reference crc
*/
- private void verifyCrc(int referenceCrc, BytesInput bytes, String exceptionMsg) {
+ private void verifyCrc(int referenceCrc, String exceptionMsg, BytesInput... inputs) throws IOException {
crc.reset();
- try (ByteBufferReleaser releaser = crcAllocator.getReleaser()) {
- crc.update(bytes.toByteBuffer(releaser));
+ for (BytesInput input : inputs) {
+ for (ByteBuffer buffer : input.toInputStream().remainingBuffers()) {
Review Comment:
Removed in [5d3d6f3e](https://github.com/apache/parquet-java/commit/5d3d6f3eb37f48cb46f28d4f040b2dc2ecfb6828), including the field, import, all constructor initialization, and close argument. Incremental checksum verification already updates CRC directly from the existing buffer views, so the scratch allocator is unnecessary.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous failure can occur after some reads have been scheduled,
+ // so falling back to normal IO is unsafe once this call has been entered.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
Review Comment:
Kept the fail-fast behavior after submission is attempted. The updated PR description now records this accepted trade-off and distinguishes it from safe pre-submission metadata/preparation fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4084775706
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1364,153 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ if (vectoredReadExecutor == null) {
+ vectoredReadExecutor = Executors.newSingleThreadExecutor(task -> {
Review Comment:
Addressed in [5d3d6f3e](https://github.com/apache/parquet-java/commit/5d3d6f3eb37f48cb46f28d4f040b2dc2ecfb6828). The executor now has zero core workers and a 60-second idle timeout, so an open reader no longer keeps a worker indefinitely between reads. A later submission recreates the worker.
It still has at most one worker per reader and a FIFO queue: failure cleanup must remain behind a submission that continues running after cancellation. An actively blocked backend call therefore still retains its worker. Added an expiry/recreation regression, and the existing timeout/cancellation cleanup tests now use the production executor factory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4178557437
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1358,32 +1355,150 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * Submission and all requested ranges share the vectored-read timeout. Failed
+ * operations invalidate the stream; cleanup waits for submission to exit before
+ * closing it, even if the backend does not respond promptly to interruption.
+ * The allocation limit bounds requested range lengths. Filesystems may merge or
+ * checksum-align ranges and allocate larger buffers; decoders may also require
+ * larger contiguous buffers. This is not a total memory limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ if (vectoredReadExecutor == null) {
+ vectoredReadExecutor = VectoredReadOperation.newExecutor();
+ }
+ VectoredReadOperation operation = new VectoredReadOperation(
+ f,
+ ranges,
+ options.getAllocator(),
+ vectoredReadExecutor,
+ HADOOP_VECTORED_READ_TIMEOUT_SECONDS,
+ TimeUnit.SECONDS);
+ try {
+ // Even a synchronous rejection can follow partial submission. The Hadoop bridge
+ // may expose futures for reads which were never scheduled. Once submission is
+ // attempted, neither missing futures nor a synchronous error permit replay.
+ operation.awaitSubmission();
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex)
+ .readFromVectoredRanges(ranges.subList(firstRange, endRange), builder, operation);
+ firstRange = endRange;
+ }
+ operation.transferTo(builder.releaser);
+ } catch (TimeoutException e) {
+ IOException failure = new IOException("Timed out submitting vectored reads", e);
+ abortVectoredRead(operation, failure);
+ throw failure;
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
+ // Consumption may also have populated the builder. Do not replay those chunks.
+ IOException failure =
+ new IOException("Vectored read failed after asynchronous reads may have been submitted", e);
+ if (operation.submissionSucceeded()) {
+ awaitRemainingVectoredReads(ranges, operation, failure);
+ }
+ abortVectoredRead(operation, failure);
Review Comment:
Fixed in [8ea4302](https://github.com/apache/parquet-java/pull/3726/commits/8ea4302a61254de697f66ef20e5edc75729bbf69). Both catch branches now run `abortVectoredRead` in `finally`, so an Error from draining a sibling cannot bypass reader invalidation and deferred cleanup. The Error still escapes, with the original read failure attached as suppressed.
Added four full-reader cases covering initial IOException, IllegalArgumentException, UnsupportedOperationException, and Error. Each forces a pending sibling to raise an Error and leaves a third read pending through stream closure; the tests verify no premature buffer release, rejected reader reuse, and complete reclamation after that last read finishes. Removing only the finally guards makes all four cases fail.
Current validation: full parquet-hadoop module 869 tests, zero failures/errors, 26 Hadoop 3.3.0 capability skips; focused Hadoop 3.4.1 run 110 tests, zero failures/errors/skips. Spotless and RAT pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "dossett (via GitHub)" <gi...@apache.org>.
dossett commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3797666452
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
Review Comment:
Add `getLength()` to every vectored read could be expensive on object stores (unlike local HDFS).
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1335,100 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ final long fileLength = file.getLength();
Review Comment:
Adding `getLength()` to every vectored read could be expensive on object stores (unlike local HDFS).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r3878936749
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -2343,32 +2396,39 @@ private void setReadMetrics(long startNs, long len) {
}
/**
- * Populate data in a parquet file range from a vectored range; will block for up
- * to {@link #HADOOP_VECTORED_READ_TIMEOUT_SECONDS} seconds.
- * @param currRange range to populated.
+ * Populate data in a parquet file range from one or more bounded vectored ranges; together
+ * they may block for up to {@link #HADOOP_VECTORED_READ_TIMEOUT_SECONDS} seconds.
+ * @param ranges bounded ranges containing this part.
* @param builder used to build chunk list to read the pages for the different columns.
* @throws IOException if there is an error while reading from the stream, including a timeout.
*/
- public void readFromVectoredRange(ParquetFileRange currRange, ChunkListBuilder builder) throws IOException {
- ByteBuffer buffer;
+ public void readFromVectoredRanges(List<ParquetFileRange> ranges, ChunkListBuilder builder) throws IOException {
+ List<ByteBuffer> buffers = new ArrayList<>(ranges.size());
+ ParquetFileRange currentRange = null;
final long timeoutSeconds = HADOOP_VECTORED_READ_TIMEOUT_SECONDS;
+ final long timeoutNanos = TimeUnit.SECONDS.toNanos(timeoutSeconds);
long readStart = System.nanoTime();
try {
- LOG.debug(
- "Waiting for vectored read to finish for range {} with timeout {} seconds",
- currRange,
- timeoutSeconds);
- buffer = FutureIO.awaitFuture(currRange.getDataReadFuture(), timeoutSeconds, TimeUnit.SECONDS);
- setReadMetrics(readStart, currRange.getLength());
+ for (ParquetFileRange range : ranges) {
+ currentRange = range;
+ LOG.debug(
+ "Waiting for vectored read to finish for range {} with timeout {} seconds",
+ range,
+ timeoutSeconds);
+ long remainingNanos = Math.max(timeoutNanos - (System.nanoTime() - readStart), 0L);
+ buffers.add(FutureIO.awaitFuture(range.getDataReadFuture(), remainingNanos, TimeUnit.NANOSECONDS));
Review Comment:
`readFromVectoredRanges()` never registers returned buffers with `builder.addBuffersToRelease(...)`; tracking/pooling allocators therefore leak them.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
+ }
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous rejection can follow partial submission. The Hadoop bridge
+ // publishes futures only after submission returns, so missing futures do not prove
+ // that no reads started. Once this call is entered, normal-read fallback is unsafe.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
+ // Consumption may also have populated the builder. Do not replay those chunks.
+ IOException failure =
+ new IOException("Vectored read failed after asynchronous reads may have been submitted", e);
+ awaitRemainingVectoredReads(ranges, readStart, failure);
Review Comment:
After partial submission, futures are copied back to `ParquetFileRange` only when `readWrappedRanges()` returns. A backend failure can leave started reads untracked, so cleanup cannot wait for them.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
+ }
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
Review Comment:
The timeout starts only after `f.readVectored(...)` returns, so a blocking submission is outside the 300-second limit.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
Review Comment:
`file.getLength()` can fail before submission, but that `IOException` bypasses the ordinary-read fallback.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1357,32 +1334,105 @@ private boolean arePartsValidForVectoredIo(List<ConsecutivePartList> allParts) {
* If directly implemented by a Filesystem then it is likely to be a more efficient
* operation such as a scatter-gather read (native IO) or set of parallel
* GET requests against an object store.
+ * The allocation limit applies to filesystem buffers; decoders can still require a
+ * contiguous buffer for an individual logical value larger than that limit.
* @param allParts all parts to be read.
* @param builder used to build chunk list to read the pages for the different columns.
- * @throws IOException any IOE.
- * @throws IllegalArgumentException arguments are invalid.
- * @throws UnsupportedOperationException if the filesystem does not support vectored IO.
+ * @throws IOException if submitting or consuming the vectored reads fails.
+ * @throws IllegalArgumentException if range preparation fails before any reads are submitted.
*/
private void readVectored(List<ConsecutivePartList> allParts, ChunkListBuilder builder) throws IOException {
-
+ final int maximumAllocation = options.getMaxAllocationSize();
+ Preconditions.checkArgument(maximumAllocation > 0, "Invalid maximum allocation size %s", maximumAllocation);
+ if (vectoredReadFileLength < 0) {
+ vectoredReadFileLength = file.getLength();
+ }
+ final long fileLength = vectoredReadFileLength;
List<ParquetFileRange> ranges = new ArrayList<>(allParts.size());
+ List<Integer> partRangeCounts = new ArrayList<>(allParts.size());
long totalSize = 0;
for (ConsecutivePartList consecutiveChunks : allParts) {
final long len = consecutiveChunks.length;
- Preconditions.checkArgument(
- len < Integer.MAX_VALUE,
- "Invalid length %s for vectored read operation. It must be less than max integer value.",
- len);
- ranges.add(new ParquetFileRange(consecutiveChunks.offset, (int) len));
+ final long start = consecutiveChunks.offset;
+ if (start < 0 || len < 0 || start > fileLength || len > fileLength - start) {
+ throw new IOException(String.format(
+ "Invalid vectored read range (offset %d, length %d) for file length %d",
+ start, len, fileLength));
+ }
+ final int firstRange = ranges.size();
+ long remaining = len;
+ long offset = start;
+ do {
+ int rangeLength = (int) Math.min(remaining, maximumAllocation);
+ ranges.add(new ParquetFileRange(offset, rangeLength));
+ offset += rangeLength;
+ remaining -= rangeLength;
+ } while (remaining > 0);
+ partRangeCounts.add(ranges.size() - firstRange);
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges", totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ final long readStart = System.nanoTime();
+ try {
+ // Even a synchronous rejection can follow partial submission. The Hadoop bridge
+ // publishes futures only after submission returns, so missing futures do not prove
+ // that no reads started. Once this call is entered, normal-read fallback is unsafe.
+ f.readVectored(ranges, options.getAllocator());
+ int firstRange = 0;
+ for (int partIndex = 0; partIndex < allParts.size(); partIndex++) {
+ int endRange = firstRange + partRangeCounts.get(partIndex);
+ allParts.get(partIndex).readFromVectoredRanges(ranges.subList(firstRange, endRange), builder);
+ firstRange = endRange;
+ }
+ } catch (IllegalArgumentException | UnsupportedOperationException e) {
+ // Consumption may also have populated the builder. Do not replay those chunks.
+ IOException failure =
+ new IOException("Vectored read failed after asynchronous reads may have been submitted", e);
+ awaitRemainingVectoredReads(ranges, readStart, failure);
+ throw failure;
+ } catch (IOException | RuntimeException e) {
+ awaitRemainingVectoredReads(ranges, readStart, e);
+ throw e;
+ }
+ }
+
+ /**
+ * Wait for submitted reads with published futures to finish before their stream can be
+ * closed. Cancelling result futures does not stop all Hadoop backends from continuing IO.
+ */
+ private void awaitRemainingVectoredReads(List<ParquetFileRange> ranges, long readStart, Throwable failure) {
+ if (Thread.currentThread().isInterrupted()
+ || failure instanceof InterruptedIOException && failure.getCause() instanceof InterruptedException) {
Review Comment:
An interrupted wait returns before sibling futures are drained; `close()` may then race with backend reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726#discussion_r4216144982
##########
parquet-hadoop/README.md:
##########
@@ -558,6 +558,26 @@ if not found then the library will use the classic non-vectored reads: it is saf
---
+**JVM system property:** `parquet.hadoop.vectored.io.threads`
+
+**Description:** Maximum number of shared vectored-read workers. Set a positive integer
Review Comment:
This executor is a JVM-global singleton read once at class initialization, so the limit is process-wide and cannot be changed per reader or per query.
Would it make sense to also let the caller supply the executor, with this shared pool as the default? That way an engine could size and share it in the way it already manages its own pools.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3719: Fix vectored read allocation limits and fallback safety [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #3726:
URL: https://github.com/apache/parquet-java/pull/3726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org