You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sundapeng (via GitHub)" <gi...@apache.org> on 2026/09/11 07:24:32 UTC
[PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
sundapeng opened a new pull request, #9740:
URL: https://github.com/apache/paimon/pull/9740
### Purpose
Opening a Mosaic row group is a chain of dependent range reads: the bucket directories first, then the projected column slots. `MosaicRecordsReader` opened one row group at a time and only after the previous one was consumed, so a file with about 120 row groups on object storage paid about 240 sequential round trips of roughly 50 ms each while reading under 5 MB. On a 11,374-column table this was about 12 s per file for a query that keeps 505 rows.
Changes:
- `mosaic.read.prefetch-row-groups` (default 8, 0 disables): the reader schedules the next matching row groups on a shared daemon pool while the current one is consumed. Skipped row groups still advance the file position, and closing the reader waits for in-flight opens and releases their Arrow batches so the allocator does not report leaks.
- `MosaicInputFileAdapter` borrows one `SeekableInputStream` per concurrent read from a small pool. Positional reads on a single stream are serialized by common implementations, which turned the parallel opens back into a chain; with the pool, per-file scan time dropped from ~12 s to ~1.7 s in the same setup.
- Read statistics (row groups, reads, bytes, wait time) are logged at debug level when the reader closes.
Depth 8 gave the best scan time on an 8-core host reading from OSS; 16 saturated the connections. End to end, the customer query went from 19.3 s to 4.0 s together with the manifest cache configuration, and to 1.7 s on a rebuilt table with lean manifests and `mosaic.stats-columns` (Parquet baseline for the same data: 2.2 s).
### Tests
- `MosaicReaderWriterTest#testPrefetchedReadMatchesSequentialRead` (new): prefetched and sequential reads return the same rows and the same file positions, including across row groups skipped by a predicate.
- `MosaicReaderWriterTest#testClosingReaderWithPendingPrefetchReleasesResources` (new): closing with prefetched batches in flight releases them.
- All `paimon-mosaic` tests pass locally.
### API and Format
New read option `mosaic.read.prefetch-row-groups`. No format change.
### Documentation
The option is added to the Mosaic section of `docs/docs/concepts/spec/fileformat.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #9740:
URL: https://github.com/apache/paimon/pull/9740#issuecomment-5632589946
0bb8884c3 changes what `mosaic.read.prefetch-max-bytes` bounds. The previous revision derived the budget from the average on-disk row group size, which underestimates decoded memory badly for wide projections: a full-table copy of the 11,374-column table on Spark executors was OOM-killed because a 1.5 MB row group decodes to about 1 GB when every column is projected. The reader now estimates the decoded size of each row group from its row count and the projected column types and only schedules further row groups while the in-flight estimate fits the budget (the next row group is always read). The narrow customer query keeps depth 8; a full projection reads on demand. `testPrefetchIsBoundedByEstimatedDecodedBytes` covers both sides of the budget.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9740:
URL: https://github.com/apache/paimon/pull/9740#issuecomment-5632638398
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #9740:
URL: https://github.com/apache/paimon/pull/9740#issuecomment-5631013514
cc @jianguotian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on code in PR #9740:
URL: https://github.com/apache/paimon/pull/9740#discussion_r3987126232
##########
paimon-mosaic/src/main/java/org/apache/paimon/format/mosaic/MosaicRecordsReader.java:
##########
@@ -136,62 +202,151 @@ public MosaicRecordsReader(
this.reader = createdReader;
this.numRowGroups = createdNumRowGroups;
this.allProjectedColumnsMissing = createdAllProjectedColumnsMissing;
- this.currentRowGroup = 0;
this.arrowBatchReader = createdArrowBatchReader;
}
@Nullable
@Override
public FileRecordIterator<InternalRow> readBatch() throws IOException {
- while (currentRowGroup < numRowGroups) {
- int numRows = reader.rowGroupNumRows(currentRowGroup);
- if (!matchesRowGroup(currentRowGroup, numRows)) {
- returnedPosition += numRows;
- currentRowGroup++;
- continue;
+ releaseCurrentVsr();
+
+ RowGroupBatch batch = nextRowGroup();
+ if (batch == null) {
+ return null;
+ }
+ // Rows of skipped row groups still count towards the file position.
+ returnedPosition = batch.startPosition - 1;
+ rowsReturned += batch.numRows;
+
+ if (allProjectedColumnsMissing) {
+ return allNullIterator(batch.numRows);
+ }
+
+ VectorSchemaRoot vsr = batch.vsr;
+ this.currentVsr = vsr;
+
+ Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+
+ return new FileRecordIterator<InternalRow>() {
+ @Override
+ public long returnedPosition() {
+ return returnedPosition;
}
- releaseCurrentVsr();
+ @Override
+ public Path filePath() {
+ return filePath;
+ }
- if (allProjectedColumnsMissing) {
- currentRowGroup++;
- return allNullIterator(numRows);
+ @Nullable
+ @Override
+ public InternalRow next() {
+ if (rows.hasNext()) {
+ returnedPosition++;
+ return rows.next();
+ }
+ return null;
}
- VectorSchemaRoot vsr = reader.readRowGroup(currentRowGroup, allocator);
- currentRowGroup++;
- this.currentVsr = vsr;
+ @Override
+ public void releaseBatch() {
+ releaseCurrentVsr();
+ }
+ };
+ }
- Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+ /** Returns the next matching row group with its data ready, or null at end of file. */
+ @Nullable
+ private RowGroupBatch nextRowGroup() throws IOException {
+ fillPrefetchQueue();
+ RowGroupBatch head = pending.poll();
+ if (head == null) {
+ return null;
+ }
+ // Keep the pipeline full while waiting for the head to arrive.
+ fillPrefetchQueue();
+ long waitStart = System.nanoTime();
+ head.await();
+ openNanos += System.nanoTime() - waitStart;
+ openedRowGroups++;
+ return head;
+ }
- return new FileRecordIterator<InternalRow>() {
- @Override
- public long returnedPosition() {
- return returnedPosition;
- }
+ /** Schedules matching row groups until the queue holds the head plus {@code prefetchDepth}. */
+ private void fillPrefetchQueue() {
+ int wanted = prefetchDepth + 1;
+ while (pending.size() < wanted && nextRowGroupToSchedule < numRowGroups) {
+ int index = nextRowGroupToSchedule++;
+ int numRows = reader.rowGroupNumRows(index);
+ long startPosition = scheduledRowCount;
+ scheduledRowCount += numRows;
+ if (!matchesRowGroup(index, numRows)) {
+ continue;
+ }
+ RowGroupBatch batch = new RowGroupBatch(index, numRows, startPosition);
+ if (allProjectedColumnsMissing) {
+ // Nothing to read from the file for this batch.
+ } else if (prefetchDepth == 0) {
+ batch.vsr = reader.readRowGroup(index, allocator);
+ } else {
+ batch.future = PREFETCH_POOL.submit(() -> reader.readRowGroup(index, allocator));
+ }
+ pending.add(batch);
+ }
+ }
- @Override
- public Path filePath() {
- return filePath;
- }
+ /** A row group whose data is being, or has been, loaded. */
+ private static final class RowGroupBatch {
+ final int index;
+ final int numRows;
+ final long startPosition;
+ @Nullable Future<VectorSchemaRoot> future;
+ @Nullable VectorSchemaRoot vsr;
+
+ RowGroupBatch(int index, int numRows, long startPosition) {
+ this.index = index;
+ this.numRows = numRows;
+ this.startPosition = startPosition;
+ }
- @Nullable
- @Override
- public InternalRow next() {
- if (rows.hasNext()) {
- returnedPosition++;
- return rows.next();
- }
- return null;
+ void await() throws IOException {
+ if (future == null) {
+ return;
+ }
+ try {
+ vsr = future.get();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IOException("Interrupted while opening row group " + index, e);
+ } catch (ExecutionException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof IOException) {
+ throw (IOException) cause;
}
-
- @Override
- public void releaseBatch() {
- releaseCurrentVsr();
+ if (cause instanceof RuntimeException) {
+ throw (RuntimeException) cause;
}
- };
+ if (cause instanceof Error) {
+ throw (Error) cause;
+ }
+ throw new IOException("Failed to open row group " + index, cause);
+ } finally {
+ future = null;
Review Comment:
Fixed in 288e76084. The head now stays in `pending` until `await()` has returned its data, so an interrupted wait leaves the batch owned by the reader. `close()` drains every queued batch with a loop that keeps waiting through `InterruptedException` and only sets the interrupt flag again after the native reader, the allocator and the input streams are closed; nothing is cancelled, because a cancelled Future says nothing about the native read.
Two tests reproduce both paths with a latch-blocked `readRowGroup`: `testInterruptedReadKeepsInFlightRowGroupUntilClose` (interrupt during `readBatch()`, then `close()` from another thread) and `testCloseWithInterruptFlagStillWaitsForInFlightRowGroups` (entering `close()` with the flag set). Both assert that `close()` blocks until the read finishes, that the native reader is closed only after it, that the allocator closes without a leak, and that the flag is still set afterwards. Both fail on the previous revision.
##########
paimon-mosaic/src/main/java/org/apache/paimon/format/mosaic/MosaicRecordsReader.java:
##########
@@ -136,62 +202,151 @@ public MosaicRecordsReader(
this.reader = createdReader;
this.numRowGroups = createdNumRowGroups;
this.allProjectedColumnsMissing = createdAllProjectedColumnsMissing;
- this.currentRowGroup = 0;
this.arrowBatchReader = createdArrowBatchReader;
}
@Nullable
@Override
public FileRecordIterator<InternalRow> readBatch() throws IOException {
- while (currentRowGroup < numRowGroups) {
- int numRows = reader.rowGroupNumRows(currentRowGroup);
- if (!matchesRowGroup(currentRowGroup, numRows)) {
- returnedPosition += numRows;
- currentRowGroup++;
- continue;
+ releaseCurrentVsr();
+
+ RowGroupBatch batch = nextRowGroup();
+ if (batch == null) {
+ return null;
+ }
+ // Rows of skipped row groups still count towards the file position.
+ returnedPosition = batch.startPosition - 1;
+ rowsReturned += batch.numRows;
+
+ if (allProjectedColumnsMissing) {
+ return allNullIterator(batch.numRows);
+ }
+
+ VectorSchemaRoot vsr = batch.vsr;
+ this.currentVsr = vsr;
+
+ Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+
+ return new FileRecordIterator<InternalRow>() {
+ @Override
+ public long returnedPosition() {
+ return returnedPosition;
}
- releaseCurrentVsr();
+ @Override
+ public Path filePath() {
+ return filePath;
+ }
- if (allProjectedColumnsMissing) {
- currentRowGroup++;
- return allNullIterator(numRows);
+ @Nullable
+ @Override
+ public InternalRow next() {
+ if (rows.hasNext()) {
+ returnedPosition++;
+ return rows.next();
+ }
+ return null;
}
- VectorSchemaRoot vsr = reader.readRowGroup(currentRowGroup, allocator);
- currentRowGroup++;
- this.currentVsr = vsr;
+ @Override
+ public void releaseBatch() {
+ releaseCurrentVsr();
+ }
+ };
+ }
- Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+ /** Returns the next matching row group with its data ready, or null at end of file. */
+ @Nullable
+ private RowGroupBatch nextRowGroup() throws IOException {
+ fillPrefetchQueue();
+ RowGroupBatch head = pending.poll();
+ if (head == null) {
+ return null;
+ }
+ // Keep the pipeline full while waiting for the head to arrive.
+ fillPrefetchQueue();
Review Comment:
Fixed in 288e76084. Depth 0 now goes through a true sequential path: the queue is only filled when it is empty, and the refill after handing over the head runs only when `prefetchDepth > 0`. The refill also runs after `currentVsr` has been assigned, so a failure while scheduling the next row group leaves the current batch owned by the reader and `close()` releases it.
`testDisabledPrefetchReadsRowGroupsOnDemand` verifies exactly one `readRowGroup` call before the first batch is consumed (fails on the previous revision), and `testRefillFailureLeavesCurrentRowGroupReleasable` makes `rowGroupNumRows(1)` throw and checks that the allocator closes without a leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #9740:
URL: https://github.com/apache/paimon/pull/9740#issuecomment-5631894649
Pushed b654310bb with a second look at the memory and connection footprint of the prefetch, beyond the two review findings:
- The depth was a plain row group count. Files written with Mosaic's default 256 MB row groups would keep up to nine decoded batches per reader; `mosaic.read.prefetch-max-bytes` (default 64 MB) now caps the look-ahead by the average row group size of the file, so such files read without look-ahead while small row groups keep the full depth.
- `MosaicInputFileAdapter` opens at most depth + 1 streams and a read that finds them all busy waits, so the connections used by one reader are bounded whatever the native fan-out per row group is.
- The per-read counters and `ioStats()` are gone (they only fed one debug line); the debug line is now written after the pending reads are drained. The unused public constructors are removed, the depth is clamped once, and the refill-failure test now really fails the refill after the first batch was handed over.
The PR description lists the tests; both options are documented in `fileformat.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9740:
URL: https://github.com/apache/paimon/pull/9740#discussion_r3986955907
##########
paimon-mosaic/src/main/java/org/apache/paimon/format/mosaic/MosaicRecordsReader.java:
##########
@@ -136,62 +202,151 @@ public MosaicRecordsReader(
this.reader = createdReader;
this.numRowGroups = createdNumRowGroups;
this.allProjectedColumnsMissing = createdAllProjectedColumnsMissing;
- this.currentRowGroup = 0;
this.arrowBatchReader = createdArrowBatchReader;
}
@Nullable
@Override
public FileRecordIterator<InternalRow> readBatch() throws IOException {
- while (currentRowGroup < numRowGroups) {
- int numRows = reader.rowGroupNumRows(currentRowGroup);
- if (!matchesRowGroup(currentRowGroup, numRows)) {
- returnedPosition += numRows;
- currentRowGroup++;
- continue;
+ releaseCurrentVsr();
+
+ RowGroupBatch batch = nextRowGroup();
+ if (batch == null) {
+ return null;
+ }
+ // Rows of skipped row groups still count towards the file position.
+ returnedPosition = batch.startPosition - 1;
+ rowsReturned += batch.numRows;
+
+ if (allProjectedColumnsMissing) {
+ return allNullIterator(batch.numRows);
+ }
+
+ VectorSchemaRoot vsr = batch.vsr;
+ this.currentVsr = vsr;
+
+ Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+
+ return new FileRecordIterator<InternalRow>() {
+ @Override
+ public long returnedPosition() {
+ return returnedPosition;
}
- releaseCurrentVsr();
+ @Override
+ public Path filePath() {
+ return filePath;
+ }
- if (allProjectedColumnsMissing) {
- currentRowGroup++;
- return allNullIterator(numRows);
+ @Nullable
+ @Override
+ public InternalRow next() {
+ if (rows.hasNext()) {
+ returnedPosition++;
+ return rows.next();
+ }
+ return null;
}
- VectorSchemaRoot vsr = reader.readRowGroup(currentRowGroup, allocator);
- currentRowGroup++;
- this.currentVsr = vsr;
+ @Override
+ public void releaseBatch() {
+ releaseCurrentVsr();
+ }
+ };
+ }
- Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+ /** Returns the next matching row group with its data ready, or null at end of file. */
+ @Nullable
+ private RowGroupBatch nextRowGroup() throws IOException {
+ fillPrefetchQueue();
+ RowGroupBatch head = pending.poll();
+ if (head == null) {
+ return null;
+ }
+ // Keep the pipeline full while waiting for the head to arrive.
+ fillPrefetchQueue();
Review Comment:
[P2] Avoid refilling ahead when prefetch is disabled
This second refill also runs when `prefetchDepth == 0`. The first refill synchronously reads group 0, `poll()` removes it, and this call synchronously reads group 1 before the first batch can be returned. Thus setting `mosaic.read.prefetch-row-groups=0` does not restore on-demand reading: even `LIMIT 1` waits for an unnecessary next-group read and retains an extra decoded batch.
There is also an error-path leak here: if reading group 1 throws, group 0 is no longer in `pending` and has not yet been assigned to `currentVsr`, so `close()` never closes it. Both the extra read and the missing batch cleanup reproduce with controlled reader mocks. Please use a true sequential path for depth 0 and keep the selected head owned until it is handed to the consumer or cleaned up after a refill failure.
##########
paimon-mosaic/src/main/java/org/apache/paimon/format/mosaic/MosaicRecordsReader.java:
##########
@@ -136,62 +202,151 @@ public MosaicRecordsReader(
this.reader = createdReader;
this.numRowGroups = createdNumRowGroups;
this.allProjectedColumnsMissing = createdAllProjectedColumnsMissing;
- this.currentRowGroup = 0;
this.arrowBatchReader = createdArrowBatchReader;
}
@Nullable
@Override
public FileRecordIterator<InternalRow> readBatch() throws IOException {
- while (currentRowGroup < numRowGroups) {
- int numRows = reader.rowGroupNumRows(currentRowGroup);
- if (!matchesRowGroup(currentRowGroup, numRows)) {
- returnedPosition += numRows;
- currentRowGroup++;
- continue;
+ releaseCurrentVsr();
+
+ RowGroupBatch batch = nextRowGroup();
+ if (batch == null) {
+ return null;
+ }
+ // Rows of skipped row groups still count towards the file position.
+ returnedPosition = batch.startPosition - 1;
+ rowsReturned += batch.numRows;
+
+ if (allProjectedColumnsMissing) {
+ return allNullIterator(batch.numRows);
+ }
+
+ VectorSchemaRoot vsr = batch.vsr;
+ this.currentVsr = vsr;
+
+ Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+
+ return new FileRecordIterator<InternalRow>() {
+ @Override
+ public long returnedPosition() {
+ return returnedPosition;
}
- releaseCurrentVsr();
+ @Override
+ public Path filePath() {
+ return filePath;
+ }
- if (allProjectedColumnsMissing) {
- currentRowGroup++;
- return allNullIterator(numRows);
+ @Nullable
+ @Override
+ public InternalRow next() {
+ if (rows.hasNext()) {
+ returnedPosition++;
+ return rows.next();
+ }
+ return null;
}
- VectorSchemaRoot vsr = reader.readRowGroup(currentRowGroup, allocator);
- currentRowGroup++;
- this.currentVsr = vsr;
+ @Override
+ public void releaseBatch() {
+ releaseCurrentVsr();
+ }
+ };
+ }
- Iterator<InternalRow> rows = arrowBatchReader.readBatch(vsr).iterator();
+ /** Returns the next matching row group with its data ready, or null at end of file. */
+ @Nullable
+ private RowGroupBatch nextRowGroup() throws IOException {
+ fillPrefetchQueue();
+ RowGroupBatch head = pending.poll();
+ if (head == null) {
+ return null;
+ }
+ // Keep the pipeline full while waiting for the head to arrive.
+ fillPrefetchQueue();
+ long waitStart = System.nanoTime();
+ head.await();
+ openNanos += System.nanoTime() - waitStart;
+ openedRowGroups++;
+ return head;
+ }
- return new FileRecordIterator<InternalRow>() {
- @Override
- public long returnedPosition() {
- return returnedPosition;
- }
+ /** Schedules matching row groups until the queue holds the head plus {@code prefetchDepth}. */
+ private void fillPrefetchQueue() {
+ int wanted = prefetchDepth + 1;
+ while (pending.size() < wanted && nextRowGroupToSchedule < numRowGroups) {
+ int index = nextRowGroupToSchedule++;
+ int numRows = reader.rowGroupNumRows(index);
+ long startPosition = scheduledRowCount;
+ scheduledRowCount += numRows;
+ if (!matchesRowGroup(index, numRows)) {
+ continue;
+ }
+ RowGroupBatch batch = new RowGroupBatch(index, numRows, startPosition);
+ if (allProjectedColumnsMissing) {
+ // Nothing to read from the file for this batch.
+ } else if (prefetchDepth == 0) {
+ batch.vsr = reader.readRowGroup(index, allocator);
+ } else {
+ batch.future = PREFETCH_POOL.submit(() -> reader.readRowGroup(index, allocator));
+ }
+ pending.add(batch);
+ }
+ }
- @Override
- public Path filePath() {
- return filePath;
- }
+ /** A row group whose data is being, or has been, loaded. */
+ private static final class RowGroupBatch {
+ final int index;
+ final int numRows;
+ final long startPosition;
+ @Nullable Future<VectorSchemaRoot> future;
+ @Nullable VectorSchemaRoot vsr;
+
+ RowGroupBatch(int index, int numRows, long startPosition) {
+ this.index = index;
+ this.numRows = numRows;
+ this.startPosition = startPosition;
+ }
- @Nullable
- @Override
- public InternalRow next() {
- if (rows.hasNext()) {
- returnedPosition++;
- return rows.next();
- }
- return null;
+ void await() throws IOException {
+ if (future == null) {
+ return;
+ }
+ try {
+ vsr = future.get();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IOException("Interrupted while opening row group " + index, e);
+ } catch (ExecutionException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof IOException) {
+ throw (IOException) cause;
}
-
- @Override
- public void releaseBatch() {
- releaseCurrentVsr();
+ if (cause instanceof RuntimeException) {
+ throw (RuntimeException) cause;
}
- };
+ if (cause instanceof Error) {
+ throw (Error) cause;
+ }
+ throw new IOException("Failed to open row group " + index, cause);
+ } finally {
+ future = null;
Review Comment:
[P1] Preserve ownership of in-flight reads when interrupted
If `Future.get()` is interrupted, the background read keeps running, but this `finally` discards its Future. The head has also already been removed from `pending`, so `close()` cannot drain it. Closing with the interrupt flag set similarly makes `discard()` abandon the other unfinished Futures. Consequently, `reader.close()` and `allocator.close()` can run while native I/O is still active. Mosaic 0.2.0 [directly frees the reader handle](https://github.com/apache/paimon-mosaic/blob/v0.2.0/jni/src/lib.rs#L599-L606), which makes this a use-after-free/JVM crash risk if a row-group open is still using it.
A latch-controlled test reproduces native-reader closure before the blocked read finishes, both after interrupting `readBatch()` and when entering `close()` with an interrupt flag. Please retain ownership of every in-flight batch, including the polled head, and drain/close them despite interruption before releasing shared resources, restoring the interrupt flag afterward. Cancelling a Future alone does not establish that the native read has stopped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [mosaic] Prefetch row groups and read with one input stream per concurrent read [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9740:
URL: https://github.com/apache/paimon/pull/9740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org