You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jianguotian (via GitHub)" <gi...@apache.org> on 2026/09/11 08:17:37 UTC

[PR] [mosaic] Push row selection down to row-group reads [paimon]

jianguotian opened a new pull request, #9742:
URL: https://github.com/apache/paimon/pull/9742

   ### Purpose
   
   File indexes expose matching row positions through `FormatReaderFactory.Context.selection()`. Exact row filtering remains in `DataFileRecordReader`, while format readers may use the selection for physical pruning. `MosaicReaderFactory` currently drops this selection, so `MosaicRecordsReader` invokes `MosaicReader.readRowGroup` for every predicate-matching row group in a retained file.
   
   This change forwards the selection and skips row groups whose file-relative half-open row range does not intersect the bitmap. It keeps a physical row-group offset separate from `returnedPosition` so pruning stays correct when the outer selection stops a batch early. It does not change exact row filtering, the Mosaic file format, or the native Mosaic library.
   
   This is orthogonal to #9740: that pull request optimizes row-group I/O and prefetch, while this change prevents unmatched row groups from being scheduled or read. If #9740 lands first, the selection check should move ahead of prefetch scheduling during rebase.
   
   In a production-like Spark A/B over identical four-split scans, five alternating pairs returned identical results; median latency decreased from 20.4 s to 5.5 s (about 3.7x).
   
   ### Tests
   
   - `mvn -pl paimon-mosaic -am -DskipTests package`
   - `mvn -pl paimon-mosaic -am -DfailIfNoTests=false -DwildcardSuites=none -Dtest='org.apache.paimon.format.mosaic.*Test' test` (86 tests)
   - Regression coverage verifies factory propagation, unmatched row-group pruning, exact file-relative positions across skipped and partially consumed groups, and Arrow buffer release.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [mosaic] Push row selection down to row-group reads [paimon]

Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #9742:
URL: https://github.com/apache/paimon/pull/9742#issuecomment-5632861268

   #9740 is merged now (6948dd55d), so this will need a rebase: `MosaicRecordsReader.readBatch()` no longer iterates row groups itself. The natural place for the selection check is `fillPrefetchQueue()`, next to `matchesRowGroup(index, numRows)`: a row group whose file-relative range does not intersect the selection is skipped before it is scheduled, with `scheduledRowCount` already tracking the physical start position you keep in `nextRowGroupStart`. I merged the two locally that way (selection passed through `MosaicReaderFactory` into the reader, `testSelectionSkipsUnmatchedRowGroupsAndPreservesPositions` adapted to the prefetch constructor) and the whole `paimon-mosaic` suite passes, so there is no semantic conflict.
   
   One small suggestion for the tests: the only test that goes through `MosaicReaderFactory` with a selection uses an empty bitmap. A positive case on the same file (`RoaringBitmap32.bitmapOf(1)` through `DataFileRecordReader`, expecting exactly the second row at position 1) would catch a factory that forwards an empty or null selection.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [mosaic] Push row selection down to row-group reads [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9742:
URL: https://github.com/apache/paimon/pull/9742


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [mosaic] Push row selection down to row-group reads [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9742:
URL: https://github.com/apache/paimon/pull/9742#issuecomment-5636503416

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org