[I] [Feature] Support file index and predicate pushdown in DataEvolutionSplitRead [paimon-cpp]

["lxy-9602 (via GitHub)" <gi...@apache.org>]

lxy-9602 opened a new issue, #214:
URL: https://github.com/apache/paimon-cpp/issues/214

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
   
   
   ### Motivation
   
   Java Paimon supports file-index evaluation and predicate pushdown for data-evolution reads since [apache/paimon#8839](https://github.com/apache/paimon/pull/8839).
   
   paimon-cpp currently does not support this optimization:
   
   - `DataEvolutionSplitRead` passes a null predicate when creating underlying format readers.
   - `DataEvolutionSplitRead::ApplyIndexAndDvReaderIfNeeded` rejects a non-null predicate.
   - File indexes therefore cannot be used to skip files or select matching row positions.
   - Parquet/ORC readers cannot use predicates to skip row groups or pages.
   
   As a result, queries with selective predicates may scan all data belonging to a data-evolution split, even when file-index metadata or format statistics could eliminate most of it.
   
   Predicate pushdown needs special handling for data evolution. Files in the same field group are merged by physical row position. Applying row-level predicates independently to these readers could filter different positions and break column alignment. In addition, when multiple files contain the same field, the newest field value wins, so an index built from an overwritten copy must not incorrectly eliminate the group.
   
   
   ### Solution
   
   Follow [apache/paimon#8839](https://github.com/apache/paimon/pull/8839) to support predicate pushdown in `DataEvolutionSplitRead`:
   
   - Push applicable predicates to file indexes and format readers for single-file groups.
   - For multi-file groups, only skip the whole group when safe; do not filter child readers independently.
   - Map predicates by field ID to support schema evolution.
   - Keep residual filtering to guarantee correctness.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support file index and predicate pushdown in DataEvolutionSplitRead [paimon-cpp]

["SteNicholas (via GitHub)" <gi...@apache.org>]

SteNicholas closed issue #214: [Feature] Support file index and predicate pushdown in DataEvolutionSplitRead
URL: https://github.com/apache/paimon-cpp/issues/214


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org