[PR] feat(parquet): wire scan_filtered through ArrayReader stack; add with_miniblock_predicate [arrow-rs]

["sahuagin (via GitHub)" <gi...@apache.org>]

sahuagin opened a new pull request, #9770:
URL: https://github.com/apache/arrow-rs/pull/9770

     Which issue does this PR close?
   
     Depends on #9769 (the decoder PR — diff includes those changes while that PR is pending).
   
     Rationale for this change
   
     Exposes the scan_filtered miniblock-level predicate pushdown added in #9769 through the full column reader stack and as a public API. For mandatory DELTA_BINARY_PACKED
     INT32/INT64 columns, entire miniblocks (32/64 values) can be skipped without decoding when a caller-supplied range predicate rules them out. This is especially effective
     for monotone columns (timestamps, sequence numbers, auto-increment IDs) where bw=0 blocks allow O(1) skipping.
   
     What changes are included in this PR?
   
     Wiring chain (bottom to top):
     - ColumnValueDecoderImpl::scan_filtered_values() — dispatches to decoder
     - GenericColumnReader::scan_filtered_records() — mandatory columns only; optional/repeated fall back to full decode to keep def/rep levels in sync
     - GenericRecordReader::scan_filtered_records() — page-switching loop with same fallback
     - ArrayReader::scan_records() — new provided trait method (default = read_records, safe for all encodings/types)
     - PrimitiveArrayReader::scan_records() — override that calls the above chain
     - StructArrayReader::scan_records() — delegates to its single child for single-column projections; multi-column projections fall back to read_records
     - ParquetRecordBatchReader::next_inner — uses scan_records in the All selection branch when a predicate is set
     - ArrowReaderBuilder::with_miniblock_predicate() — fluent setter; works for both ParquetRecordBatchReaderBuilder (sync) and ParquetRecordBatchStreamBuilder (async)
   
     New public API:
     pub type MiniblockPredicate = Arc<dyn Fn(i64, i64) -> bool + Send + Sync>;
   
     // on ArrowReaderBuilder<T>:
     pub fn with_miniblock_predicate(self, predicate: MiniblockPredicate) -> Self;
   
     Example:
     let pred: MiniblockPredicate = Arc::new(|_lo, hi| hi >= 1_000_000);
     let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
         .with_projection(mask)
         .with_miniblock_predicate(pred)
         .build()?;
   
     Limitations (documented in MiniblockPredicate rustdoc):
     - Multi-column projections fall back to full decode (each column has independent value ranges)
     - Optional/repeated columns fall back (def/rep level synchronization required)
     - No file format changes; miniblock ranges are computed on-the-fly from data already present in DELTA_BINARY_PACKED block headers
   
     Are these changes tested?
   
     - test_scan_records_delta_binary_packed_mandatory in primitive_array.rs: unit test exercising the full wiring chain on a mandatory INT64 DELTA column, verifying correct
     miniblock-level skipping and no false negatives
     - test_with_miniblock_predicate_single_column in arrow_reader/mod.rs: end-to-end test through ParquetRecordBatchReaderBuilder verifying the public API, correct output,
     and that fewer rows are returned than a full read
   
     Are there any user-facing changes?
   
     Yes — two additive public API items:
     - MiniblockPredicate type alias
     - ArrowReaderBuilder::with_miniblock_predicate() method
   
     No breaking changes. The new scan_records method on the ArrayReader trait has a provided default (read_records) so no existing implementations are affected.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(parquet): wire scan_filtered through ArrayReader stack; add with_miniblock_predicate [arrow-rs]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/arrow-rs/pull/9770#issuecomment-4888576945

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(parquet): wire scan_filtered through ArrayReader stack; add with_miniblock_predicate [arrow-rs]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] closed pull request #9770: feat(parquet): wire scan_filtered through ArrayReader stack; add with_miniblock_predicate
URL: https://github.com/apache/arrow-rs/pull/9770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org