You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "hhhizzz (via GitHub)" <gi...@apache.org> on 2025/10/28 18:30:08 UTC
[PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
hhhizzz opened a new pull request, #8733:
URL: https://github.com/apache/arrow-rs/pull/8733
# Which issue does this PR close?
- Closes #8565
- Closes #5523
# Rationale for this change
This change improves the performance of reading Parquet files.
# What changes are included in this PR?
This pull request introduces significant improvements to row selection and filtering in the Parquet Arrow reader, optimizing batch reading and handling of sparse data. The most important changes include a new mask-based row selection state, enhancements to synthetic page handling, and expanded test coverage for these features.
**Row selection and filtering improvements:**
* Introduced `RowSelectionState` in `read_plan.rs`, which dynamically chooses between a bitmap mask array and selector queue for efficient row selection during batch reads. This enables streaming with contiguous mask segments and reduces overhead for sparse selections.
* Updated `ParquetRecordBatchReader` to leverage the mask-based selection, streaming record batches using boolean masks and applying Arrow filtering for selected rows. This avoids intermediate materialization and improves performance for sparse row selections. If the average length of the `RowSelector` is less than 8, it will be replaced by a bitmap mask.
* If the average RowSelector length is less than 8, it is automatically replaced by a bitmap mask.
* Added a benchmark to determine this threshold value (8).
**Synthetic page and definition level handling:**
A challenge with the mask-based approach is that some pages may be skipped, and due to the streaming design of the reader, it’s not always possible to determine in advance which pages will be skipped.
To address this, additional logic was added to return None when a page is skipped, ensuring correct handling in such cases.
Together, these improvements enhance both efficiency and correctness in row selection, filtering, and sparse data processing for the Parquet Arrow reader.
# Are these changes tested?
* Added new tests for interleaved skip/select row selections and mask-based sparse row selection, ensuring correctness of the new mask-based streaming logic and synthetic page handling.
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483681768
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
I think this will cause the reader to produce batches potentially much smaller than the requested batch size, which at the very least should be documented. Ideally we'd just store the mask somewhere, and flush + filter when we have enough rows.
Otherwise consumers like DataFusion are going to need to wrap ParquetExec in a CoalesceBatches and effectively buffer all the data again.
Edit edit: Filtering small batches may also have performance implications - the filter kernels are optimized for filtering thousands of rows at a time.
Edit edit edit: actually realised the selector is for the entire row group!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483684259
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
I am similarly still confused.
@hhhizzz your explanation makes sense to me in theory, but I just tested out removing the synthetic page code from this PR and the tests all still seem to pass. So that means we either have a testing gap or there is something else going on:
- https://github.com/hhhizzz/arrow-rs/pull/5
I looked more carefully, and it seems to me that the calculation of what pages to fetch is still based on `RowSelection` (not the `RowSelectionCursor` / `RowSelectionBacking`):
https://github.com/apache/arrow-rs/blob/cc1444a3232fa11b8485e2794a88f342bd7f97e2/parquet/src/arrow/async_reader/mod.rs#L983-L995
Thus it does feel possible to have the the situation you explain where pages needed to evaluate the row selection weren't fetched 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2510979304
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -212,6 +217,14 @@ impl<T> ArrowReaderBuilder<T> {
}
}
+ /// Configure how row selections should be materialised during execution
Review Comment:
It would be nice, eventually, to change RowSelection so the callers could provide a bitmap as well.
I will think about how this might look, but it is not something to change in this PR
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -43,6 +43,7 @@ use crate::file::reader::{ChunkReader, SerializedPageReader};
use crate::schema::types::SchemaDescriptor;
use crate::arrow::arrow_reader::metrics::ArrowReaderMetrics;
+// Exposed so integration tests and benchmarks can temporarily override the threshold.
Review Comment:
I think there are other use cases too (aka I am not sure this comment is accurate anymore)
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -934,12 +950,16 @@ impl<T: ChunkReader + 'static> ParquetRecordBatchReaderBuilder<T> {
let array_reader = ArrayReaderBuilder::new(&reader, &metrics)
.build_array_reader(fields.as_deref(), &projection)?;
- let read_plan = plan_builder
+ let mut plan_builder = plan_builder
.limited(reader.num_rows())
.with_offset(offset)
.with_limit(limit)
- .build_limited()
- .build();
+ .build_limited();
+
+ let preferred_strategy = plan_builder.preferred_selection_strategy();
Review Comment:
Elsewhere in the PR you used the term "resolved" to reflect the chosen strategy -- perhaps we can do the same here
```rust
let resolved_strategy = plan_builder.resolve_selection_strategy();
plan_builder = plan_builder.with_selection_strategy(resolved_strategy);
```
I also found it slightly strange that the PlanBuilder has both methods -- maybe it would be simpler if build() simply resolved the strategy directly.
However, I think we can do this as a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3533466456
Thank you again @hhhizzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3467320795
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main rowselectionempty
----- ---- -----------------
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 1099.8±2.52µs ? ?/sec 1.16 1274.8±3.26µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 1269.3±2.75µs ? ?/sec 1.03 1307.9±4.94µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 1107.4±3.98µs ? ?/sec 1.16 1281.4±2.80µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.05 513.1±6.17µs ? ?/sec 1.00 486.6±3.42µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.00 660.4±3.25µs ? ?/sec 1.02 673.5±6.14µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.03 506.2±2.98µs ? ?/sec 1.00 493.5±3.45µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.04 572.5±2.07µs ? ?/sec 1.00 552.8±2.47µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.00 725.3±4.38µs ? ?/sec 1.01 731.3±3.60µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.03 585.1±3.07µs ? ?/sec 1.00 565.3±3.97µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 238.9±3.00µs ? ?/sec 1.14 271.5±2.70µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.00 214.6±0.58µs ? ?/sec 1.24 266.2±1.12µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 235.8±2.71µs ? ?/sec 1.18 278.6±3.64µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.21 354.3±2.88µs ? ?/sec 1.00 292.4±4.95µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.16 326.1±0.58µs ? ?/sec 1.00 282.3±1.53µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.04 291.0±2.36µs ? ?/sec 1.00 279.3±1.39µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.20 359.2±5.50µs ? ?/sec 1.00 299.9±2.58µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 980.1±3.50µs ? ?/sec 1.14 1122.1±9.70µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 847.1±2.21µs ? ?/sec 1.14 966.6±2.17µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 985.8±2.30µs ? ?/sec 1.15 1133.0±3.28µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 306.5±4.21µs ? ?/sec 1.46 446.5±4.50µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 478.8±1.42µs ? ?/sec 1.32 633.6±7.02µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 311.6±5.15µs ? ?/sec 1.46 456.1±3.48µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 161.2±0.81µs ? ?/sec 1.26 202.8±0.55µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 303.1±0.74µs ? ?/sec 1.13 343.6±0.47µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 166.7±0.32µs ? ?/sec 1.25 208.1±0.38µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 77.7±0.31µs ? ?/sec 1.52 118.2±0.36µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 260.3±0.57µs ? ?/sec 1.16 300.7±0.87µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 84.1±0.30µs ? ?/sec 1.46 123.2±0.17µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 738.5±1.43µs ? ?/sec 1.00 737.3±2.45µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 580.0±2.20µs ? ?/sec 1.02 591.8±2.08µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.00 743.9±2.69µs ? ?/sec 1.00 743.2±2.36µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.00 64.7±4.61µs ? ?/sec 1.01 65.5±5.32µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.00 245.5±1.68µs ? ?/sec 1.03 252.8±1.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.00 73.1±6.90µs ? ?/sec 1.03 75.4±1.73µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.00 94.5±0.22µs ? ?/sec 1.00 94.6±0.23µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.00 235.9±0.94µs ? ?/sec 1.00 235.0±1.30µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.00 99.7±0.48µs ? ?/sec 1.01 100.2±1.65µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.05 9.8±0.13µs ? ?/sec 1.00 9.3±0.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.00 193.5±0.31µs ? ?/sec 1.00 192.7±0.72µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.04 15.1±0.28µs ? ?/sec 1.00 14.4±0.14µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.00 184.5±0.56µs ? ?/sec 1.00 185.1±0.80µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.00 346.8±0.82µs ? ?/sec 1.00 348.1±2.56µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.00 190.0±0.52µs ? ?/sec 1.00 190.9±0.99µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.02 14.7±0.32µs ? ?/sec 1.00 14.4±0.31µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.00 262.2±0.84µs ? ?/sec 1.00 262.6±1.26µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.00 20.1±0.58µs ? ?/sec 1.02 20.5±0.36µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.00 364.7±1.81µs ? ?/sec 1.01 367.8±1.49µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.00 383.5±1.16µs ? ?/sec 1.01 388.4±1.63µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.00 372.1±0.72µs ? ?/sec 1.01 374.9±1.16µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.00 26.4±0.30µs ? ?/sec 1.05 27.7±0.53µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.00 215.4±0.94µs ? ?/sec 1.01 218.2±1.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.00 30.6±0.46µs ? ?/sec 1.16 35.5±0.52µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 124.8±0.51µs ? ?/sec 1.00 124.3±0.20µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.12 140.0±0.78µs ? ?/sec 1.00 125.1±1.22µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.00 127.4±0.77µs ? ?/sec 1.00 127.4±0.33µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.00 178.7±0.42µs ? ?/sec 1.00 178.3±1.56µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.14 236.1±1.84µs ? ?/sec 1.00 207.6±1.98µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.00 183.6±0.37µs ? ?/sec 1.00 183.9±1.78µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.01 76.3±0.21µs ? ?/sec 1.00 75.5±0.25µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.15 181.2±0.83µs ? ?/sec 1.00 157.3±2.15µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.01 83.0±0.37µs ? ?/sec 1.00 82.1±0.18µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 135.3±0.37µs ? ?/sec 1.06 143.3±2.28µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.11 214.0±0.96µs ? ?/sec 1.00 192.2±2.48µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 141.2±0.32µs ? ?/sec 1.05 148.7±0.34µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 74.7±0.33µs ? ?/sec 1.00 74.6±0.31µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.16 177.7±0.61µs ? ?/sec 1.00 153.6±0.51µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.01 78.6±0.27µs ? ?/sec 1.00 77.9±0.28µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 111.9±0.23µs ? ?/sec 1.02 114.5±0.26µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.00 124.4±0.42µs ? ?/sec 1.07 133.7±0.60µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.02 117.7±2.19µs ? ?/sec 1.00 115.7±0.32µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 169.4±0.63µs ? ?/sec 1.01 170.7±0.31µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 211.4±1.55µs ? ?/sec 1.13 239.2±0.76µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.00 175.8±1.56µs ? ?/sec 1.00 175.9±0.33µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 202.0±0.31µs ? ?/sec 1.00 201.2±0.43µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 225.6±0.64µs ? ?/sec 1.12 252.9±3.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 208.6±1.41µs ? ?/sec 1.00 208.3±0.65µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 144.6±0.53µs ? ?/sec 1.07 154.4±4.00µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 193.6±1.10µs ? ?/sec 1.15 221.9±0.54µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 148.7±0.40µs ? ?/sec 1.05 155.5±1.67µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 107.7±0.78µs ? ?/sec 1.01 108.3±2.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 172.1±1.13µs ? ?/sec 1.16 199.4±1.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.00 115.7±1.55µs ? ?/sec 1.00 115.4±1.66µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.01 102.1±0.21µs ? ?/sec 1.00 101.1±0.22µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.14 117.8±0.30µs ? ?/sec 1.00 103.5±0.31µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.01 105.6±0.24µs ? ?/sec 1.00 104.2±0.58µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.01 139.5±0.29µs ? ?/sec 1.00 137.4±1.31µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.15 194.3±0.71µs ? ?/sec 1.00 168.5±2.16µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.02 144.8±0.33µs ? ?/sec 1.00 142.0±1.28µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 42.5±0.26µs ? ?/sec 1.04 44.2±0.32µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.21 143.2±0.50µs ? ?/sec 1.00 118.0±0.46µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 47.7±0.10µs ? ?/sec 1.03 49.1±0.21µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.00 102.7±0.31µs ? ?/sec 1.07 110.3±0.30µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.14 177.3±0.34µs ? ?/sec 1.00 155.5±0.82µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.00 108.2±0.24µs ? ?/sec 1.07 115.8±0.87µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.01 38.5±0.12µs ? ?/sec 1.00 38.2±0.14µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.22 142.1±0.35µs ? ?/sec 1.00 116.0±0.41µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.01 44.0±0.18µs ? ?/sec 1.00 43.7±0.13µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.01 98.5±0.30µs ? ?/sec 1.00 97.3±0.17µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.15 111.4±0.31µs ? ?/sec 1.00 96.6±0.70µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.01 101.3±0.19µs ? ?/sec 1.00 100.7±0.20µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.02 128.6±0.19µs ? ?/sec 1.00 126.3±0.42µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.16 175.9±1.18µs ? ?/sec 1.00 151.9±0.43µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.00 130.9±0.42µs ? ?/sec 1.00 131.3±0.34µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.05 25.8±0.31µs ? ?/sec 1.00 24.5±0.20µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.25 127.1±0.73µs ? ?/sec 1.00 101.7±0.59µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.04 31.1±0.27µs ? ?/sec 1.00 30.0±0.44µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.00 83.6±0.33µs ? ?/sec 1.09 91.2±0.28µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.13 156.0±0.29µs ? ?/sec 1.00 137.5±0.39µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.00 89.1±0.20µs ? ?/sec 1.09 97.0±0.65µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.02 18.0±0.43µs ? ?/sec 1.00 17.7±0.51µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.27 122.7±0.52µs ? ?/sec 1.00 96.2±0.21µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.01 25.9±0.45µs ? ?/sec 1.00 25.6±0.87µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.00 85.7±0.23µs ? ?/sec 1.00 86.0±0.27µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.00 93.8±0.90µs ? ?/sec 1.13 106.0±0.42µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.00 88.8±0.86µs ? ?/sec 1.00 89.1±0.37µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.00 117.5±1.43µs ? ?/sec 1.00 117.2±0.44µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.00 149.7±1.76µs ? ?/sec 1.17 174.8±0.37µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.02 122.7±0.63µs ? ?/sec 1.00 120.5±0.42µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 149.5±0.77µs ? ?/sec 1.00 148.5±0.50µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.00 170.3±0.59µs ? ?/sec 1.15 196.4±0.60µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.00 154.6±0.44µs ? ?/sec 1.00 154.5±0.89µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.00 90.7±0.39µs ? ?/sec 1.09 98.7±0.52µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 138.0±1.48µs ? ?/sec 1.21 167.0±0.54µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.00 97.0±0.69µs ? ?/sec 1.07 104.1±0.48µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.00 43.0±0.70µs ? ?/sec 1.05 45.2±2.20µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.00 112.4±1.30µs ? ?/sec 1.22 137.3±0.54µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.00 49.1±0.66µs ? ?/sec 1.05 51.5±2.74µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.02 98.9±0.24µs ? ?/sec 1.00 96.9±0.29µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.15 114.0±0.17µs ? ?/sec 1.00 99.3±0.18µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.02 102.1±0.19µs ? ?/sec 1.00 99.9±0.40µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.01 130.5±0.72µs ? ?/sec 1.00 128.9±0.36µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.16 184.9±0.64µs ? ?/sec 1.00 159.8±0.61µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 135.1±1.03µs ? ?/sec 1.00 134.8±0.57µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.06 36.6±0.13µs ? ?/sec 1.00 34.3±0.08µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.21 137.3±0.65µs ? ?/sec 1.00 113.1±0.30µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.00 41.6±0.11µs ? ?/sec 1.00 41.6±0.10µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.00 95.3±0.29µs ? ?/sec 1.07 102.4±0.89µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.14 170.2±0.84µs ? ?/sec 1.00 148.7±0.42µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.00 101.1±0.28µs ? ?/sec 1.07 107.8±0.28µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.00 30.5±0.13µs ? ?/sec 1.01 30.7±0.17µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.23 134.3±0.75µs ? ?/sec 1.00 109.6±0.50µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.00 36.1±0.13µs ? ?/sec 1.00 36.1±0.18µs ? ?/sec
arrow_array_reader/ListArray/plain encoded optional strings half NULLs 1.00 7.0±0.02ms ? ?/sec 1.02 7.1±0.04ms ? ?/sec
arrow_array_reader/ListArray/plain encoded optional strings no NULLs 1.00 12.8±0.09ms ? ?/sec 1.04 13.3±0.18ms ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.05 506.2±3.03µs ? ?/sec 1.00 483.7±4.49µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.00 657.7±2.06µs ? ?/sec 1.03 676.4±6.01µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.01 505.5±4.10µs ? ?/sec 1.00 498.2±2.68µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.07 735.7±5.14µs ? ?/sec 1.00 686.6±3.75µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.02 800.6±3.27µs ? ?/sec 1.00 784.2±3.19µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.06 741.7±4.61µs ? ?/sec 1.00 699.1±3.62µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.02 301.0±1.47µs ? ?/sec 1.00 296.4±1.47µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.07 385.6±6.15µs ? ?/sec 1.00 358.7±5.47µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.01 306.8±4.70µs ? ?/sec 1.00 302.4±1.34µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 229.3±4.89µs ? ?/sec 1.20 274.1±2.95µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.00 214.2±0.62µs ? ?/sec 1.24 266.2±1.71µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 233.3±2.41µs ? ?/sec 1.20 279.1±2.72µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.00 462.7±6.19µs ? ?/sec 1.04 479.3±1.83µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.00 337.0±1.23µs ? ?/sec 1.10 371.2±1.76µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.00 470.2±2.72µs ? ?/sec 1.04 488.4±1.63µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.00 109.6±2.85µs ? ?/sec 1.07 116.8±0.24µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.09 121.1±0.33µs ? ?/sec 1.00 111.2±0.32µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 112.6±0.38µs ? ?/sec 1.06 119.8±0.53µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.00 143.9±0.74µs ? ?/sec 1.09 156.3±0.51µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.12 198.0±2.59µs ? ?/sec 1.00 177.4±0.38µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 148.1±0.41µs ? ?/sec 1.09 161.7±0.49µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 42.5±0.39µs ? ?/sec 1.04 44.1±0.20µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.23 146.9±2.60µs ? ?/sec 1.00 119.2±0.27µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 47.6±0.14µs ? ?/sec 1.03 49.0±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.00 103.0±0.21µs ? ?/sec 1.07 110.1±0.24µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.14 176.9±0.47µs ? ?/sec 1.00 154.6±0.30µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.00 107.9±0.35µs ? ?/sec 1.07 115.7±0.38µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.01 38.6±0.17µs ? ?/sec 1.00 38.2±0.10µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.22 141.8±0.45µs ? ?/sec 1.00 116.3±0.36µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.00 43.8±0.12µs ? ?/sec 1.00 43.8±0.14µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.02 100.9±0.22µs ? ?/sec 1.00 98.5±0.32µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.15 112.6±0.34µs ? ?/sec 1.00 97.5±0.31µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.03 104.4±0.26µs ? ?/sec 1.00 101.4±0.34µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.01 128.7±0.36µs ? ?/sec 1.00 127.4±0.30µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.19 181.8±0.85µs ? ?/sec 1.00 152.7±1.58µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.01 133.8±0.46µs ? ?/sec 1.00 132.7±1.95µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.08 27.0±0.40µs ? ?/sec 1.00 25.0±0.49µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.27 125.3±0.33µs ? ?/sec 1.00 98.6±0.30µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.06 31.8±0.35µs ? ?/sec 1.00 30.1±0.40µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.00 86.3±0.39µs ? ?/sec 1.07 92.7±0.38µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.16 159.3±0.86µs ? ?/sec 1.00 136.9±0.42µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.00 91.0±0.38µs ? ?/sec 1.08 98.2±0.35µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.00 21.0±0.58µs ? ?/sec 1.01 21.2±0.69µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.27 123.4±0.41µs ? ?/sec 1.00 96.9±0.44µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.00 26.8±0.83µs ? ?/sec 1.02 27.2±0.92µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.00 85.8±0.28µs ? ?/sec 1.00 86.1±0.25µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.00 93.8±0.22µs ? ?/sec 1.13 105.9±0.35µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.00 89.0±0.32µs ? ?/sec 1.00 88.7±0.27µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 116.6±0.51µs ? ?/sec 1.02 118.9±0.48µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.00 158.1±0.82µs ? ?/sec 1.15 181.9±0.65µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.00 122.8±0.53µs ? ?/sec 1.00 123.2±0.55µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 149.6±0.69µs ? ?/sec 1.00 148.1±1.10µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.00 169.8±1.23µs ? ?/sec 1.15 195.9±1.84µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.01 155.7±0.46µs ? ?/sec 1.00 153.8±0.51µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.00 91.4±0.37µs ? ?/sec 1.07 98.0±0.51µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.00 138.7±1.24µs ? ?/sec 1.20 166.6±0.54µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.00 96.7±0.42µs ? ?/sec 1.07 103.8±0.87µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.00 41.6±0.89µs ? ?/sec 1.11 46.4±1.77µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.00 113.8±0.33µs ? ?/sec 1.22 138.6±0.71µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.00 48.4±0.62µs ? ?/sec 1.12 54.4±2.27µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.04 106.1±0.91µs ? ?/sec 1.00 102.3±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.15 118.0±1.14µs ? ?/sec 1.00 102.2±0.32µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.04 109.4±0.92µs ? ?/sec 1.00 105.3±0.27µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.01 137.6±1.41µs ? ?/sec 1.00 136.7±0.36µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.16 189.3±1.94µs ? ?/sec 1.00 163.6±0.32µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.00 142.7±1.37µs ? ?/sec 1.00 142.2±1.46µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 36.5±0.31µs ? ?/sec 1.00 36.3±0.11µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.22 135.2±0.37µs ? ?/sec 1.00 111.1±0.41µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.00 41.3±0.32µs ? ?/sec 1.00 41.3±0.12µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.00 95.2±0.83µs ? ?/sec 1.07 101.8±0.25µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.15 169.5±0.50µs ? ?/sec 1.00 147.0±0.40µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.00 100.7±0.21µs ? ?/sec 1.07 107.8±0.47µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.02 30.8±0.10µs ? ?/sec 1.00 30.2±0.12µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.23 134.4±0.41µs ? ?/sec 1.00 109.0±0.37µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.00 36.2±0.12µs ? ?/sec 1.00 36.3±0.16µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2511187329
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -20,19 +20,70 @@
use crate::arrow::array_reader::ArrayReader;
use crate::arrow::arrow_reader::{
- ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
+ ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelectionCursor,
+ RowSelectionStrategy, RowSelector,
};
use crate::errors::{ParquetError, Result};
use arrow_array::Array;
use arrow_select::filter::prep_null_mask_filter;
-use std::collections::VecDeque;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+// The average selector length threshold for choosing between
+// `RowSelectionStrategy::Mask` and `RowSelectionStrategy::Selectors`.
+// If the average selector length is less than this value,
+// `RowSelectionStrategy::Mask` is preferred.
+const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 32;
+
+// The logic in `preferred_selection_strategy` depends on the constant
+// `AVG_SELECTOR_LEN_MASK_THRESHOLD`. To allow unit testing of this logic,
+// we use a mutable global variable that can be temporarily changed during tests.
+//
+// An `AtomicUsize` is used because the Rust test runner (`cargo test`) runs tests
+// in parallel by default. The atomic operations prevent data races between
+// different test threads that might try to modify this value simultaneously.
+//
+// For the production code path, `load(Ordering::Relaxed)` is used. This is the
+// weakest memory ordering and for a simple load on most modern architectures,
+// it compiles down to a regular memory read with negligible performance overhead.
+// The more expensive atomic operations with stronger ordering are only used in the
+// test-only functions below.
+static AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE: AtomicUsize =
+ AtomicUsize::new(AVG_SELECTOR_LEN_MASK_THRESHOLD);
+
+#[inline(always)]
+fn avg_selector_len_mask_threshold() -> usize {
+ AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.load(Ordering::Relaxed)
+}
+
+/// An RAII guard that restores the previous value of the override when it is dropped.
+/// This ensures that any change to the global threshold is temporary and scoped to
+/// the test or benchmark where it's used, even in the case of a panic.
+pub struct AvgSelectorLenMaskThresholdGuard {
+ previous: usize,
+}
+
+impl Drop for AvgSelectorLenMaskThresholdGuard {
+ fn drop(&mut self) {
+ AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.store(self.previous, Ordering::SeqCst);
+ }
+}
+
+/// Override AVG_SELECTOR_LEN_MASK_THRESHOLD (primarily for tests / benchmarks).
+///
+/// Returns an [`AvgSelectorLenMaskThresholdGuard`] that restores the previous value on drop.
+pub fn set_avg_selector_len_mask_threshold(value: usize) -> AvgSelectorLenMaskThresholdGuard {
+ let previous = AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.swap(value, Ordering::SeqCst);
+ AvgSelectorLenMaskThresholdGuard { previous }
+}
/// A builder for [`ReadPlan`]
#[derive(Clone, Debug)]
pub struct ReadPlanBuilder {
batch_size: usize,
/// Current to apply, includes all filters
selection: Option<RowSelection>,
+ /// Strategy to use when materialising the row selection
+ selection_strategy: RowSelectionStrategy,
Review Comment:
The more I have thought about this, the more I am thinking that maybe combining `RowSelectionStrategy` and `RowSelection` somehow would make a lot of sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3459994262
@codex Help me to review the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2486603258
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
Thank you so much @hhhizzz -- this is super exciting and I will give top priority to reviewing this PR as you make changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3542019551
> @alamb @zhuqi-lucas @tustvold Thank you — **I know this might be a bit off the main topic, but I still want to express my appreciation.** Recently my life has been in a pretty rough state, and I’m currently between jobs. But contributing here has given me a really valuable opportunity to stretch my programming skills. I’ve learned a lot through the process, and I’m grateful for that.
>
> We still have a few follow-up tickets related to this improvement, and I’ll continue exploring them. Thanks again for your patience and support.
It has been my pleasure reviewing your code and interacting with you in the context of this PR. It is a really nice contribution and I think (hopefully) has finally broken us through the barrier needed to get this feature rolled out to everyone. 🙏
I look forward to continue working together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3467451997
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing rowselectionempty (14647e18080a1823ee74da79b35a1095cfcdc885) to 57447434d1921701456543f9dfb92741e5d86734 [diff](https://github.com/apache/arrow-rs/compare/57447434d1921701456543f9dfb92741e5d86734..14647e18080a1823ee74da79b35a1095cfcdc885)
BENCH_NAME=arrow_reader_row_filter
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader_row_filter
BENCH_FILTER=
BENCH_BRANCH_NAME=rowselectionempty
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2480294162
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
Thanks for the thorough code review!
Yes—this is the trickiest part of the PR. When no pages are skipped, everything works as expected. But some pages can be skipped during row-group construction, use the Sparse `ColumnChunkData`, meaning their values and definition/repetition levels are never read. Row selection still works because `skip_records()` handles this case and skips the page accordingly.
However, with the Boolean-array design, all values must be read and decoded before filtering. `ParquetRecordBatchReader` is a streaming reader; it has no concept of pages, so we can’t rely on page size to drive skipping there. I think the most practical approach, therefore, is to return dummy null values as placeholders for the skipped pages. If I missed something or there's better way to do so, just let me know. 😊
## A simple example:
the page size is 2, the mask is 100001, row selection should be read(1) skip(4) read(1)
the ColumnChunkData would be page1(10), page2(skipped), page3(01)
Using the `rowselection` to skip(4), the page2 won't be read at all.
But using the bit mask, we need all 6 value be read, but the page2 is not in the memory, which is why I need to construct this synthetic page.
---
For completeness, I prototyped reconstructing the readers to handle skipped pages directly, but it introduces a breaking change: every array_reader would need a page-size parameter. That’s undesirable—users shouldn’t need page-level details just to read Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483701565
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
Review Comment:
This will likely lead to OOMs for any non-trivial parquet file - it will fetch the entire selector (i.e. the entire row group) into memory and then filter it
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
I think this will cause the reader to produce batches potentially much smaller than the requested batch size, which at the very least should be documented. Ideally we'd just store the mask somewhere, and flush + filter when we have enough rows.
Otherwise consumers like DataFusion are going to need to wrap ParquetExec in a CoalesceBatches and effectively buffer all the data again.
~Edit edit: Filtering small batches may also have performance implications - the filter kernels are optimized for filtering thousands of rows at a time.~
Edit edit edit: actually realised the selector is for the entire row group!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504566235
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
I've totally removed the synthetic page. For the new design we can continue discuss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2505421194
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -20,19 +20,70 @@
use crate::arrow::array_reader::ArrayReader;
use crate::arrow::arrow_reader::{
- ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
+ ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelectionCursor,
+ RowSelectionStrategy, RowSelector,
};
use crate::errors::{ParquetError, Result};
use arrow_array::Array;
use arrow_select::filter::prep_null_mask_filter;
-use std::collections::VecDeque;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+// The average selector length threshold for choosing between
+// `RowSelectionStrategy::Mask` and `RowSelectionStrategy::Selectors`.
+// If the average selector length is less than this value,
+// `RowSelectionStrategy::Mask` is preferred.
+const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 32;
+
+// The logic in `preferred_selection_strategy` depends on the constant
+// `AVG_SELECTOR_LEN_MASK_THRESHOLD`. To allow unit testing of this logic,
+// we use a mutable global variable that can be temporarily changed during tests.
+//
+// An `AtomicUsize` is used because the Rust test runner (`cargo test`) runs tests
+// in parallel by default. The atomic operations prevent data races between
+// different test threads that might try to modify this value simultaneously.
+//
+// For the production code path, `load(Ordering::Relaxed)` is used. This is the
+// weakest memory ordering and for a simple load on most modern architectures,
+// it compiles down to a regular memory read with negligible performance overhead.
+// The more expensive atomic operations with stronger ordering are only used in the
+// test-only functions below.
+static AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE: AtomicUsize =
Review Comment:
I am a little worried about this mechanism for testing -- I understand its appeal, but it might be better to avoid statics entirely and thread through the threshold value as an option on the selection strategy directly
For example
```rust
enum RowSelectionStrategy {
/// automatically pick the filter representation
/// based on heuristics
Auto,
/// If the average number of rows is selected is more than
/// the threshold, uses the Mask policy, otherwise uses Selectors
Threshold {
threshold: usize
},
Selection,
Mask
}
```
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -691,6 +736,180 @@ fn union_row_selections(left: &[RowSelector], right: &[RowSelector]) -> RowSelec
iter.collect()
}
+/// Cursor for iterating a [`RowSelection`] during execution within a
+/// [`ReadPlan`](crate::arrow::arrow_reader::ReadPlan).
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to the internal `RowSelectionBacking`.
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+pub enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+#[derive(Debug)]
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ pub(crate) fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ let storage = match strategy {
+ RowSelectionStrategy::Mask => {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ }
+ RowSelectionStrategy::Selectors => RowSelectionBacking::Selectors(selectors.into()),
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
Review Comment:
this always returns `Some` so we could change the signature to always return `RowSelector` rather than `Option<RowSelector>`
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -20,19 +20,70 @@
use crate::arrow::array_reader::ArrayReader;
use crate::arrow::arrow_reader::{
- ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
+ ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelectionCursor,
+ RowSelectionStrategy, RowSelector,
};
use crate::errors::{ParquetError, Result};
use arrow_array::Array;
use arrow_select::filter::prep_null_mask_filter;
-use std::collections::VecDeque;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+// The average selector length threshold for choosing between
+// `RowSelectionStrategy::Mask` and `RowSelectionStrategy::Selectors`.
+// If the average selector length is less than this value,
+// `RowSelectionStrategy::Mask` is preferred.
+const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 32;
+
+// The logic in `preferred_selection_strategy` depends on the constant
+// `AVG_SELECTOR_LEN_MASK_THRESHOLD`. To allow unit testing of this logic,
+// we use a mutable global variable that can be temporarily changed during tests.
+//
+// An `AtomicUsize` is used because the Rust test runner (`cargo test`) runs tests
+// in parallel by default. The atomic operations prevent data races between
+// different test threads that might try to modify this value simultaneously.
+//
+// For the production code path, `load(Ordering::Relaxed)` is used. This is the
+// weakest memory ordering and for a simple load on most modern architectures,
+// it compiles down to a regular memory read with negligible performance overhead.
+// The more expensive atomic operations with stronger ordering are only used in the
+// test-only functions below.
+static AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE: AtomicUsize =
+ AtomicUsize::new(AVG_SELECTOR_LEN_MASK_THRESHOLD);
+
+#[inline(always)]
+fn avg_selector_len_mask_threshold() -> usize {
+ AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.load(Ordering::Relaxed)
+}
+
+/// An RAII guard that restores the previous value of the override when it is dropped.
+/// This ensures that any change to the global threshold is temporary and scoped to
+/// the test or benchmark where it's used, even in the case of a panic.
+pub struct AvgSelectorLenMaskThresholdGuard {
+ previous: usize,
+}
+
+impl Drop for AvgSelectorLenMaskThresholdGuard {
+ fn drop(&mut self) {
+ AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.store(self.previous, Ordering::SeqCst);
+ }
+}
+
+/// Override AVG_SELECTOR_LEN_MASK_THRESHOLD (primarily for tests / benchmarks).
+///
+/// Returns an [`AvgSelectorLenMaskThresholdGuard`] that restores the previous value on drop.
+pub fn set_avg_selector_len_mask_threshold(value: usize) -> AvgSelectorLenMaskThresholdGuard {
+ let previous = AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.swap(value, Ordering::SeqCst);
+ AvgSelectorLenMaskThresholdGuard { previous }
+}
/// A builder for [`ReadPlan`]
#[derive(Clone, Debug)]
pub struct ReadPlanBuilder {
batch_size: usize,
/// Current to apply, includes all filters
selection: Option<RowSelection>,
+ /// Strategy to use when materialising the row selection
+ selection_strategy: RowSelectionStrategy,
Review Comment:
I like this formulation as it makes this PR logic quite clear.
Also, As a follow on PR we could potentially combine this logic so the ReadPlanBuilder holds a `Option<RowSelectionCursor>` rather than a `selection` and selection_strategy`
This might allow us to both
1. avoid converting BooleanArray --> RowSelection and then back again
2. implement the page filtering in [`InMemoryRowGroup::fetch_ranges`](https://github.com/apache/arrow-rs/blob/e9ea12b138f8fa11c6046e94cb2b55b2830a0d7c/parquet/src/arrow/in_memory_row_group.rs#L58-L57) in terms of masked selections as well
However, we can totally do this as a follow on PR -- I will file a ticket when we merge this PR
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -16,12 +16,24 @@
// under the License.
use arrow_array::{Array, BooleanArray};
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder};
use arrow_select::filter::SlicesIterator;
use std::cmp::Ordering;
use std::collections::VecDeque;
use std::ops::Range;
-use crate::file::page_index::offset_index::PageLocation;
+use crate::arrow::ProjectionMask;
+use crate::file::page_index::offset_index::{OffsetIndexMetaData, PageLocation};
+
+/// Strategy for materialising [`RowSelection`] during execution.
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+pub enum RowSelectionStrategy {
Review Comment:
I liked the idea previously of having an Auto mode here based on heuristics that we could change over time.
I can help propose an upate to this API if you are amenable to this idea
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -213,6 +225,39 @@ impl RowSelection {
ranges
}
+ /// Returns true if this selection would skip any data pages within the provided columns
+ fn selection_skips_any_page(
+ &self,
+ projection: &ProjectionMask,
+ columns: &[OffsetIndexMetaData],
+ ) -> bool {
+ columns.iter().enumerate().any(|(leaf_idx, column)| {
+ if !projection.leaf_included(leaf_idx) {
+ return false;
+ }
+
+ let locations = column.page_locations();
+ if locations.is_empty() {
+ return false;
+ }
+
+ let ranges = self.scan_ranges(locations);
+ !ranges.is_empty() && ranges.len() < locations.len()
+ })
+ }
+
+ /// Returns true if selectors should be forced, preventing mask materialisation
+ pub(crate) fn should_force_selectors(
+ &self,
+ projection: &ProjectionMask,
+ offset_index: Option<&[OffsetIndexMetaData]>,
+ ) -> bool {
+ match offset_index {
+ Some(columns) => self.selection_skips_any_page(projection, columns),
Review Comment:
This is a nice solution for the initial PR -- I think we can file a follow on ticket to figure out how to optimize this further in follow on PRs.
I think given my reading above, we can probably move move this logic into InMemoryRowGroup::fetch and then actually decide what to do with pages as necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2528093842
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
+ // Prefer a bitmap mask when the selectors are short on average, as the mask
+ // (re)construction cost is amortized by a simpler execution path during reads.
+ let use_mask = match strategy {
+ RowSelectionStrategy::Mask => true,
+ RowSelectionStrategy::Auto => {
+ selector_count == 0
+ || total_rows < selector_count.saturating_mul(AVG_SELECTOR_LEN_MASK_THRESHOLD)
+ }
+ RowSelectionStrategy::Selectors => unreachable!(),
+ };
+
+ let storage = if use_mask {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ } else {
+ RowSelectionBacking::Selectors(selectors.into())
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
+ }
+ RowSelectionBacking::Mask(_) => None,
+ }
+ }
+
+ /// Return a selector to the front, rewinding the position (sparse-only)
+ pub fn return_selector(&mut self, selector: RowSelector) {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ self.position = self.position.saturating_sub(selector.row_count);
+ selectors.push_front(selector);
+ }
+ RowSelectionBacking::Mask(_) => {
+ unreachable!("return_selector called for mask-based RowSelectionCursor")
+ }
+ }
+ }
+
+ /// Returns `true` if the cursor is backed by a boolean mask
+ pub fn is_mask_backed(&self) -> bool {
+ matches!(self.storage, RowSelectionBacking::Mask(_))
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = match &self.storage {
+ RowSelectionBacking::Mask(mask) => mask,
+ RowSelectionBacking::Selectors(_) => return None,
+ };
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
+ initial_skip += 1;
+ cursor += 1;
+ }
+
+ let mask_start = cursor;
+ let mut chunk_rows = 0;
+ let mut selected_rows = 0;
+
+ // Advance until enough rows have been selected to satisfy the batch size,
+ // or until the mask is exhausted. This mirrors the behaviour of the legacy
+ // `RowSelector` queue-based iteration.
+ while cursor < mask.len() && selected_rows < batch_size {
+ chunk_rows += 1;
+ if mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ (initial_skip, chunk_rows, selected_rows, mask_start, cursor)
+ };
+
+ self.position = end_position;
+
+ Some(MaskChunk {
+ initial_skip,
+ chunk_rows,
+ selected_rows,
+ mask_start,
+ })
+ }
+
+ /// Materialise the boolean values for a mask-backed chunk
+ pub fn mask_values_for(&self, chunk: &MaskChunk) -> Option<BooleanArray> {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => {
+ if chunk.mask_start.saturating_add(chunk.chunk_rows) > mask.len() {
+ return None;
+ }
+ Some(BooleanArray::from(
+ mask.slice(chunk.mask_start, chunk.chunk_rows),
+ ))
+ }
+ RowSelectionBacking::Selectors(_) => None,
+ }
+ }
+}
+
+fn boolean_mask_from_selectors(selectors: &[RowSelector]) -> BooleanBuffer {
Review Comment:
- Tracking ticket: https://github.com/apache/arrow-rs/issues/8844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2480467292
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let avg_selector_lengths: &[usize] = &[16, 20, 24, 28, 32, 36, 40];
+ let parquet_data = build_parquet_data(TOTAL_ROWS);
+
+ let scenarios = [
+ Scenario {
+ name: "uniform50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Constant,
+ },
+ Scenario {
+ name: "spread50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Uniform { spread: 0.9 },
+ },
+ Scenario {
+ name: "sparse20",
+ select_ratio: 0.2,
+ start_with_select: false,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 6.0,
+ long_prob: 0.1,
+ },
+ },
+ Scenario {
+ name: "dense80",
+ select_ratio: 0.8,
+ start_with_select: true,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 4.0,
+ long_prob: 0.05,
+ },
+ },
+ ];
+
+ for scenario in scenarios.iter() {
+ for (offset, &avg_len) in avg_selector_lengths.iter().enumerate() {
+ let selectors =
+ generate_selectors(avg_len, TOTAL_ROWS, scenario, BASE_SEED + offset as u64);
+ let stats = SelectorStats::new(&selectors);
+ let suffix = format!(
+ "{}-avg{:.1}-sel{:02}",
+ scenario.name,
+ stats.average_selector_len,
+ (stats.select_ratio * 100.0).round() as u32
+ );
+
+ let bench_input = BenchInput {
+ parquet_data: parquet_data.clone(),
+ selection: RowSelection::from(selectors.clone()),
+ };
+
+ for (label, strategy) in READ_STRATEGIES.iter().copied() {
+ c.bench_with_input(
+ BenchmarkId::new(label, &suffix),
+ &bench_input,
+ |b, input| {
+ b.iter(|| {
+ let total = run_read(&input.parquet_data, &input.selection, strategy);
+ hint::black_box(total);
+ });
+ },
+ );
+ }
+ }
+ }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
+
+struct BenchInput {
+ parquet_data: Bytes,
+ selection: RowSelection,
+}
+
+fn run_read(
+ parquet_data: &Bytes,
+ selection: &RowSelection,
+ strategy: RowSelectionStrategy,
+) -> usize {
+ let reader = ParquetRecordBatchReaderBuilder::try_new(parquet_data.clone())
+ .unwrap()
+ .with_batch_size(BATCH_SIZE)
+ .with_row_selection(selection.clone())
+ .with_row_selection_strategy(strategy)
+ .build()
+ .unwrap();
+
+ let mut total_rows = 0usize;
+ for batch in reader {
+ let batch = batch.unwrap();
+ total_rows += batch.num_rows();
+ }
+ total_rows
+}
+
+fn build_parquet_data(total_rows: usize) -> Bytes {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "value",
+ DataType::Int32,
+ false,
+ )]));
+ let values = Int32Array::from_iter_values((0..total_rows).map(|v| v as i32));
+ let columns: Vec<ArrayRef> = vec![Arc::new(values) as ArrayRef];
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let mut writer = ArrowWriter::try_new(Vec::new(), schema, None).unwrap();
+ writer.write(&batch).unwrap();
+ let buffer = writer.into_inner().unwrap();
+ Bytes::from(buffer)
+}
+
+#[derive(Clone)]
+struct Scenario {
+ name: &'static str,
+ select_ratio: f64,
+ start_with_select: bool,
+ distribution: RunDistribution,
+}
+
+#[derive(Clone)]
+enum RunDistribution {
+ Constant,
+ Uniform { spread: f64 },
+ Bimodal { long_factor: f64, long_prob: f64 },
+}
+
+fn generate_selectors(
+ avg_selector_len: usize,
+ total_rows: usize,
+ scenario: &Scenario,
+ seed: u64,
+) -> Vec<RowSelector> {
+ assert!(
+ (0.0..=1.0).contains(&scenario.select_ratio),
+ "select_ratio must be in [0, 1]"
+ );
+
+ let mut select_mean = scenario.select_ratio * 2.0 * avg_selector_len as f64;
+ let mut skip_mean = (1.0 - scenario.select_ratio) * 2.0 * avg_selector_len as f64;
+
+ select_mean = select_mean.max(1.0);
+ skip_mean = skip_mean.max(1.0);
+
+ let sum = select_mean + skip_mean;
+ // Rebalance the sampled select/skip run lengths so their sum matches the requested
+ // average selector length while respecting the configured selectivity ratio.
+ let scale = if sum == 0.0 {
+ 1.0
+ } else {
+ (2.0 * avg_selector_len as f64) / sum
+ };
+ select_mean *= scale;
+ skip_mean *= scale;
+
+ let mut rng = StdRng::seed_from_u64(seed ^ (avg_selector_len as u64).wrapping_mul(0x9E3779B1));
+ let mut selectors = Vec::with_capacity(total_rows / avg_selector_len.max(1));
+ let mut remaining = total_rows;
+ let mut is_select = scenario.start_with_select;
+
+ while remaining > 0 {
+ let mean = if is_select { select_mean } else { skip_mean };
+ let len = sample_length(mean, &scenario.distribution, &mut rng).max(1);
+ let len = len.min(remaining);
+ selectors.push(if is_select {
+ RowSelector::select(len)
+ } else {
+ RowSelector::skip(len)
+ });
+ remaining -= len;
+ if remaining == 0 {
+ break;
+ }
+ is_select = !is_select;
+ }
+
+ let selection: RowSelection = selectors.into();
+ selection.into()
+}
+
+fn sample_length(mean: f64, distribution: &RunDistribution, rng: &mut StdRng) -> usize {
+ match distribution {
+ RunDistribution::Constant => mean.round().max(1.0) as usize,
+ RunDistribution::Uniform { spread } => {
+ let spread = spread.clamp(0.0, 0.99);
+ let lower = (mean * (1.0 - spread)).max(1.0);
+ let upper = (mean * (1.0 + spread)).max(lower + f64::EPSILON);
+ if (upper - lower) < 1.0 {
+ lower.round().max(1.0) as usize
+ } else {
+ let low = lower.floor() as usize;
+ let high = upper.ceil() as usize;
+ rng.random_range(low..=high).max(1)
+ }
+ }
+ RunDistribution::Bimodal {
+ long_factor,
+ long_prob,
+ } => {
+ let long_prob = long_prob.clamp(0.0, 0.5);
+ let short_prob = 1.0 - long_prob;
+ let short_factor = if short_prob == 0.0 {
+ 1.0 / long_factor.max(f64::EPSILON)
+ } else {
+ (1.0 - long_prob * long_factor).max(0.0) / short_prob
+ };
+ let use_long = rng.random_bool(long_prob);
+ let factor = if use_long {
+ *long_factor
+ } else {
+ short_factor.max(0.1)
+ };
+ (mean * factor).round().max(1.0) as usize
+ }
+ }
+}
+
+struct SelectorStats {
Review Comment:
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483677316
##########
parquet/src/column/reader.rs:
##########
@@ -403,7 +465,27 @@ where
/// Returns false if there's no page left.
fn read_new_page(&mut self) -> Result<bool> {
loop {
- match self.page_reader.get_next_page()? {
+ let page_result = match self.page_reader.get_next_page() {
+ Ok(page) => page,
+ Err(err) => {
+ return match err {
+ ParquetError::General(message)
+ if message
+ .starts_with("Invalid offset in sparse column chunk data:") =>
+ {
+ let metadata = self.page_reader.peek_next_page()?;
+ // Some writers omit data pages for sparse column chunks and encode the gap
+ // as a reader-visible error. Use the metadata peek to synthesise a page of
+ // null definition levels so downstream consumers see consistent row counts.
+ self.try_create_synthetic_page(metadata)?;
Review Comment:
Additionally I think it would imply that the predicate pushdown is "reversing" earlier forms of pushdown and relying on the IO implementation to have chosen to do a sparse read - this feels unfortunate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2511608078
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -233,12 +283,12 @@ pub struct ReadPlan {
/// The number of rows to read in each batch
batch_size: usize,
/// Row ranges to be selected from the data source
- selection: Option<VecDeque<RowSelector>>,
+ selection: Option<RowSelectionCursor>,
}
impl ReadPlan {
/// Returns a mutable reference to the selection, if any
- pub fn selection_mut(&mut self) -> Option<&mut VecDeque<RowSelector>> {
+ pub fn selection_mut(&mut self) -> Option<&mut RowSelectionCursor> {
Review Comment:
🤔 it seems like this is a public API (so we can't change its signature)
https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ReadPlan.html#method.selection_mut
We can get around this by making a function like
```rust
pub fn row_selection_cursor_mut(&mut self) -> Option<&mut RowSelectionCursor> {
```
And then
```rust
#[deprecated(since = "57.1.0", note = "Use `row_selection_cursor_mut` instead")]
pub fn selection_mut(&mut self) -> Option<&mut VecDeque<RowSelector>> {
...
```
I am still playing around to see if I can make the RowSelectionCursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3467528186
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main rowselectionempty
----- ---- -----------------
arrow_reader_row_filter/float64 <= 99.0/all_columns/async 1.00 1720.5±11.69µs ? ?/sec 1.01 1739.0±13.11µs ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/all_columns/sync 1.00 2.0±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/async 1.00 1557.1±7.68µs ? ?/sec 1.02 1586.2±7.77µs ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/sync 1.00 1656.7±8.38µs ? ?/sec 1.01 1672.2±14.29µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/async 1.00 1519.3±8.50µs ? ?/sec 1.00 1524.9±11.77µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/sync 1.00 1870.1±14.00µs ? ?/sec 1.00 1860.9±12.07µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/async 1.00 1350.6±4.84µs ? ?/sec 1.00 1356.5±10.51µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/sync 1.00 1450.9±6.97µs ? ?/sec 1.01 1467.6±12.90µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/async 1.00 1693.8±7.84µs ? ?/sec 1.03 1745.4±12.91µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/sync 1.00 1976.9±14.72µs ? ?/sec 1.01 2.0±0.02ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/async 1.00 1550.6±5.40µs ? ?/sec 1.02 1583.6±10.16µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/sync 1.00 1634.7±18.11µs ? ?/sec 1.02 1663.9±11.89µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/async 1.00 935.4±4.28µs ? ?/sec 1.01 945.1±5.97µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/sync 1.00 988.9±5.01µs ? ?/sec 1.00 993.7±9.68µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/async 1.00 865.3±3.69µs ? ?/sec 1.01 870.3±9.16µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/sync 1.01 984.7±4.82µs ? ?/sec 1.00 978.6±6.66µs ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/async 1.47 4.1±0.02ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/sync 1.57 4.1±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/async 1.34 3.6±0.01ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/sync 1.43 3.5±0.01ms ? ?/sec 1.00 2.4±0.03ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/async 1.00 1917.4±8.96µs ? ?/sec 1.01 1945.3±16.00µs ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/sync 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/async 1.00 1752.7±8.89µs ? ?/sec 1.00 1756.5±10.41µs ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/sync 1.00 1885.0±8.43µs ? ?/sec 1.00 1888.4±14.49µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/async 1.01 1254.9±4.74µs ? ?/sec 1.00 1243.6±11.57µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/sync 1.00 1383.5±5.02µs ? ?/sec 1.00 1387.9±10.40µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/async 1.00 1134.1±10.68µs ? ?/sec 1.00 1134.0±7.09µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/sync 1.00 1264.6±7.34µs ? ?/sec 1.00 1261.6±9.84µs ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/async 1.32 4.2±0.02ms ? ?/sec 1.00 3.2±0.48ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/sync 1.35 4.9±0.02ms ? ?/sec 1.00 3.6±0.06ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/async 1.35 3.5±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/sync 1.37 3.4±0.01ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504572156
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
Review Comment:
https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504574820
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
Review Comment:
Updated the comment for more comparison and the scripts to generate the charts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504578882
##########
parquet/src/column/reader.rs:
##########
@@ -541,6 +626,44 @@ where
}
}
+ fn try_create_synthetic_page(&mut self, metadata: Option<PageMetadata>) -> Result<()> {
+ if self.descr.max_rep_level() != 0 {
+ return Err(general_err!(
+ "cannot synthesise sparse page for column with repetition levels ({message})"
+ ));
+ }
+
+ if self.descr.max_def_level() == 0 {
+ return Err(general_err!(
+ "cannot synthesise sparse page for required column ({message})"
+ ));
+ }
Review Comment:
Removed the code for this synthetic page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3505287719
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main rowselectionempty
----- ---- -----------------
arrow_reader_row_filter/float64 <= 99.0/all_columns/async 1.00 1699.3±14.65µs ? ?/sec 1.02 1738.5±6.85µs ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/all_columns/sync 1.00 1821.0±15.25µs ? ?/sec 1.04 1898.2±19.11µs ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/async 1.00 1544.0±15.65µs ? ?/sec 1.03 1596.7±20.29µs ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/sync 1.00 1533.1±12.36µs ? ?/sec 1.03 1577.1±10.47µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/async 1.00 1499.6±6.78µs ? ?/sec 1.03 1540.3±12.13µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/sync 1.00 1667.4±15.52µs ? ?/sec 1.04 1738.0±11.34µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/async 1.00 1309.9±6.34µs ? ?/sec 1.05 1380.8±5.28µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/sync 1.00 1334.8±6.96µs ? ?/sec 1.04 1394.3±7.96µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/async 1.00 1690.8±10.69µs ? ?/sec 1.02 1723.3±7.82µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/sync 1.00 1808.9±10.91µs ? ?/sec 1.02 1852.9±9.83µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/async 1.00 1549.3±14.02µs ? ?/sec 1.02 1579.4±7.61µs ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/sync 1.00 1537.5±8.09µs ? ?/sec 1.02 1562.4±10.80µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/async 1.00 900.4±5.92µs ? ?/sec 1.02 919.7±4.78µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/sync 1.00 849.2±4.31µs ? ?/sec 1.05 890.1±10.01µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/async 1.00 822.2±2.84µs ? ?/sec 1.05 866.4±11.77µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/sync 1.00 848.1±4.90µs ? ?/sec 1.03 872.6±7.16µs ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/async 1.47 4.0±0.01ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/sync 1.07 3.9±0.02ms ? ?/sec 1.00 3.6±0.06ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/async 1.34 3.5±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/sync 1.44 3.4±0.02ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/async 1.00 1898.4±10.93µs ? ?/sec 1.03 1962.0±7.79µs ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/sync 1.00 2.0±0.01ms ? ?/sec 1.03 2.1±0.01ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/async 1.00 1731.8±8.79µs ? ?/sec 1.03 1783.8±13.71µs ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/sync 1.00 1738.6±10.43µs ? ?/sec 1.03 1797.4±20.43µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/async 1.00 1236.6±8.72µs ? ?/sec 1.02 1258.2±5.12µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/sync 1.00 1268.2±9.72µs ? ?/sec 1.02 1289.1±6.44µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/async 1.00 1122.2±11.26µs ? ?/sec 1.04 1165.9±5.60µs ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/sync 1.00 1135.2±8.30µs ? ?/sec 1.04 1182.6±6.68µs ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/async 1.31 4.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/sync 1.36 4.8±0.02ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/async 1.32 3.5±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/sync 1.39 3.4±0.01ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2528262842
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
+ // Prefer a bitmap mask when the selectors are short on average, as the mask
+ // (re)construction cost is amortized by a simpler execution path during reads.
+ let use_mask = match strategy {
+ RowSelectionStrategy::Mask => true,
+ RowSelectionStrategy::Auto => {
+ selector_count == 0
+ || total_rows < selector_count.saturating_mul(AVG_SELECTOR_LEN_MASK_THRESHOLD)
+ }
+ RowSelectionStrategy::Selectors => unreachable!(),
+ };
+
+ let storage = if use_mask {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ } else {
+ RowSelectionBacking::Selectors(selectors.into())
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
+ }
+ RowSelectionBacking::Mask(_) => None,
+ }
+ }
+
+ /// Return a selector to the front, rewinding the position (sparse-only)
+ pub fn return_selector(&mut self, selector: RowSelector) {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ self.position = self.position.saturating_sub(selector.row_count);
+ selectors.push_front(selector);
+ }
+ RowSelectionBacking::Mask(_) => {
+ unreachable!("return_selector called for mask-based RowSelectionCursor")
+ }
+ }
+ }
+
+ /// Returns `true` if the cursor is backed by a boolean mask
+ pub fn is_mask_backed(&self) -> bool {
+ matches!(self.storage, RowSelectionBacking::Mask(_))
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = match &self.storage {
+ RowSelectionBacking::Mask(mask) => mask,
+ RowSelectionBacking::Selectors(_) => return None,
+ };
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
Review Comment:
Filed a ticket to track
- https://github.com/apache/arrow-rs/issues/8847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3533718023
I went quite crazy this morning and filed a bunch of follow on tickets
- [ ] https://github.com/apache/arrow-rs/issues/8844
- [ ] https://github.com/apache/arrow-rs/issues/8845
- [ ] https://github.com/apache/arrow-rs/issues/8846
- [ ] https://github.com/apache/arrow-rs/issues/8847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483681768
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
I think this will cause the reader to produce batches potentially much smaller than the requested batch size, which at the very least should be documented. Ideally we'd just store the mask somewhere, and flush + filter when we have enough rows.
Otherwise consumers like DataFusion are going to need to wrap ParquetExec in a CoalesceBatches and effectively buffer all the data again.
Edit edit: Filtering small batches may also have performance implications - the filter kernels are optimized for filtering thousands of rows at a time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2478972703
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
Review Comment:
I found this code quite clear and easy to read -- thank you 🙏
I do think it would be good if we could add some of the background context here as a commenet
Specifically it is not obvious from just the code that this benchmark can be used to determine the value of `AVG_SELECTOR_LEN_MASK_THRESHOLD`) -- perhaps you can reuse some of the description from this PR?
Also, how did you generate these charts? If it is straightforward perhaps you can also describe that in the comments
https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -271,6 +275,14 @@ impl<T> ArrowReaderBuilder<T> {
}
}
+ /// Override the strategy used to execute the configured [`RowSelection`]
Review Comment:
I think this is a very nice API and allows users to control the behavior
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1035,8 +1050,85 @@ impl ParquetRecordBatchReader {
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
Some(selection) => {
+ if selection.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ loop {
+ let mask_chunk = match selection.next_mask_chunk(batch_size) {
+ Some(batch) => batch,
+ None => return Ok(None),
+ };
Review Comment:
A small nit is you could make this somewhat more concise via
```suggestion
let Some(mask_chunk) = selection.next_mask_chunk(batch_size) else {
return Ok(None);
};
```
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
+ // Prefer a bitmap mask when the selectors are short on average, as the mask
+ // (re)construction cost is amortized by a simpler execution path during reads.
+ let use_mask = match strategy {
+ RowSelectionStrategy::Mask => true,
+ RowSelectionStrategy::Auto => {
+ selector_count == 0
+ || total_rows < selector_count.saturating_mul(AVG_SELECTOR_LEN_MASK_THRESHOLD)
+ }
+ RowSelectionStrategy::Selectors => unreachable!(),
+ };
+
+ let storage = if use_mask {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ } else {
+ RowSelectionBacking::Selectors(selectors.into())
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
+ }
+ RowSelectionBacking::Mask(_) => None,
Review Comment:
perhaps this should also be `unreachable!` to signal it is not expected as well as mirror the code in `return_selector`
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
+ // Prefer a bitmap mask when the selectors are short on average, as the mask
+ // (re)construction cost is amortized by a simpler execution path during reads.
+ let use_mask = match strategy {
+ RowSelectionStrategy::Mask => true,
+ RowSelectionStrategy::Auto => {
+ selector_count == 0
+ || total_rows < selector_count.saturating_mul(AVG_SELECTOR_LEN_MASK_THRESHOLD)
+ }
+ RowSelectionStrategy::Selectors => unreachable!(),
+ };
+
+ let storage = if use_mask {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ } else {
+ RowSelectionBacking::Selectors(selectors.into())
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
+ }
+ RowSelectionBacking::Mask(_) => None,
+ }
+ }
+
+ /// Return a selector to the front, rewinding the position (sparse-only)
+ pub fn return_selector(&mut self, selector: RowSelector) {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ self.position = self.position.saturating_sub(selector.row_count);
+ selectors.push_front(selector);
+ }
+ RowSelectionBacking::Mask(_) => {
+ unreachable!("return_selector called for mask-based RowSelectionCursor")
+ }
+ }
+ }
+
+ /// Returns `true` if the cursor is backed by a boolean mask
+ pub fn is_mask_backed(&self) -> bool {
+ matches!(self.storage, RowSelectionBacking::Mask(_))
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = match &self.storage {
+ RowSelectionBacking::Mask(mask) => mask,
+ RowSelectionBacking::Selectors(_) => return None,
+ };
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
+ initial_skip += 1;
+ cursor += 1;
+ }
+
+ let mask_start = cursor;
+ let mut chunk_rows = 0;
+ let mut selected_rows = 0;
+
+ // Advance until enough rows have been selected to satisfy the batch size,
+ // or until the mask is exhausted. This mirrors the behaviour of the legacy
+ // `RowSelector` queue-based iteration.
+ while cursor < mask.len() && selected_rows < batch_size {
+ chunk_rows += 1;
+ if mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ (initial_skip, chunk_rows, selected_rows, mask_start, cursor)
+ };
+
+ self.position = end_position;
+
+ Some(MaskChunk {
+ initial_skip,
+ chunk_rows,
+ selected_rows,
+ mask_start,
+ })
+ }
+
+ /// Materialise the boolean values for a mask-backed chunk
+ pub fn mask_values_for(&self, chunk: &MaskChunk) -> Option<BooleanArray> {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => {
+ if chunk.mask_start.saturating_add(chunk.chunk_rows) > mask.len() {
+ return None;
+ }
+ Some(BooleanArray::from(
+ mask.slice(chunk.mask_start, chunk.chunk_rows),
+ ))
+ }
+ RowSelectionBacking::Selectors(_) => None,
+ }
+ }
+}
+
+fn boolean_mask_from_selectors(selectors: &[RowSelector]) -> BooleanBuffer {
Review Comment:
I think we can do even better than this (as a follow on PR)
The current code still converts the result of a filter (BooleanArray) to a RowSelection,
https://github.com/apache/arrow-rs/blob/cc1444a3232fa11b8485e2794a88f342bd7f97e2/parquet/src/arrow/arrow_reader/read_plan.rs#L113-L112
and then `boolean_mask_from_selectors` converts it back to a BooleanArray
However I think we could apply the result of evaluating the filter directly to a `RowSelectionBacking::Mask`
In fact, @XiangpengHao even has some (relatively crazy) techniques to combine masks quickly in https://github.com/apache/arrow-rs/pull/6624#discussion_r2127391600
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -23,16 +23,31 @@ use crate::arrow::arrow_reader::{
ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
};
use crate::errors::{ParquetError, Result};
-use arrow_array::Array;
+use arrow_array::{Array, BooleanArray};
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder};
use arrow_select::filter::prep_null_mask_filter;
use std::collections::VecDeque;
+/// Strategy for materialising [`RowSelection`] during execution.
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+pub enum RowSelectionStrategy {
+ /// Automatically choose between mask- and selector-backed execution
Review Comment:
```suggestion
/// Automatically choose between mask- and selector-backed execution
/// based on heuristics
```
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let avg_selector_lengths: &[usize] = &[16, 20, 24, 28, 32, 36, 40];
+ let parquet_data = build_parquet_data(TOTAL_ROWS);
+
+ let scenarios = [
+ Scenario {
+ name: "uniform50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Constant,
+ },
+ Scenario {
+ name: "spread50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Uniform { spread: 0.9 },
+ },
+ Scenario {
+ name: "sparse20",
+ select_ratio: 0.2,
+ start_with_select: false,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 6.0,
+ long_prob: 0.1,
+ },
+ },
+ Scenario {
+ name: "dense80",
+ select_ratio: 0.8,
+ start_with_select: true,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 4.0,
+ long_prob: 0.05,
+ },
+ },
+ ];
+
+ for scenario in scenarios.iter() {
+ for (offset, &avg_len) in avg_selector_lengths.iter().enumerate() {
+ let selectors =
+ generate_selectors(avg_len, TOTAL_ROWS, scenario, BASE_SEED + offset as u64);
+ let stats = SelectorStats::new(&selectors);
+ let suffix = format!(
+ "{}-avg{:.1}-sel{:02}",
+ scenario.name,
+ stats.average_selector_len,
+ (stats.select_ratio * 100.0).round() as u32
+ );
+
+ let bench_input = BenchInput {
+ parquet_data: parquet_data.clone(),
+ selection: RowSelection::from(selectors.clone()),
+ };
+
+ for (label, strategy) in READ_STRATEGIES.iter().copied() {
+ c.bench_with_input(
+ BenchmarkId::new(label, &suffix),
+ &bench_input,
+ |b, input| {
+ b.iter(|| {
+ let total = run_read(&input.parquet_data, &input.selection, strategy);
+ hint::black_box(total);
+ });
+ },
+ );
+ }
+ }
+ }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
+
+struct BenchInput {
+ parquet_data: Bytes,
+ selection: RowSelection,
+}
+
+fn run_read(
+ parquet_data: &Bytes,
+ selection: &RowSelection,
+ strategy: RowSelectionStrategy,
+) -> usize {
+ let reader = ParquetRecordBatchReaderBuilder::try_new(parquet_data.clone())
+ .unwrap()
+ .with_batch_size(BATCH_SIZE)
+ .with_row_selection(selection.clone())
+ .with_row_selection_strategy(strategy)
+ .build()
+ .unwrap();
+
+ let mut total_rows = 0usize;
+ for batch in reader {
+ let batch = batch.unwrap();
+ total_rows += batch.num_rows();
+ }
+ total_rows
+}
+
+fn build_parquet_data(total_rows: usize) -> Bytes {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "value",
+ DataType::Int32,
+ false,
+ )]));
+ let values = Int32Array::from_iter_values((0..total_rows).map(|v| v as i32));
+ let columns: Vec<ArrayRef> = vec![Arc::new(values) as ArrayRef];
Review Comment:
I recommend we also test with some variable length rows too -- as the selection overhead may be different for StringArray/StringViewArray than a i32
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let avg_selector_lengths: &[usize] = &[16, 20, 24, 28, 32, 36, 40];
+ let parquet_data = build_parquet_data(TOTAL_ROWS);
+
+ let scenarios = [
+ Scenario {
+ name: "uniform50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Constant,
+ },
+ Scenario {
+ name: "spread50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Uniform { spread: 0.9 },
+ },
+ Scenario {
+ name: "sparse20",
+ select_ratio: 0.2,
+ start_with_select: false,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 6.0,
+ long_prob: 0.1,
+ },
+ },
+ Scenario {
+ name: "dense80",
+ select_ratio: 0.8,
+ start_with_select: true,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 4.0,
+ long_prob: 0.05,
+ },
+ },
+ ];
+
+ for scenario in scenarios.iter() {
+ for (offset, &avg_len) in avg_selector_lengths.iter().enumerate() {
+ let selectors =
+ generate_selectors(avg_len, TOTAL_ROWS, scenario, BASE_SEED + offset as u64);
+ let stats = SelectorStats::new(&selectors);
+ let suffix = format!(
+ "{}-avg{:.1}-sel{:02}",
+ scenario.name,
+ stats.average_selector_len,
+ (stats.select_ratio * 100.0).round() as u32
+ );
+
+ let bench_input = BenchInput {
+ parquet_data: parquet_data.clone(),
+ selection: RowSelection::from(selectors.clone()),
+ };
+
+ for (label, strategy) in READ_STRATEGIES.iter().copied() {
+ c.bench_with_input(
+ BenchmarkId::new(label, &suffix),
+ &bench_input,
+ |b, input| {
+ b.iter(|| {
+ let total = run_read(&input.parquet_data, &input.selection, strategy);
+ hint::black_box(total);
+ });
+ },
+ );
+ }
+ }
+ }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
+
+struct BenchInput {
+ parquet_data: Bytes,
+ selection: RowSelection,
+}
+
+fn run_read(
+ parquet_data: &Bytes,
+ selection: &RowSelection,
+ strategy: RowSelectionStrategy,
+) -> usize {
+ let reader = ParquetRecordBatchReaderBuilder::try_new(parquet_data.clone())
+ .unwrap()
+ .with_batch_size(BATCH_SIZE)
+ .with_row_selection(selection.clone())
+ .with_row_selection_strategy(strategy)
+ .build()
+ .unwrap();
+
+ let mut total_rows = 0usize;
+ for batch in reader {
+ let batch = batch.unwrap();
+ total_rows += batch.num_rows();
+ }
+ total_rows
+}
+
+fn build_parquet_data(total_rows: usize) -> Bytes {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "value",
+ DataType::Int32,
+ false,
+ )]));
+ let values = Int32Array::from_iter_values((0..total_rows).map(|v| v as i32));
+ let columns: Vec<ArrayRef> = vec![Arc::new(values) as ArrayRef];
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let mut writer = ArrowWriter::try_new(Vec::new(), schema, None).unwrap();
+ writer.write(&batch).unwrap();
+ let buffer = writer.into_inner().unwrap();
+ Bytes::from(buffer)
+}
+
+#[derive(Clone)]
+struct Scenario {
+ name: &'static str,
+ select_ratio: f64,
+ start_with_select: bool,
+ distribution: RunDistribution,
+}
+
+#[derive(Clone)]
+enum RunDistribution {
+ Constant,
+ Uniform { spread: f64 },
+ Bimodal { long_factor: f64, long_prob: f64 },
+}
+
+fn generate_selectors(
+ avg_selector_len: usize,
+ total_rows: usize,
+ scenario: &Scenario,
+ seed: u64,
+) -> Vec<RowSelector> {
+ assert!(
+ (0.0..=1.0).contains(&scenario.select_ratio),
+ "select_ratio must be in [0, 1]"
+ );
+
+ let mut select_mean = scenario.select_ratio * 2.0 * avg_selector_len as f64;
+ let mut skip_mean = (1.0 - scenario.select_ratio) * 2.0 * avg_selector_len as f64;
+
+ select_mean = select_mean.max(1.0);
+ skip_mean = skip_mean.max(1.0);
+
+ let sum = select_mean + skip_mean;
+ // Rebalance the sampled select/skip run lengths so their sum matches the requested
+ // average selector length while respecting the configured selectivity ratio.
+ let scale = if sum == 0.0 {
+ 1.0
+ } else {
+ (2.0 * avg_selector_len as f64) / sum
+ };
+ select_mean *= scale;
+ skip_mean *= scale;
+
+ let mut rng = StdRng::seed_from_u64(seed ^ (avg_selector_len as u64).wrapping_mul(0x9E3779B1));
+ let mut selectors = Vec::with_capacity(total_rows / avg_selector_len.max(1));
+ let mut remaining = total_rows;
+ let mut is_select = scenario.start_with_select;
+
+ while remaining > 0 {
+ let mean = if is_select { select_mean } else { skip_mean };
+ let len = sample_length(mean, &scenario.distribution, &mut rng).max(1);
+ let len = len.min(remaining);
+ selectors.push(if is_select {
+ RowSelector::select(len)
+ } else {
+ RowSelector::skip(len)
+ });
+ remaining -= len;
+ if remaining == 0 {
+ break;
+ }
+ is_select = !is_select;
+ }
+
+ let selection: RowSelection = selectors.into();
+ selection.into()
+}
+
+fn sample_length(mean: f64, distribution: &RunDistribution, rng: &mut StdRng) -> usize {
+ match distribution {
+ RunDistribution::Constant => mean.round().max(1.0) as usize,
+ RunDistribution::Uniform { spread } => {
+ let spread = spread.clamp(0.0, 0.99);
+ let lower = (mean * (1.0 - spread)).max(1.0);
+ let upper = (mean * (1.0 + spread)).max(lower + f64::EPSILON);
+ if (upper - lower) < 1.0 {
+ lower.round().max(1.0) as usize
+ } else {
+ let low = lower.floor() as usize;
+ let high = upper.ceil() as usize;
+ rng.random_range(low..=high).max(1)
+ }
+ }
+ RunDistribution::Bimodal {
+ long_factor,
+ long_prob,
+ } => {
+ let long_prob = long_prob.clamp(0.0, 0.5);
+ let short_prob = 1.0 - long_prob;
+ let short_factor = if short_prob == 0.0 {
+ 1.0 / long_factor.max(f64::EPSILON)
+ } else {
+ (1.0 - long_prob * long_factor).max(0.0) / short_prob
+ };
+ let use_long = rng.random_bool(long_prob);
+ let factor = if use_long {
+ *long_factor
+ } else {
+ short_factor.max(0.1)
+ };
+ (mean * factor).round().max(1.0) as usize
+ }
+ }
+}
+
+struct SelectorStats {
Review Comment:
this is a neat structure
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
Review Comment:
I recommend we pull this constant somewhere that is easier to find along with a comment about what it is and how it was chosen. I suggest simply making it a constant in this module
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
+ // Prefer a bitmap mask when the selectors are short on average, as the mask
+ // (re)construction cost is amortized by a simpler execution path during reads.
+ let use_mask = match strategy {
+ RowSelectionStrategy::Mask => true,
+ RowSelectionStrategy::Auto => {
+ selector_count == 0
+ || total_rows < selector_count.saturating_mul(AVG_SELECTOR_LEN_MASK_THRESHOLD)
+ }
+ RowSelectionStrategy::Selectors => unreachable!(),
+ };
+
+ let storage = if use_mask {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ } else {
+ RowSelectionBacking::Selectors(selectors.into())
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
+ }
+ RowSelectionBacking::Mask(_) => None,
+ }
+ }
+
+ /// Return a selector to the front, rewinding the position (sparse-only)
+ pub fn return_selector(&mut self, selector: RowSelector) {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ self.position = self.position.saturating_sub(selector.row_count);
+ selectors.push_front(selector);
+ }
+ RowSelectionBacking::Mask(_) => {
+ unreachable!("return_selector called for mask-based RowSelectionCursor")
+ }
+ }
+ }
+
+ /// Returns `true` if the cursor is backed by a boolean mask
+ pub fn is_mask_backed(&self) -> bool {
+ matches!(self.storage, RowSelectionBacking::Mask(_))
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = match &self.storage {
+ RowSelectionBacking::Mask(mask) => mask,
+ RowSelectionBacking::Selectors(_) => return None,
+ };
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
Review Comment:
I suspect there are all sorts of bit level hacks we can do to make this faster (as a follow on PR) - for example leveraging the code to count the number of 1s a u64 at a time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2479110401
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
I don't understand the need for the synthetic page -- it seems like a workaround for some case that should be handled in the control flow loop in `ParquetRecordBatchReader::next_inner`
Specifically, given skipping / scanning data pages works with the `RowSelection` approach, why does a mask approach cause additional problems? In some way the mask approach should actually decode *more* rows, not less (as then the filter is applied afterwards)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3512589913
> Thank you ! I think the only thing need to be discussed is https://github.com/apache/arrow-rs/pull/8733#discussion_r2505421194
> If the threshold should be included in the RowSelectionStrategy, if so the API for the Builders should be updated. As the last commit.
Yes, I am trying some options to see if I can combine the RowSelectionStrategy and the RowSelection together into the public API...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3513843541
Here are some notes to myself to file PRs for follow on work:
1. Add a way to avoid converting from a Mask --> Selection with the result of evaluating predicates (I think this will be a natural follow on)
2. Update the code for data page skipping to be mask aware (and not skip pages when doing so would prevent mask evaluation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2506939972
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -691,6 +736,180 @@ fn union_row_selections(left: &[RowSelector], right: &[RowSelector]) -> RowSelec
iter.collect()
}
+/// Cursor for iterating a [`RowSelection`] during execution within a
+/// [`ReadPlan`](crate::arrow::arrow_reader::ReadPlan).
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to the internal `RowSelectionBacking`.
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+pub enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+#[derive(Debug)]
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ pub(crate) fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ let storage = match strategy {
+ RowSelectionStrategy::Mask => {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ }
+ RowSelectionStrategy::Selectors => RowSelectionBacking::Selectors(selectors.into()),
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
Review Comment:
Good find! Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3503703950
Thank you -- I plan to review this PR again shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165
Use `cargo bench --bench row_selection_state ` can test the relationship between the length of rowselection and time consuming.
On my M2 Mac Macbook:
<img width="1780" height="980" alt="output-2" src="https://github.com/user-attachments/assets/b9ec3d74-d208-4dcc-8b0f-fc1d75e34046" />
when the length is `15`, the perf of the bit mask and rowselector could be the same.
On codespace AMD EPYC 7763 64-Core Processor (4 core)
<img width="2053" height="980" alt="output" src="https://github.com/user-attachments/assets/8cc63778-58f9-4303-8b4e-e09e784b910e" />
The Cross comes to around `30`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483681768
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
I think this will cause the reader to produce batches potentially much smaller than the requested batch size, which at the very least should be documented. Ideally we'd just store the mask somewhere, and flush + filter when we have enough rows.
Otherwise consumers like DataFusion are going to need to wrap ParquetExec in a CoalesceBatches and effectively buffer all the data again.
Edit edit: Filtering small batches may also have performance implications - the filter kernels are optimized for filtering thousands of rows at a time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483702970
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
Actually I realise now that MaskChunk ensures that we read enough such that once filtered it has the desired batch size 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2485931247
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
Thank you alamb, it looks like there's still something unresolved for the PR. I'm going to resolve it in the next few days. At mean time I may update or rebase the branch multiple times. So I converted the PR into draft.
The things left are:
1. Add benchmark for the different types of value to determine the final length to do the `selection`/`bitmask` converting
2. Add some guidance or tool to draw the charts, then we can collect more statistics data from different platform.
3. For the design of synthetic page, We all agree it's not a good idea, I need to find another method to handle the sparse page.
4. Add new tests to test if the `bitmask` method can handle all kinds of skipped page in sparse column chunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3464072808
😮 thank you @hhhizzz -- I plan to review this PR carefully, but it will likely take me a few days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2511016276
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -934,12 +950,16 @@ impl<T: ChunkReader + 'static> ParquetRecordBatchReaderBuilder<T> {
let array_reader = ArrayReaderBuilder::new(&reader, &metrics)
.build_array_reader(fields.as_deref(), &projection)?;
- let read_plan = plan_builder
+ let mut plan_builder = plan_builder
.limited(reader.num_rows())
.with_offset(offset)
.with_limit(limit)
- .build_limited()
- .build();
+ .build_limited();
+
+ let preferred_strategy = plan_builder.preferred_selection_strategy();
Review Comment:
I choose `preferred_selection_strategy` because this is just one option **preferred** by `readPlan`, it will decide the final `strategy` after it include the page(offset) into consideration.
Here the `sync reader` won't do page skip so the preferred is the final result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2511754194
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -233,12 +283,12 @@ pub struct ReadPlan {
/// The number of rows to read in each batch
batch_size: usize,
/// Row ranges to be selected from the data source
- selection: Option<VecDeque<RowSelector>>,
+ selection: Option<RowSelectionCursor>,
}
impl ReadPlan {
/// Returns a mutable reference to the selection, if any
- pub fn selection_mut(&mut self) -> Option<&mut VecDeque<RowSelector>> {
+ pub fn selection_mut(&mut self) -> Option<&mut RowSelectionCursor> {
Review Comment:
I propose this change in
- https://github.com/hhhizzz/arrow-rs/pull/8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504551776
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
Review Comment:
Moved to another place and added the function to manipulate in benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3513827376
Here is another PR
- https://github.com/hhhizzz/arrow-rs/pull/9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3533465515
Let's gogogogogogo!
I am working on filing follow on tickets too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2477903033
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -249,3 +254,177 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>) -> Self {
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 8;
Review Comment:
But it's more reasonable for this PR:
> Added a benchmark to determine this threshold value (8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2478069494
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -249,3 +254,177 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>) -> Self {
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 8;
Review Comment:
Nice @hhhizzz , i am wandering if we can change to more stable choice, such as statistic based choice, but it's a good start for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3464073790
fyi @zhuqi-lucas and @XiangpengHao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483681768
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
I think this will cause the reader to produce batches potentially much smaller than the requested batch size, which at the very least should be documented. Ideally we'd just store the mask somewhere, and flush + filter when we have enough rows.
Otherwise consumers like DataFusion are going to need to wrap ParquetExec in a CoalesceBatches and effectively buffer all the data again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3505266953
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main rowselectionempty
----- ---- -----------------
arrow_reader_clickbench/async/Q1 1.00 2.3±0.02ms ? ?/sec 1.01 2.4±0.01ms ? ?/sec
arrow_reader_clickbench/async/Q10 1.01 12.8±0.26ms ? ?/sec 1.00 12.7±0.22ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.01 14.7±0.27ms ? ?/sec 1.00 14.6±0.25ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.06 26.8±0.21ms ? ?/sec 1.00 25.4±0.21ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.24 38.1±0.29ms ? ?/sec 1.00 30.7±0.24ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.28 36.1±0.20ms ? ?/sec 1.00 28.2±0.29ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.02 5.5±0.10ms ? ?/sec 1.00 5.4±0.06ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.03 126.2±0.60ms ? ?/sec 1.00 122.5±0.45ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 155.1±1.68ms ? ?/sec 1.01 156.1±2.15ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.02 309.4±9.40ms ? ?/sec 1.00 303.9±13.81ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.00 399.7±1.82ms ? ?/sec 1.02 406.8±2.26ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.23 42.1±0.43ms ? ?/sec 1.00 34.2±0.33ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.00 97.6±0.43ms ? ?/sec 1.03 100.2±0.46ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.00 98.0±0.39ms ? ?/sec 1.00 98.4±0.35ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.75 53.5±0.36ms ? ?/sec 1.00 30.6±0.25ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.00 118.8±0.43ms ? ?/sec 1.02 120.7±0.57ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.00 95.0±0.32ms ? ?/sec 1.02 96.5±0.39ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.00 36.4±0.26ms ? ?/sec 1.02 37.0±0.26ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.02 46.3±0.26ms ? ?/sec 1.00 45.3±0.31ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.05 44.5±0.26ms ? ?/sec 1.00 42.2±0.48ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.13 34.9±0.24ms ? ?/sec 1.00 31.0±0.27ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.15 13.5±0.10ms ? ?/sec 1.00 11.8±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 2.1±0.02ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q10 1.01 9.4±0.06ms ? ?/sec 1.00 9.3±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.02 11.1±0.07ms ? ?/sec 1.00 10.8±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 33.9±0.25ms ? ?/sec 1.09 37.0±1.20ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.16 44.4±0.22ms ? ?/sec 1.00 38.3±0.36ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.23 43.2±0.24ms ? ?/sec 1.00 35.1±0.31ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.01 4.4±0.02ms ? ?/sec 1.00 4.4±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.00 173.6±0.55ms ? ?/sec 1.02 176.7±0.74ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.00 230.4±1.58ms ? ?/sec 1.01 232.6±1.08ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.00 476.4±2.32ms ? ?/sec 1.01 481.6±3.24ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.00 424.3±9.53ms ? ?/sec 1.01 428.3±7.19ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.17 50.2±0.47ms ? ?/sec 1.00 43.0±0.31ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.00 151.4±0.86ms ? ?/sec 1.02 154.1±1.13ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.00 147.6±0.74ms ? ?/sec 1.00 147.6±0.68ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.69 51.5±0.31ms ? ?/sec 1.00 30.5±0.31ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.00 152.9±0.96ms ? ?/sec 1.01 154.6±0.74ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 89.4±0.44ms ? ?/sec 1.00 89.1±0.44ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 29.6±0.16ms ? ?/sec 1.01 29.9±0.16ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.05 33.9±0.32ms ? ?/sec 1.00 32.2±0.24ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.07 43.4±0.34ms ? ?/sec 1.00 40.5±0.34ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.12 32.8±0.28ms ? ?/sec 1.00 29.2±0.25ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.16 12.7±0.08ms ? ?/sec 1.00 11.0±0.07ms ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3505267049
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing rowselectionempty (fe238774fd3c2e0202e7328c6d1e3174afbd79a1) to d379b981768215b16182f78aebddf940a4b45c99 [diff](https://github.com/apache/arrow-rs/compare/d379b981768215b16182f78aebddf940a4b45c99..fe238774fd3c2e0202e7328c6d1e3174afbd79a1)
BENCH_NAME=arrow_reader_row_filter
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader_row_filter
BENCH_FILTER=
BENCH_BRANCH_NAME=rowselectionempty
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2507014855
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -20,19 +20,70 @@
use crate::arrow::array_reader::ArrayReader;
use crate::arrow::arrow_reader::{
- ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
+ ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelectionCursor,
+ RowSelectionStrategy, RowSelector,
};
use crate::errors::{ParquetError, Result};
use arrow_array::Array;
use arrow_select::filter::prep_null_mask_filter;
-use std::collections::VecDeque;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+// The average selector length threshold for choosing between
+// `RowSelectionStrategy::Mask` and `RowSelectionStrategy::Selectors`.
+// If the average selector length is less than this value,
+// `RowSelectionStrategy::Mask` is preferred.
+const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 32;
+
+// The logic in `preferred_selection_strategy` depends on the constant
+// `AVG_SELECTOR_LEN_MASK_THRESHOLD`. To allow unit testing of this logic,
+// we use a mutable global variable that can be temporarily changed during tests.
+//
+// An `AtomicUsize` is used because the Rust test runner (`cargo test`) runs tests
+// in parallel by default. The atomic operations prevent data races between
+// different test threads that might try to modify this value simultaneously.
+//
+// For the production code path, `load(Ordering::Relaxed)` is used. This is the
+// weakest memory ordering and for a simple load on most modern architectures,
+// it compiles down to a regular memory read with negligible performance overhead.
+// The more expensive atomic operations with stronger ordering are only used in the
+// test-only functions below.
+static AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE: AtomicUsize =
Review Comment:
To use this type of method, I have to put the strategy into `ReaderBuilder` API. This change is in the last commit. Let me know if this new parameter is allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3505197413
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main rowselectionempty
----- ---- -----------------
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 1278.4±2.22µs ? ?/sec 1.00 1277.0±18.54µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 1295.0±3.05µs ? ?/sec 1.00 1294.9±2.34µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 1286.6±2.18µs ? ?/sec 1.00 1280.9±3.83µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.01 509.5±3.39µs ? ?/sec 1.00 504.4±4.43µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.01 678.0±5.93µs ? ?/sec 1.00 674.4±2.58µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.02 519.1±3.85µs ? ?/sec 1.00 509.3±2.85µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.00 548.0±3.73µs ? ?/sec 1.04 568.5±3.73µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.00 721.9±2.35µs ? ?/sec 1.03 741.7±3.45µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.00 559.4±2.83µs ? ?/sec 1.04 579.6±2.78µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 238.4±2.88µs ? ?/sec 1.08 257.7±2.45µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.00 220.6±0.45µs ? ?/sec 1.06 233.1±0.89µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 242.9±3.00µs ? ?/sec 1.08 262.2±2.44µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.00 290.3±1.09µs ? ?/sec 1.00 289.4±2.17µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.01 283.9±0.73µs ? ?/sec 1.00 281.2±0.69µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.00 256.9±0.95µs ? ?/sec 1.00 256.6±1.26µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.00 298.7±1.27µs ? ?/sec 1.02 304.7±1.70µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 1031.7±2.84µs ? ?/sec 1.08 1113.0±2.21µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 876.5±9.69µs ? ?/sec 1.11 970.1±1.75µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 1040.0±2.45µs ? ?/sec 1.08 1121.3±7.67µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 405.9±1.74µs ? ?/sec 1.08 439.1±2.60µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 538.6±2.11µs ? ?/sec 1.19 640.0±2.09µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 416.1±3.47µs ? ?/sec 1.07 445.1±2.10µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 153.1±0.55µs ? ?/sec 1.32 201.7±0.49µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 296.5±0.66µs ? ?/sec 1.11 330.0±0.68µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 158.1±1.18µs ? ?/sec 1.31 206.9±0.84µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 76.5±0.19µs ? ?/sec 1.53 117.3±0.25µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 258.9±0.58µs ? ?/sec 1.11 287.4±0.94µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 82.6±0.39µs ? ?/sec 1.48 122.2±0.26µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 693.6±4.41µs ? ?/sec 1.06 737.4±1.92µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 515.4±1.61µs ? ?/sec 1.15 595.3±2.82µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.00 699.9±2.12µs ? ?/sec 1.06 744.2±1.69µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.08 67.3±5.83µs ? ?/sec 1.00 62.6±6.53µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.00 204.8±1.65µs ? ?/sec 1.27 260.5±1.12µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.09 74.3±6.86µs ? ?/sec 1.00 68.3±5.76µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.00 85.7±0.24µs ? ?/sec 1.10 94.0±0.39µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.04 228.7±0.82µs ? ?/sec 1.00 219.8±0.71µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.00 91.2±0.34µs ? ?/sec 1.08 98.6±0.34µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.01 9.2±0.09µs ? ?/sec 1.00 9.1±0.12µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.07 190.2±0.58µs ? ?/sec 1.00 177.1±0.60µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.01 14.4±0.18µs ? ?/sec 1.00 14.2±0.14µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.00 170.2±0.53µs ? ?/sec 1.08 184.0±0.62µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.00 335.3±2.09µs ? ?/sec 1.02 340.5±0.84µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.00 175.7±0.50µs ? ?/sec 1.08 189.6±0.86µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.03 14.6±0.44µs ? ?/sec 1.00 14.2±0.23µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.01 258.6±0.53µs ? ?/sec 1.00 256.5±0.65µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.04 20.8±0.62µs ? ?/sec 1.00 19.9±0.45µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.00 340.8±1.08µs ? ?/sec 1.08 369.1±16.38µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.00 328.6±0.86µs ? ?/sec 1.19 391.6±1.13µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.00 347.5±0.52µs ? ?/sec 1.07 372.3±0.94µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.00 26.8±0.32µs ? ?/sec 1.01 27.1±0.35µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.00 175.1±0.72µs ? ?/sec 1.27 222.8±0.67µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.00 33.0±0.38µs ? ?/sec 1.05 34.5±0.61µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.03 112.6±0.60µs ? ?/sec 1.00 109.7±0.31µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.00 129.7±0.28µs ? ?/sec 1.01 131.3±0.31µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.01 113.8±0.27µs ? ?/sec 1.00 112.7±0.42µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.01 161.9±0.61µs ? ?/sec 1.00 160.1±0.31µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.00 222.7±0.61µs ? ?/sec 1.01 223.8±2.22µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.01 167.1±0.33µs ? ?/sec 1.00 165.0±0.38µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 76.7±0.26µs ? ?/sec 1.02 78.1±0.24µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 177.0±1.54µs ? ?/sec 1.01 178.6±0.61µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 81.7±0.27µs ? ?/sec 1.00 81.8±0.33µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.07 143.2±0.36µs ? ?/sec 1.00 134.4±0.34µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.02 215.7±1.97µs ? ?/sec 1.00 210.5±0.51µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.07 149.8±0.53µs ? ?/sec 1.00 140.3±0.41µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 71.4±0.31µs ? ?/sec 1.02 73.0±0.20µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.00 175.1±0.50µs ? ?/sec 1.01 177.7±0.98µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.02 78.0±0.35µs ? ?/sec 1.00 76.6±0.25µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.01 109.2±0.18µs ? ?/sec 1.00 108.1±0.29µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.02 134.8±0.33µs ? ?/sec 1.00 131.5±1.38µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.02 112.0±0.19µs ? ?/sec 1.00 110.3±0.92µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 162.4±0.26µs ? ?/sec 1.00 161.9±1.12µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 233.7±1.73µs ? ?/sec 1.00 233.9±1.05µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.01 167.9±0.61µs ? ?/sec 1.00 166.7±0.71µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 201.9±0.51µs ? ?/sec 1.01 203.4±0.81µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 250.1±0.59µs ? ?/sec 1.00 251.2±0.51µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 207.2±0.33µs ? ?/sec 1.02 210.5±0.88µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 142.7±0.30µs ? ?/sec 1.06 151.9±1.57µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 218.9±1.67µs ? ?/sec 1.05 230.5±0.86µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 148.6±0.34µs ? ?/sec 1.06 157.9±1.22µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 107.2±1.51µs ? ?/sec 1.06 113.2±1.46µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 198.2±0.58µs ? ?/sec 1.01 201.1±1.93µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.00 112.5±0.63µs ? ?/sec 1.05 117.8±0.95µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.01 78.5±0.23µs ? ?/sec 1.00 77.4±0.17µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.01 104.3±0.25µs ? ?/sec 1.00 103.1±0.33µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.02 81.4±0.45µs ? ?/sec 1.00 80.1±0.20µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.01 109.2±0.35µs ? ?/sec 1.00 107.9±0.25µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.01 177.0±0.76µs ? ?/sec 1.00 175.0±1.24µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.01 113.5±0.21µs ? ?/sec 1.00 112.4±0.37µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 41.9±0.16µs ? ?/sec 1.05 44.0±0.12µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.00 141.3±0.44µs ? ?/sec 1.01 142.8±0.58µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 46.7±0.73µs ? ?/sec 1.03 48.1±0.15µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.09 110.9±0.28µs ? ?/sec 1.00 102.1±0.40µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.02 179.8±0.47µs ? ?/sec 1.00 176.8±2.73µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.08 115.5±0.26µs ? ?/sec 1.00 106.6±0.28µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.01 37.8±0.12µs ? ?/sec 1.00 37.6±0.16µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.00 139.1±0.49µs ? ?/sec 1.01 140.1±0.47µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.00 42.6±0.12µs ? ?/sec 1.00 42.6±0.12µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.01 84.9±0.16µs ? ?/sec 1.00 83.8±0.75µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.01 103.4±0.49µs ? ?/sec 1.00 102.8±0.28µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.01 87.6±1.17µs ? ?/sec 1.00 87.1±0.90µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.01 110.7±0.38µs ? ?/sec 1.00 109.5±0.98µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.00 169.0±1.10µs ? ?/sec 1.00 168.7±3.00µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.01 115.2±0.56µs ? ?/sec 1.00 114.1±1.20µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.03 26.1±0.28µs ? ?/sec 1.00 25.3±0.18µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.00 124.2±0.36µs ? ?/sec 1.00 123.9±0.69µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.03 29.4±0.33µs ? ?/sec 1.00 28.6±0.21µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.11 93.4±0.44µs ? ?/sec 1.00 84.4±0.29µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.03 162.3±0.68µs ? ?/sec 1.00 157.7±0.54µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.10 98.1±0.52µs ? ?/sec 1.00 89.3±0.40µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.00 17.7±0.33µs ? ?/sec 1.00 17.7±0.55µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.00 121.9±0.32µs ? ?/sec 1.00 121.9±1.35µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.01 24.0±0.42µs ? ?/sec 1.00 23.7±0.37µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.01 82.3±0.20µs ? ?/sec 1.00 81.7±0.26µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.01 104.2±0.68µs ? ?/sec 1.00 103.5±0.29µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.01 83.4±0.44µs ? ?/sec 1.00 82.9±0.54µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.02 111.0±0.35µs ? ?/sec 1.00 108.5±0.68µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.04 177.0±0.59µs ? ?/sec 1.00 170.6±0.92µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.03 114.3±0.50µs ? ?/sec 1.00 111.4±0.51µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 149.5±0.58µs ? ?/sec 1.00 148.3±0.53µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.01 195.3±0.48µs ? ?/sec 1.00 194.1±0.72µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.01 154.3±0.41µs ? ?/sec 1.00 153.1±0.43µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.00 91.1±0.56µs ? ?/sec 1.08 98.3±0.59µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 164.4±0.31µs ? ?/sec 1.01 166.1±0.49µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.00 95.7±0.46µs ? ?/sec 1.08 103.1±0.44µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.06 45.3±1.90µs ? ?/sec 1.00 42.7±0.70µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.01 140.2±0.60µs ? ?/sec 1.00 138.3±0.67µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.02 54.6±2.30µs ? ?/sec 1.00 53.6±2.65µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.00 82.5±0.18µs ? ?/sec 1.00 82.4±0.27µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.00 104.0±0.21µs ? ?/sec 1.00 103.7±0.44µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.00 85.1±0.84µs ? ?/sec 1.00 85.1±0.78µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.00 110.6±0.24µs ? ?/sec 1.00 110.3±0.23µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.00 173.5±0.39µs ? ?/sec 1.00 173.2±7.35µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 115.2±0.26µs ? ?/sec 1.00 114.8±0.19µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.05 35.8±0.19µs ? ?/sec 1.00 34.0±0.11µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.00 134.5±0.57µs ? ?/sec 1.02 136.8±0.50µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.02 40.6±0.18µs ? ?/sec 1.00 39.8±0.14µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.10 103.4±0.37µs ? ?/sec 1.00 93.9±0.29µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.02 172.3±1.18µs ? ?/sec 1.00 168.5±1.20µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.09 108.1±0.25µs ? ?/sec 1.00 98.9±0.37µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.00 30.0±0.13µs ? ?/sec 1.00 30.0±0.08µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.00 131.7±0.56µs ? ?/sec 1.01 133.3±0.38µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.00 34.5±0.08µs ? ?/sec 1.00 34.5±0.08µs ? ?/sec
arrow_array_reader/ListArray/plain encoded optional strings half NULLs 1.00 7.2±0.04ms ? ?/sec 1.02 7.4±0.04ms ? ?/sec
arrow_array_reader/ListArray/plain encoded optional strings no NULLs 1.00 12.8±0.15ms ? ?/sec 1.01 12.9±0.11ms ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.00 511.3±2.50µs ? ?/sec 1.00 511.2±3.78µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.00 673.2±2.54µs ? ?/sec 1.00 674.3±2.39µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.00 517.8±2.73µs ? ?/sec 1.00 517.4±4.10µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.00 664.1±5.28µs ? ?/sec 1.03 681.2±2.56µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.00 770.1±3.56µs ? ?/sec 1.03 792.8±3.69µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.00 675.6±3.08µs ? ?/sec 1.03 692.7±6.52µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.01 313.7±0.69µs ? ?/sec 1.00 311.9±0.98µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.00 372.4±1.07µs ? ?/sec 1.00 371.5±1.83µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.00 319.1±2.18µs ? ?/sec 1.00 317.8±4.03µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 240.0±2.89µs ? ?/sec 1.05 252.3±3.14µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.00 212.3±0.62µs ? ?/sec 1.06 224.4±0.69µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 235.1±3.80µs ? ?/sec 1.09 255.5±4.32µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.00 439.2±4.03µs ? ?/sec 1.09 478.4±2.36µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.00 330.9±1.44µs ? ?/sec 1.05 348.4±5.40µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.00 447.2±1.58µs ? ?/sec 1.09 485.3±2.04µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.01 93.8±0.21µs ? ?/sec 1.00 93.1±0.21µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.00 112.4±0.22µs ? ?/sec 1.01 113.2±0.29µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 96.3±0.16µs ? ?/sec 1.00 95.9±0.69µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.01 128.8±0.30µs ? ?/sec 1.00 128.0±0.28µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.01 188.0±0.52µs ? ?/sec 1.00 186.5±0.39µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 133.1±0.41µs ? ?/sec 1.00 132.6±0.46µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 42.0±0.13µs ? ?/sec 1.04 43.7±0.07µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.00 141.8±0.23µs ? ?/sec 1.01 143.6±1.65µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 46.5±0.14µs ? ?/sec 1.03 48.0±0.21µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.10 111.0±0.18µs ? ?/sec 1.00 101.1±0.20µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.02 179.6±1.77µs ? ?/sec 1.00 175.5±0.51µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.09 115.7±0.30µs ? ?/sec 1.00 105.8±0.30µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.00 37.7±0.09µs ? ?/sec 1.00 37.9±0.09µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.00 138.9±0.35µs ? ?/sec 1.02 141.1±0.68µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.00 42.5±0.14µs ? ?/sec 1.00 42.4±0.16µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.02 85.4±0.30µs ? ?/sec 1.00 83.8±0.22µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.01 103.2±0.29µs ? ?/sec 1.00 102.6±0.40µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.02 87.9±0.27µs ? ?/sec 1.00 86.4±0.20µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.01 111.0±0.50µs ? ?/sec 1.00 109.6±0.56µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.01 169.1±0.45µs ? ?/sec 1.00 168.0±0.58µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.01 115.4±0.38µs ? ?/sec 1.00 113.8±0.72µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.03 26.7±0.35µs ? ?/sec 1.00 25.8±0.23µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.00 124.1±0.26µs ? ?/sec 1.02 126.2±0.56µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.01 30.9±0.34µs ? ?/sec 1.00 30.7±0.29µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.10 93.8±0.32µs ? ?/sec 1.00 84.9±0.29µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.03 162.4±0.43µs ? ?/sec 1.00 157.8±0.34µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.11 98.6±0.65µs ? ?/sec 1.00 89.2±0.25µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.01 21.0±0.98µs ? ?/sec 1.00 20.7±0.48µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.00 122.5±0.53µs ? ?/sec 1.01 123.7±1.85µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.01 25.4±0.61µs ? ?/sec 1.00 25.1±0.48µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.01 81.3±0.24µs ? ?/sec 1.00 80.3±0.24µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.00 104.0±0.56µs ? ?/sec 1.00 104.0±0.59µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.01 84.0±0.20µs ? ?/sec 1.00 83.0±0.51µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 109.4±0.41µs ? ?/sec 1.00 109.6±0.54µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.04 176.6±0.75µs ? ?/sec 1.00 169.7±0.64µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.03 115.1±0.42µs ? ?/sec 1.00 111.5±0.54µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.02 150.6±0.33µs ? ?/sec 1.00 148.2±0.54µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.01 197.4±0.41µs ? ?/sec 1.00 194.6±0.51µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.02 156.3±0.59µs ? ?/sec 1.00 152.9±0.38µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.00 90.6±0.43µs ? ?/sec 1.09 98.3±0.38µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.03 170.6±0.38µs ? ?/sec 1.00 166.4±0.61µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.00 94.8±0.46µs ? ?/sec 1.08 102.5±0.42µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.08 45.7±1.69µs ? ?/sec 1.00 42.3±0.96µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.01 139.5±0.61µs ? ?/sec 1.00 137.5±0.49µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.01 52.7±1.92µs ? ?/sec 1.00 52.2±1.62µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.01 89.7±0.23µs ? ?/sec 1.00 88.5±0.25µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.01 108.0±0.21µs ? ?/sec 1.00 107.4±0.34µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.02 92.4±0.17µs ? ?/sec 1.00 90.9±0.22µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.01 120.3±0.24µs ? ?/sec 1.00 119.2±0.24µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.01 178.3±0.36µs ? ?/sec 1.00 177.4±0.38µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.01 124.8±0.31µs ? ?/sec 1.00 123.9±0.39µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 34.0±0.11µs ? ?/sec 1.05 35.6±0.10µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.00 135.4±0.28µs ? ?/sec 1.02 137.7±9.25µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.02 40.6±0.14µs ? ?/sec 1.00 39.8±0.13µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.11 103.3±0.22µs ? ?/sec 1.00 93.4±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.03 172.5±0.36µs ? ?/sec 1.00 168.2±0.42µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.09 108.0±0.23µs ? ?/sec 1.00 98.7±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.00 29.7±0.08µs ? ?/sec 1.01 30.1±0.07µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.00 131.7±0.32µs ? ?/sec 1.02 134.0±0.73µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.01 34.5±0.21µs ? ?/sec 1.00 34.2±0.25µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2506343981
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -16,12 +16,24 @@
// under the License.
use arrow_array::{Array, BooleanArray};
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder};
use arrow_select::filter::SlicesIterator;
use std::cmp::Ordering;
use std::collections::VecDeque;
use std::ops::Range;
-use crate::file::page_index::offset_index::PageLocation;
+use crate::arrow::ProjectionMask;
+use crate::file::page_index::offset_index::{OffsetIndexMetaData, PageLocation};
+
+/// Strategy for materialising [`RowSelection`] during execution.
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+pub enum RowSelectionStrategy {
Review Comment:
Yes, my charts indicate that there are many rules for setting the `RowSelectionStrategy`, like the column type, column count, string length, and their combinations... We can create tickets and collaborate on improving these over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2500284787
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
@alamb Hi, I’ve finished most parts of the change.
The main idea is to **record whether any page has been skipped** — if so, the code will fall back to using `RowSelector`.
I’ve added the corresponding unit tests, and noticed that **the sync version currently doesn’t perform page skipping**. I still included unit tests for it in case this functionality is added later.
Next, I’ll refine the comments and collect statistics on the average selector length across different scenarios. The logic for that is already in the code — I just need to write a small script to simplify the process and generate the charts.
In the meantime, could you please review the current design and let me know if it looks acceptable or if there are any improvements I could make?😁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3508343985
> The benchmarks are looking great. Thank you (again @hhhizzz)
>
> I have been working on the DataFusion 51 release but then I will get back to this one. I expect we'll be able to get this one in this week ❤️
Thank you ! I think the only thing need to be discussed is https://github.com/apache/arrow-rs/pull/8733#discussion_r2505421194
If the threshold should be included in the `RowSelectionStrategy`, if so the API for the Builders should be updated. As the last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3508267575
The benchmarks are looking great. Thank you (again @hhhizzz)
I have been working on the DataFusion 51 release but then I will get back to this one. I expect we'll be able to get this one in this week ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3457930431
Benchmark | Change | Verdict
-- | -- | --
int64 == 9999 / all_columns / async | ⚪ -0.33% | Within noise
int64 == 9999 / all_columns / sync | 🔴 +4.50% | Regressed
int64 == 9999 / exclude_filter_column / async | 🟢 -2.12% | Improved
int64 == 9999 / exclude_filter_column / sync | 🟢 -4.59% | Improved
float64 > 99.0 / all_columns / async | ⚪ +0.92% | Within noise
float64 > 99.0 / all_columns / sync | ⚪ +0.96% | Within noise
float64 > 99.0 / exclude_filter_column / async | 🔴 +5.30% | Regressed
float64 > 99.0 / exclude_filter_column / sync | ⚪ -1.10% | Within noise
ts ≥ 9000 / all_columns / async | ⚪ -0.09% | Within noise
ts ≥ 9000 / all_columns / sync | 🟢 -3.82% | Improved
ts ≥ 9000 / exclude_filter_column / async | 🟢 -4.03% | Improved
ts ≥ 9000 / exclude_filter_column / sync | 🟢 -3.60% | Improved
int64 > 90 / all_columns / async | 🟢 -46.49% | Improved
int64 > 90 / all_columns / sync | 🟢 -46.47% | Improved
int64 > 90 / exclude_filter_column / async | 🟢 -37.76% | Improved
int64 > 90 / exclude_filter_column / sync | 🟢 -40.25% | Improved
float64 ≤ 99.0 / all_columns / async | 🔴 +2.48% | Regressed
float64 ≤ 99.0 / all_columns / sync | 🟢 -4.50% | Improved
float64 ≤ 99.0 / exclude_filter_column / async | 🔴 +2.15% | Regressed
float64 ≤ 99.0 / exclude_filter_column / sync | 🔴 +5.22% | Regressed
ts < 9000 / all_columns / async | 🔴 +6.12% | Regressed
ts < 9000 / all_columns / sync | 🔴 +1.49% | Regressed
ts < 9000 / exclude_filter_column / async | 🔴 +3.00% | Regressed
ts < 9000 / exclude_filter_column / sync | 🔴 +5.60% | Regressed
utf8View <> '' / all_columns / async | 🟢 -39.15% | Improved
utf8View <> '' / all_columns / sync | 🟢 -35.00% | Improved
utf8View <> '' / exclude_filter_column / async | 🟢 -34.05% | Improved
utf8View <> '' / exclude_filter_column / sync | 🟢 -33.03% | Improved
float64 > 99.0 AND ts ≥ 9000 / all_columns / async | 🔴 +5.01% | Regressed
float64 > 99.0 AND ts ≥ 9000 / all_columns / sync | 🔴 +3.97% | Regressed
float64 > 99.0 AND ts ≥ 9000 / exclude_filter_column / async | 🔴 +4.67% | Regressed
float64 > 99.0 AND ts ≥ 9000 / exclude_filter_column / sync | ⚪ +0.93% | Within noise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2528241286
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -16,12 +16,24 @@
// under the License.
use arrow_array::{Array, BooleanArray};
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder};
use arrow_select::filter::SlicesIterator;
use std::cmp::Ordering;
use std::collections::VecDeque;
use std::ops::Range;
-use crate::file::page_index::offset_index::PageLocation;
+use crate::arrow::ProjectionMask;
+use crate::file::page_index::offset_index::{OffsetIndexMetaData, PageLocation};
+
+/// Strategy for materialising [`RowSelection`] during execution.
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+pub enum RowSelectionStrategy {
Review Comment:
Filed a ticket to track:
- https://github.com/apache/arrow-rs/issues/8846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3535496150
@alamb @zhuqi-lucas @tustvold Thank you — **I know this might be a bit off the main topic, but I still want to express my appreciation.**
Recently my life has been in a pretty rough state, and I’m currently between jobs. But contributing here has given me a really valuable opportunity to stretch my programming skills. I’ve learned a lot through the process, and I’m grateful for that.
We still have a few follow-up tickets related to this improvement, and I’ll continue exploring them. Thanks again for your patience and support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2503572616
##########
parquet/benches/row_selection_state.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint;
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, Int32Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use bytes::Bytes;
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+use parquet::arrow::ArrowWriter;
+use parquet::arrow::arrow_reader::{
+ ParquetRecordBatchReaderBuilder, RowSelection, RowSelectionStrategy, RowSelector,
+};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+
+const TOTAL_ROWS: usize = 1 << 20;
+const BATCH_SIZE: usize = 1 << 10;
+const BASE_SEED: u64 = 0xA55AA55A;
+const READ_STRATEGIES: &[(&str, RowSelectionStrategy)] = &[
+ ("read_mask", RowSelectionStrategy::Mask),
+ ("read_selectors", RowSelectionStrategy::Selectors),
+];
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let avg_selector_lengths: &[usize] = &[16, 20, 24, 28, 32, 36, 40];
+ let parquet_data = build_parquet_data(TOTAL_ROWS);
+
+ let scenarios = [
+ Scenario {
+ name: "uniform50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Constant,
+ },
+ Scenario {
+ name: "spread50",
+ select_ratio: 0.5,
+ start_with_select: false,
+ distribution: RunDistribution::Uniform { spread: 0.9 },
+ },
+ Scenario {
+ name: "sparse20",
+ select_ratio: 0.2,
+ start_with_select: false,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 6.0,
+ long_prob: 0.1,
+ },
+ },
+ Scenario {
+ name: "dense80",
+ select_ratio: 0.8,
+ start_with_select: true,
+ distribution: RunDistribution::Bimodal {
+ long_factor: 4.0,
+ long_prob: 0.05,
+ },
+ },
+ ];
+
+ for scenario in scenarios.iter() {
+ for (offset, &avg_len) in avg_selector_lengths.iter().enumerate() {
+ let selectors =
+ generate_selectors(avg_len, TOTAL_ROWS, scenario, BASE_SEED + offset as u64);
+ let stats = SelectorStats::new(&selectors);
+ let suffix = format!(
+ "{}-avg{:.1}-sel{:02}",
+ scenario.name,
+ stats.average_selector_len,
+ (stats.select_ratio * 100.0).round() as u32
+ );
+
+ let bench_input = BenchInput {
+ parquet_data: parquet_data.clone(),
+ selection: RowSelection::from(selectors.clone()),
+ };
+
+ for (label, strategy) in READ_STRATEGIES.iter().copied() {
+ c.bench_with_input(
+ BenchmarkId::new(label, &suffix),
+ &bench_input,
+ |b, input| {
+ b.iter(|| {
+ let total = run_read(&input.parquet_data, &input.selection, strategy);
+ hint::black_box(total);
+ });
+ },
+ );
+ }
+ }
+ }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
+
+struct BenchInput {
+ parquet_data: Bytes,
+ selection: RowSelection,
+}
+
+fn run_read(
+ parquet_data: &Bytes,
+ selection: &RowSelection,
+ strategy: RowSelectionStrategy,
+) -> usize {
+ let reader = ParquetRecordBatchReaderBuilder::try_new(parquet_data.clone())
+ .unwrap()
+ .with_batch_size(BATCH_SIZE)
+ .with_row_selection(selection.clone())
+ .with_row_selection_strategy(strategy)
+ .build()
+ .unwrap();
+
+ let mut total_rows = 0usize;
+ for batch in reader {
+ let batch = batch.unwrap();
+ total_rows += batch.num_rows();
+ }
+ total_rows
+}
+
+fn build_parquet_data(total_rows: usize) -> Bytes {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "value",
+ DataType::Int32,
+ false,
+ )]));
+ let values = Int32Array::from_iter_values((0..total_rows).map(|v| v as i32));
+ let columns: Vec<ArrayRef> = vec![Arc::new(values) as ArrayRef];
Review Comment:
Added in the bench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504577957
##########
parquet/src/column/reader.rs:
##########
@@ -403,7 +465,27 @@ where
/// Returns false if there's no page left.
fn read_new_page(&mut self) -> Result<bool> {
loop {
- match self.page_reader.get_next_page()? {
+ let page_result = match self.page_reader.get_next_page() {
+ Ok(page) => page,
+ Err(err) => {
+ return match err {
+ ParquetError::General(message)
+ if message
+ .starts_with("Invalid offset in sparse column chunk data:") =>
+ {
+ let metadata = self.page_reader.peek_next_page()?;
+ // Some writers omit data pages for sparse column chunks and encode the gap
+ // as a reader-visible error. Use the metadata peek to synthesise a page of
+ // null definition levels so downstream consumers see consistent row counts.
+ self.try_create_synthetic_page(metadata)?;
Review Comment:
Removed the codes for this synthetic page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2478130749
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -249,3 +254,177 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>) -> Self {
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 8;
Review Comment:
That's great idea, let me do some more investigation on different platform and put the result here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3466914651
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing rowselectionempty (14647e18080a1823ee74da79b35a1095cfcdc885) to 57447434d1921701456543f9dfb92741e5d86734 [diff](https://github.com/apache/arrow-rs/compare/57447434d1921701456543f9dfb92741e5d86734..14647e18080a1823ee74da79b35a1095cfcdc885)
BENCH_NAME=arrow_reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader
BENCH_FILTER=
BENCH_BRANCH_NAME=rowselectionempty
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3467451607
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main rowselectionempty
----- ---- -----------------
arrow_reader_clickbench/async/Q1 1.00 2.3±0.01ms ? ?/sec 1.00 2.4±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 13.2±0.46ms ? ?/sec 1.02 13.5±0.37ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.00 14.8±0.28ms ? ?/sec 1.04 15.4±0.42ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.02 28.0±0.25ms ? ?/sec 1.00 27.4±0.41ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.19 39.1±0.24ms ? ?/sec 1.00 32.8±0.35ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.22 37.0±0.30ms ? ?/sec 1.00 30.3±0.62ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.00 5.6±0.10ms ? ?/sec 1.00 5.6±0.11ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.00 133.9±14.45ms ? ?/sec 1.24 165.7±11.71ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 153.7±15.94ms ? ?/sec 1.22 187.2±19.78ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 276.6±19.55ms ? ?/sec 1.18 327.1±32.72ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.00 433.1±8.95ms ? ?/sec 1.00 433.2±2.25ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.19 44.0±0.51ms ? ?/sec 1.00 36.9±0.47ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.00 105.0±0.78ms ? ?/sec 1.02 107.0±0.55ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.00 105.2±0.54ms ? ?/sec 1.03 108.1±0.52ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.63 53.8±0.43ms ? ?/sec 1.00 33.1±0.32ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.02 125.1±0.45ms ? ?/sec 1.00 122.3±0.64ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.04 98.8±0.57ms ? ?/sec 1.00 95.3±0.61ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.00 37.2±0.28ms ? ?/sec 1.05 39.0±0.26ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.00 48.2±0.33ms ? ?/sec 1.05 50.4±0.54ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.45 45.7±1.20ms ? ?/sec 1.00 31.5±0.43ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.39 36.4±0.68ms ? ?/sec 1.00 26.1±0.36ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.12 13.7±0.17ms ? ?/sec 1.00 12.3±0.18ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q10 1.01 9.5±0.10ms ? ?/sec 1.00 9.4±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 11.0±0.12ms ? ?/sec 1.01 11.1±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.06 38.8±0.27ms ? ?/sec 1.00 36.7±2.59ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.02 49.9±0.29ms ? ?/sec 1.00 48.7±0.47ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.08 47.8±0.27ms ? ?/sec 1.00 44.4±2.40ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.01 4.3±0.03ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.00 178.4±0.91ms ? ?/sec 1.01 181.0±0.82ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.01 242.7±1.06ms ? ?/sec 1.00 239.6±1.94ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.00 484.3±3.98ms ? ?/sec 1.01 489.9±3.94ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.00 440.8±15.88ms ? ?/sec 1.00 441.8±14.12ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.10 51.6±0.80ms ? ?/sec 1.00 46.8±0.42ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.00 155.5±0.99ms ? ?/sec 1.03 159.6±0.98ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.00 151.5±0.72ms ? ?/sec 1.03 156.2±1.49ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.66 52.1±0.36ms ? ?/sec 1.00 31.4±0.39ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.00 155.1±1.31ms ? ?/sec 1.00 154.8±1.54ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.06 90.0±0.37ms ? ?/sec 1.00 85.1±0.76ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 30.0±0.17ms ? ?/sec 1.01 30.3±0.26ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.00 34.7±0.35ms ? ?/sec 1.02 35.2±0.51ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.64 44.1±0.40ms ? ?/sec 1.00 26.9±0.42ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.47 33.3±0.34ms ? ?/sec 1.00 22.6±0.36ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.15 12.8±0.14ms ? ?/sec 1.00 11.1±0.11ms ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483677029
##########
parquet/src/column/reader.rs:
##########
@@ -403,7 +465,27 @@ where
/// Returns false if there's no page left.
fn read_new_page(&mut self) -> Result<bool> {
loop {
- match self.page_reader.get_next_page()? {
+ let page_result = match self.page_reader.get_next_page() {
+ Ok(page) => page,
+ Err(err) => {
+ return match err {
+ ParquetError::General(message)
+ if message
+ .starts_with("Invalid offset in sparse column chunk data:") =>
+ {
+ let metadata = self.page_reader.peek_next_page()?;
+ // Some writers omit data pages for sparse column chunks and encode the gap
+ // as a reader-visible error. Use the metadata peek to synthesise a page of
+ // null definition levels so downstream consumers see consistent row counts.
+ self.try_create_synthetic_page(metadata)?;
Review Comment:
This feels very fragile and likely to result in weird record shredding bugs - https://github.com/apache/arrow-rs/pull/8733/files#r2483674920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483674920
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
This might be a stupid idea, but could we construct masks per-leaf column. In the case of no row-group level pruning this would just be the same as normal, but otherwise it will be a selection vector that accounts for this.
I'm a bit wary of introducing artificial pages purely because the logic around record boundaries and page boundaries is rather subtle, and whilst I don't think this will break it, it does introduce the possibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483674920
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
This might be a stupid idea, but could we construct masks per-leaf column. In the case of no row-group level pruning this would just be the same as normal, but otherwise it will be a selection vector that accounts for this.
I'm a bit wary of introducing artificial pages purely because the logic around record boundaries and page boundaries is rather subtle, and whilst I don't think this will break it, it does introduce the possibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483678349
##########
parquet/src/column/reader.rs:
##########
@@ -541,6 +626,44 @@ where
}
}
+ fn try_create_synthetic_page(&mut self, metadata: Option<PageMetadata>) -> Result<()> {
+ if self.descr.max_rep_level() != 0 {
+ return Err(general_err!(
+ "cannot synthesise sparse page for column with repetition levels ({message})"
+ ));
+ }
+
+ if self.descr.max_def_level() == 0 {
+ return Err(general_err!(
+ "cannot synthesise sparse page for required column ({message})"
+ ));
+ }
Review Comment:
I think this would mean we error if we try to pushdown on a column with either repetition levels or a required column - this seems like quite a major regression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483701565
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
Review Comment:
~This will likely lead to OOMs for any non-trivial parquet file - it will fetch the entire selector (i.e. the entire row group) into memory and then filter it.~
Edit: Nvm mixed up MaskChunk and SelectionCursor
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
I think this will cause the reader to produce batches potentially much smaller than the requested batch size, which at the very least should be documented. Ideally we'd just store the mask somewhere, and flush + filter when we have enough rows.
Otherwise consumers like DataFusion are going to need to wrap ParquetExec in a CoalesceBatches and effectively buffer all the data again.
Edit edit: Filtering small batches may also have performance implications - the filter kernels are optimized for filtering thousands of rows at a time.
Edit edit edit: actually realised the selector is for the entire row group!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483818627
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
The error comes from my directing test on a parquet, let me added new tests for the scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2480526458
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1035,8 +1050,85 @@ impl ParquetRecordBatchReader {
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
Some(selection) => {
+ if selection.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ loop {
+ let mask_chunk = match selection.next_mask_chunk(batch_size) {
+ Some(batch) => batch,
+ None => return Ok(None),
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
+ // The column reader exposes the projection as a struct array; convert this
+ // into a record batch before applying the boolean filter mask.
+ let struct_array = array.as_struct_opt().ok_or_else(|| {
+ ArrowError::ParquetError(
+ "Struct array reader should return struct array".to_string(),
+ )
+ })?;
+
+ let filtered_batch =
+ filter_record_batch(&RecordBatch::from(struct_array), &mask)?;
+
+ if filtered_batch.num_rows() != mask_chunk.selected_rows {
+ return Err(general_err!(
+ "filtered rows mismatch selection - expected {}, got {}",
+ mask_chunk.selected_rows,
+ filtered_batch.num_rows()
+ ));
+ }
+
+ if filtered_batch.num_rows() == 0 {
+ if selection.is_empty() {
Review Comment:
Yes, updated to use while loop to check empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2477352297
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -249,3 +254,177 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>) -> Self {
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 8;
Review Comment:
@alamb It looks like similar to my original implementation which is fixed for choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2478057648
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -249,3 +254,177 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>) -> Self {
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 8;
Review Comment:
I added bench in the code, you can also try on your machine. I find it varies heavily on different platform, on my Mac, it's 8, but on my x86 PC, the value can be set to around 30.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2479120538
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1035,8 +1050,85 @@ impl ParquetRecordBatchReader {
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
Some(selection) => {
+ if selection.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ loop {
+ let mask_chunk = match selection.next_mask_chunk(batch_size) {
+ Some(batch) => batch,
+ None => return Ok(None),
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
+ // The column reader exposes the projection as a struct array; convert this
+ // into a record batch before applying the boolean filter mask.
+ let struct_array = array.as_struct_opt().ok_or_else(|| {
+ ArrowError::ParquetError(
+ "Struct array reader should return struct array".to_string(),
+ )
+ })?;
+
+ let filtered_batch =
+ filter_record_batch(&RecordBatch::from(struct_array), &mask)?;
+
+ if filtered_batch.num_rows() != mask_chunk.selected_rows {
+ return Err(general_err!(
+ "filtered rows mismatch selection - expected {}, got {}",
+ mask_chunk.selected_rows,
+ filtered_batch.num_rows()
+ ));
+ }
+
+ if filtered_batch.num_rows() == 0 {
+ if selection.is_empty() {
Review Comment:
would it be better to check for empty selection before trying to apply filters ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3467321221
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing rowselectionempty (14647e18080a1823ee74da79b35a1095cfcdc885) to 57447434d1921701456543f9dfb92741e5d86734 [diff](https://github.com/apache/arrow-rs/compare/57447434d1921701456543f9dfb92741e5d86734..14647e18080a1823ee74da79b35a1095cfcdc885)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader_clickbench
BENCH_FILTER=
BENCH_BRANCH_NAME=rowselectionempty
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483674920
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
This might be a stupid idea, but could we construct masks per-leaf column. In the case of no row-group level pruning this would just be the same as normal, but otherwise it will be a selection vector that accounts for this.
I'm a bit wary of introducing artificial pages purely because the logic around record boundaries and page boundaries is rather subtle, and whilst I don't think this will break it, it does introduce the possibility.
Edit: In fact I don't think you need this, you just need to make sure that the mask selections can't "reverse" the RowSelection provided by the user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483674920
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
This might be a stupid idea, but could we construct masks per-leaf column. In the case of no row-group level pruning this would just be the same as normal, but otherwise it will be a selection vector that accounts for this.
I'm a bit wary of introducing artificial pages purely because the logic around record boundaries and page boundaries is rather subtle, and whilst I don't think this will break it, it does introduce the possibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2484771487
##########
parquet/src/column/reader.rs:
##########
@@ -214,6 +219,49 @@ where
let remaining_records = max_records - total_records_read;
let remaining_levels = self.num_buffered_values - self.num_decoded_values;
+ if self.synthetic_page {
Review Comment:
I have been thinking about how to test this scenario and have some ideas (it is probably time to do some fuzz testing / testing with very selective predicates). I hope to help write some additional tests later this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2483702970
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1034,9 +1049,82 @@ impl ParquetRecordBatchReader {
let mut read_records = 0;
let batch_size = self.batch_size();
match self.read_plan.selection_mut() {
- Some(selection) => {
- while read_records < batch_size && !selection.is_empty() {
- let front = selection.pop_front().unwrap();
+ Some(selection_cursor) => {
+ if selection_cursor.is_mask_backed() {
+ // Stream the record batch reader using contiguous segments of the selection
+ // mask, avoiding the need to materialize intermediate `RowSelector` ranges.
+ while !selection_cursor.is_empty() {
+ let Some(mask_chunk) = selection_cursor.next_mask_chunk(batch_size) else {
+ return Ok(None);
+ };
+
+ if mask_chunk.initial_skip > 0 {
+ let skipped =
+ self.array_reader.skip_records(mask_chunk.initial_skip)?;
+ if skipped != mask_chunk.initial_skip {
+ return Err(general_err!(
+ "failed to skip rows, expected {}, got {}",
+ mask_chunk.initial_skip,
+ skipped
+ ));
+ }
+ }
+
+ if mask_chunk.chunk_rows == 0 {
+ if selection_cursor.is_empty() && mask_chunk.selected_rows == 0 {
+ return Ok(None);
+ }
+ continue;
+ }
+
+ let mask = selection_cursor
+ .mask_values_for(&mask_chunk)
+ .ok_or_else(|| general_err!("row selection mask out of bounds"))?;
+
+ let read = self.array_reader.read_records(mask_chunk.chunk_rows)?;
+ if read == 0 {
+ return Err(general_err!(
+ "reached end of column while expecting {} rows",
+ mask_chunk.chunk_rows
+ ));
+ }
+ if read != mask_chunk.chunk_rows {
+ return Err(general_err!(
+ "insufficient rows read from array reader - expected {}, got {}",
+ mask_chunk.chunk_rows,
+ read
+ ));
+ }
+
+ let array = self.array_reader.consume_batch()?;
Review Comment:
Actually I realise now that MaskChunk ensures that we read as much as is necessary to satisfy the chunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2506350668
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -20,19 +20,70 @@
use crate::arrow::array_reader::ArrayReader;
use crate::arrow::arrow_reader::{
- ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
+ ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelectionCursor,
+ RowSelectionStrategy, RowSelector,
};
use crate::errors::{ParquetError, Result};
use arrow_array::Array;
use arrow_select::filter::prep_null_mask_filter;
-use std::collections::VecDeque;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+// The average selector length threshold for choosing between
+// `RowSelectionStrategy::Mask` and `RowSelectionStrategy::Selectors`.
+// If the average selector length is less than this value,
+// `RowSelectionStrategy::Mask` is preferred.
+const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 32;
+
+// The logic in `preferred_selection_strategy` depends on the constant
+// `AVG_SELECTOR_LEN_MASK_THRESHOLD`. To allow unit testing of this logic,
+// we use a mutable global variable that can be temporarily changed during tests.
+//
+// An `AtomicUsize` is used because the Rust test runner (`cargo test`) runs tests
+// in parallel by default. The atomic operations prevent data races between
+// different test threads that might try to modify this value simultaneously.
+//
+// For the production code path, `load(Ordering::Relaxed)` is used. This is the
+// weakest memory ordering and for a simple load on most modern architectures,
+// it compiles down to a regular memory read with negligible performance overhead.
+// The more expensive atomic operations with stronger ordering are only used in the
+// test-only functions below.
+static AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE: AtomicUsize =
+ AtomicUsize::new(AVG_SELECTOR_LEN_MASK_THRESHOLD);
+
+#[inline(always)]
+fn avg_selector_len_mask_threshold() -> usize {
+ AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.load(Ordering::Relaxed)
+}
+
+/// An RAII guard that restores the previous value of the override when it is dropped.
+/// This ensures that any change to the global threshold is temporary and scoped to
+/// the test or benchmark where it's used, even in the case of a panic.
+pub struct AvgSelectorLenMaskThresholdGuard {
+ previous: usize,
+}
+
+impl Drop for AvgSelectorLenMaskThresholdGuard {
+ fn drop(&mut self) {
+ AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.store(self.previous, Ordering::SeqCst);
+ }
+}
+
+/// Override AVG_SELECTOR_LEN_MASK_THRESHOLD (primarily for tests / benchmarks).
+///
+/// Returns an [`AvgSelectorLenMaskThresholdGuard`] that restores the previous value on drop.
+pub fn set_avg_selector_len_mask_threshold(value: usize) -> AvgSelectorLenMaskThresholdGuard {
+ let previous = AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE.swap(value, Ordering::SeqCst);
+ AvgSelectorLenMaskThresholdGuard { previous }
+}
/// A builder for [`ReadPlan`]
#[derive(Clone, Debug)]
pub struct ReadPlanBuilder {
batch_size: usize,
/// Current to apply, includes all filters
selection: Option<RowSelection>,
+ /// Strategy to use when materialising the row selection
+ selection_strategy: RowSelectionStrategy,
Review Comment:
I removed `Auto` from the current design and didn't add it to the Builder API because `Mask` is currently unstable, requiring us to fall back to `Selection` in certain cases. I believe this unstable setting shouldn't be directly exposed to users. Moving forward, we could consider:
1. Allowing users to choose only `Auto`, `Selection`, or other stable strategies, implemented via runtime checks or a new enum. `Mask` would only be selected heuristically when `Auto` is chosen.
2. Stabilizing `Mask` by updating the fetch process, ensuring all pages are fetched if the user selects `Mask`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2506342419
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -20,19 +20,70 @@
use crate::arrow::array_reader::ArrayReader;
use crate::arrow::arrow_reader::{
- ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelector,
+ ArrowPredicate, ParquetRecordBatchReader, RowSelection, RowSelectionCursor,
+ RowSelectionStrategy, RowSelector,
};
use crate::errors::{ParquetError, Result};
use arrow_array::Array;
use arrow_select::filter::prep_null_mask_filter;
-use std::collections::VecDeque;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+// The average selector length threshold for choosing between
+// `RowSelectionStrategy::Mask` and `RowSelectionStrategy::Selectors`.
+// If the average selector length is less than this value,
+// `RowSelectionStrategy::Mask` is preferred.
+const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 32;
+
+// The logic in `preferred_selection_strategy` depends on the constant
+// `AVG_SELECTOR_LEN_MASK_THRESHOLD`. To allow unit testing of this logic,
+// we use a mutable global variable that can be temporarily changed during tests.
+//
+// An `AtomicUsize` is used because the Rust test runner (`cargo test`) runs tests
+// in parallel by default. The atomic operations prevent data races between
+// different test threads that might try to modify this value simultaneously.
+//
+// For the production code path, `load(Ordering::Relaxed)` is used. This is the
+// weakest memory ordering and for a simple load on most modern architectures,
+// it compiles down to a regular memory read with negligible performance overhead.
+// The more expensive atomic operations with stronger ordering are only used in the
+// test-only functions below.
+static AVG_SELECTOR_LEN_MASK_THRESHOLD_OVERRIDE: AtomicUsize =
Review Comment:
This is awesome! I've been struggling for days to implement the bench, and I never would have thought of using the `RowSelectionStrategy` enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504557120
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -248,3 +275,190 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>, strategy: RowSelectionStrategy) -> Self {
+ if matches!(strategy, RowSelectionStrategy::Selectors) {
+ return Self {
+ storage: RowSelectionBacking::Selectors(selectors.into()),
+ position: 0,
+ };
+ }
+
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 16;
+ // Prefer a bitmap mask when the selectors are short on average, as the mask
+ // (re)construction cost is amortized by a simpler execution path during reads.
+ let use_mask = match strategy {
+ RowSelectionStrategy::Mask => true,
+ RowSelectionStrategy::Auto => {
+ selector_count == 0
+ || total_rows < selector_count.saturating_mul(AVG_SELECTOR_LEN_MASK_THRESHOLD)
+ }
+ RowSelectionStrategy::Selectors => unreachable!(),
+ };
+
+ let storage = if use_mask {
+ RowSelectionBacking::Mask(boolean_mask_from_selectors(&selectors))
+ } else {
+ RowSelectionBacking::Selectors(selectors.into())
+ };
+
+ Self {
+ storage,
+ position: 0,
+ }
+ }
+
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ match &self.storage {
+ RowSelectionBacking::Mask(mask) => self.position >= mask.len(),
+ RowSelectionBacking::Selectors(selectors) => selectors.is_empty(),
+ }
+ }
+
+ /// Current position within the overall selection
+ pub fn position(&self) -> usize {
+ self.position
+ }
+
+ /// Return the next [`RowSelector`] when using the sparse representation
+ pub fn next_selector(&mut self) -> Option<RowSelector> {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ let selector = selectors.pop_front()?;
+ self.position += selector.row_count;
+ Some(selector)
+ }
+ RowSelectionBacking::Mask(_) => None,
+ }
+ }
+
+ /// Return a selector to the front, rewinding the position (sparse-only)
+ pub fn return_selector(&mut self, selector: RowSelector) {
+ match &mut self.storage {
+ RowSelectionBacking::Selectors(selectors) => {
+ self.position = self.position.saturating_sub(selector.row_count);
+ selectors.push_front(selector);
+ }
+ RowSelectionBacking::Mask(_) => {
+ unreachable!("return_selector called for mask-based RowSelectionCursor")
+ }
+ }
+ }
+
+ /// Returns `true` if the cursor is backed by a boolean mask
+ pub fn is_mask_backed(&self) -> bool {
+ matches!(self.storage, RowSelectionBacking::Mask(_))
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = match &self.storage {
+ RowSelectionBacking::Mask(mask) => mask,
+ RowSelectionBacking::Selectors(_) => return None,
+ };
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
Review Comment:
I'll learn the https://github.com/apache/arrow-rs/pull/6624#discussion_r2127391600 and try to improve it in the following PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#discussion_r2504548028
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -249,3 +254,177 @@ impl ReadPlan {
self.batch_size
}
}
+
+/// Cursor for iterating a [`RowSelection`] during execution within a [`ReadPlan`].
+///
+/// This keeps per-reader state such as the current position and delegates the
+/// actual storage strategy to [`RowSelectionBacking`].
+#[derive(Debug)]
+pub struct RowSelectionCursor {
+ /// Backing storage describing how the selection is materialised
+ storage: RowSelectionBacking,
+ /// Current absolute offset into the selection
+ position: usize,
+}
+
+/// Backing storage that powers [`RowSelectionCursor`].
+///
+/// The cursor either walks a boolean mask (dense representation) or a queue
+/// of [`RowSelector`] ranges (sparse representation).
+#[derive(Debug)]
+enum RowSelectionBacking {
+ Mask(BooleanBuffer),
+ Selectors(VecDeque<RowSelector>),
+}
+
+/// Result of computing the next chunk to read when using a bitmap mask
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+impl RowSelectionCursor {
+ /// Create a cursor, choosing an efficient backing representation
+ fn new(selectors: Vec<RowSelector>) -> Self {
+ let total_rows: usize = selectors.iter().map(|s| s.row_count).sum();
+ let selector_count = selectors.len();
+ const AVG_SELECTOR_LEN_MASK_THRESHOLD: usize = 8;
Review Comment:
Added here: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3504925982
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing rowselectionempty (fe238774fd3c2e0202e7328c6d1e3174afbd79a1) to d379b981768215b16182f78aebddf940a4b45c99 [diff](https://github.com/apache/arrow-rs/compare/d379b981768215b16182f78aebddf940a4b45c99..fe238774fd3c2e0202e7328c6d1e3174afbd79a1)
BENCH_NAME=arrow_reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader
BENCH_FILTER=
BENCH_BRANCH_NAME=rowselectionempty
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3505197659
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing rowselectionempty (fe238774fd3c2e0202e7328c6d1e3174afbd79a1) to d379b981768215b16182f78aebddf940a4b45c99 [diff](https://github.com/apache/arrow-rs/compare/d379b981768215b16182f78aebddf940a4b45c99..fe238774fd3c2e0202e7328c6d1e3174afbd79a1)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader_clickbench
BENCH_FILTER=
BENCH_BRANCH_NAME=rowselectionempty
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Adaptive Parquet Predicate Pushdown [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8733:
URL: https://github.com/apache/arrow-rs/pull/8733#issuecomment-3518382226
I plan to leave this PR open for a few more days to gather any additional comments and then merge it in
I may also do some prototyping of additional optimizations (avoiding the masks and fetches entirely) for fun
Thank you again @hhhizzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org