You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/08/04 22:46:57 UTC
[PR] POC: page-granular scan planning for the Parquet push decoder [arrow-rs]
adriangb opened a new pull request, #10555:
URL: https://github.com/apache/arrow-rs/pull/10555
# Which issue does this PR close?
- Part of #6946 (Avoid buffering the entire row group in Async parquet reader)
- Related: #8668 (PushDecoder: Add a peek API to support pre-fetching)
# Rationale for this change
`ParquetPushDecoder` reports what it needs one **row group** at a time: `DecodeResult::NeedsData` does not resolve until every projected byte of the row group is buffered. That is the right granularity when the caller wants a whole row group's reader, but for a caller scheduling its own I/O it means:
- resident bytes are bounded by row-group size, not by anything the caller chooses;
- decoding cannot start until the last byte of the row group lands, so time-to-first-batch is a whole row group of latency; and
- there is no way to ask "what does the *next batch* need?", which is the question a scheduler actually has.
This adds `plan_scan_ranges`, which decomposes a scan into the individual pages it will read, in the order decoding needs them, each tagged with the span of selected rows it serves. A caller can then fetch only what the next batch needs, drop pages once its decode cursor passes them, and read ahead as far as its own byte budget allows.
The plan is **demand, not schedule**: it says which bytes the query will read and when they are first needed. Request sizing, readahead depth and whether to merge nearby ranges stay caller policy, because they depend on the storage medium rather than on the file. (`ObjectStore::get_ranges` already coalesces for stores that use the default implementation, which is one reason this deliberately does not.)
## Units: why row tags rather than a fixed quantum
The obvious shape would be `next_batch_ranges()` or `ranges_for(n_rows)`. Both bake in a unit the caller may not want. A fixed row count can imply an unbounded byte count — 8192 rows of 1MB values is 8GB — while a pure byte budget cannot express "enough to make progress".
So each planned range carries `first_row` / `last_row` in selected-row space instead. A caller budgeting in bytes takes pages until the budget fills; a caller thinking in rows takes pages until the tags cover its window; and `last_row` is what makes eviction possible at all.
One floor remains and is documented: `ParquetRecordBatchReader` decodes `batch_size` rows at a time, so no fetch plan can subdivide a batch. For schemas with very large values the lever is a smaller `batch_size`, or byte-based batch sizing, which is a separate gap.
# What changes are included in this PR?
A single new module, `parquet::arrow::push_decoder::scan_plan`, exporting `plan_scan_ranges`, `ScanPlan` and `PlannedRange`. No existing code path is touched — nothing calls it inside the crate yet, so this is purely additive.
`plan_scan_ranges(metadata, row_groups, projection, selection)` returns `None` when the file has no offset index, since page locations are what make page-granular planning possible; callers fall back to row-group-granular fetching.
The natural follow-up, if this shape is acceptable, is to use it inside the decoder itself so `NeedsData` can resolve at batch granularity rather than row-group granularity, and to make `RowFilter` evaluation incremental (today predicates are evaluated eagerly over a whole row group inside `build()`, which is why the DataFusion side below falls back to row-group granularity whenever filter pushdown is on).
# Are these changes tested?
Yes — six unit tests covering page ordering, projection, selection-driven page skipping (including selected-row-space tagging), row-group subsets, the no-offset-index fallback, and that planned bytes exactly cover the projected column chunks when nothing is skipped.
It is also exercised end to end by a DataFusion POC that schedules I/O against this plan: apache/datafusion#24086. Measured there against DataFusion `main`, with simulated object-store latency:
| benchmark | baseline | with batch-granular scheduling |
|---|---|---|
| TPC-H SF10 | 635.1s | **100.0s** (22 of 22 queries faster) |
| TPC-H SF1 | 110.0s | **65.0s** (21 of 22 faster, 0 slower) |
| TPC-DS SF1 | 380.1s | 380.1s, peak memory −18% |
| single 300MB row group | 845ms, 700ms TTFB, 283MB peak | **708ms, 112ms TTFB, 100MB peak** |
ClickBench is unchanged there — because the published `hits_*.parquet` files contain **no page index at all** (0 of 105 columns), so page-granular planning cannot run on them. Rewriting three of those files with `write_page_index=True` and changing nothing else makes the same queries **1.54x faster** (36 queries: 35 faster, 0 slower, 1 neutral), which is a reasonable argument that the capability is worth having even where files do not use it yet.
# Are there any user-facing changes?
New public API only (`plan_scan_ranges`, `ScanPlan`, `PlannedRange`); no behavior change to existing paths. Marked draft because the API shape is the thing worth discussing — in particular whether this belongs as a free function or as a method on the decoder that can also advance it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6084700500
Its in@
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5944816142
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5944634999)
**Target:** push_decoder
Comparing claude/push-decoder-peek-59 (a1a2c7e383cdf90c89b1b85ef57e0822445fa2c0) to 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81 [diff](https://github.com/apache/arrow-rs/compare/0d91bca5e174821c56fc6529c6bfb6d6a52e8d81..a1a2c7e383cdf90c89b1b85ef57e0822445fa2c0)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
env:
BENCH_FILTER: scan_plan
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
push_decoder/scan_plan/first_range/all/10000cols_10pages 1.00 377.1±10.11µs ? ?/sec 1.12 423.0±4.10µs ? ?/sec
push_decoder/scan_plan/first_range/all/1000cols_100pages 1.00 35.9±0.12µs ? ?/sec 1.13 40.6±0.16µs ? ?/sec
push_decoder/scan_plan/first_range/all/1000cols_10pages 1.00 35.9±0.10µs ? ?/sec 1.12 40.3±0.15µs ? ?/sec
push_decoder/scan_plan/first_range/all/100cols_10pages 1.00 4.1±0.02µs ? ?/sec 1.15 4.7±0.01µs ? ?/sec
push_decoder/scan_plan/first_range/narrow/10000cols_10pages 1.28 8.0±0.01µs ? ?/sec 1.00 6.3±0.01µs ? ?/sec
push_decoder/scan_plan/first_range/narrow/1000cols_100pages 1.04 1585.2±5.00ns ? ?/sec 1.00 1521.7±2.57ns ? ?/sec
push_decoder/scan_plan/first_range/narrow/1000cols_10pages 1.03 1569.7±5.66ns ? ?/sec 1.00 1518.9±3.05ns ? ?/sec
push_decoder/scan_plan/first_range/narrow/100cols_10pages 1.00 962.5±5.15ns ? ?/sec 1.09 1047.9±2.50ns ? ?/sec
push_decoder/scan_plan/first_range/selection/10000cols_10pages 1.10 2.7±0.03ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
push_decoder/scan_plan/first_range/selection/1000cols_100pages 1.07 376.6±5.66µs ? ?/sec 1.00 352.1±0.28µs ? ?/sec
push_decoder/scan_plan/first_range/selection/1000cols_10pages 1.01 240.9±0.96µs ? ?/sec 1.00 238.8±0.26µs ? ?/sec
push_decoder/scan_plan/first_range/selection/100cols_10pages 1.03 24.9±0.10µs ? ?/sec 1.00 24.3±0.04µs ? ?/sec
push_decoder/scan_plan/first_reader/all/10000cols_10pages 1.00 4.4±0.20ms ? ?/sec
push_decoder/scan_plan/first_reader/all/1000cols_100pages 1.00 616.4±1.68µs ? ?/sec
push_decoder/scan_plan/first_reader/all/1000cols_10pages 1.00 432.3±1.67µs ? ?/sec
push_decoder/scan_plan/first_reader/all/100cols_10pages 1.00 43.1±0.09µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/10000cols_10pages 1.00 176.2±0.54µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/1000cols_100pages 1.00 22.3±0.03µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/1000cols_10pages 1.00 21.6±0.03µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/100cols_10pages 1.00 6.4±0.01µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/10000cols_10pages 1.00 9.7±0.08ms ? ?/sec
push_decoder/scan_plan/first_reader/selection/1000cols_100pages 1.00 1227.3±3.63µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/1000cols_10pages 1.00 960.0±4.87µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/100cols_10pages 1.00 93.6±1.60µs ? ?/sec
push_decoder/scan_plan/whole_scan/all/10000cols_10pages 1.00 7.0±0.03ms ? ?/sec 1.01 7.1±0.01ms ? ?/sec
push_decoder/scan_plan/whole_scan/all/1000cols_100pages 1.00 6.5±0.01ms ? ?/sec 1.00 6.5±0.01ms ? ?/sec
push_decoder/scan_plan/whole_scan/all/1000cols_10pages 1.00 628.9±10.12µs ? ?/sec 1.00 628.3±0.89µs ? ?/sec
push_decoder/scan_plan/whole_scan/all/100cols_10pages 1.00 33.2±0.05µs ? ?/sec 1.04 34.5±0.10µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/10000cols_10pages 1.11 10.6±0.01µs ? ?/sec 1.00 9.5±0.54µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/1000cols_100pages 1.00 27.3±0.05µs ? ?/sec 1.01 27.7±0.05µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/1000cols_10pages 1.00 4.1±0.00µs ? ?/sec 1.00 4.2±0.01µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/100cols_10pages 1.00 3.5±0.01µs ? ?/sec 1.05 3.7±0.01µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/10000cols_10pages 1.00 9.7±0.03ms ? ?/sec 1.00 9.7±0.08ms ? ?/sec
push_decoder/scan_plan/whole_scan/selection/1000cols_100pages 1.02 1048.9±3.78µs ? ?/sec 1.00 1025.6±4.91µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/1000cols_10pages 1.00 899.6±2.91µs ? ?/sec 1.00 896.3±1.40µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/100cols_10pages 1.02 61.5±0.28µs ? ?/sec 1.00 60.1±0.16µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>push_decoder — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1h21l1mx-74cc708c-q9bk
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5944634999-3015-fh9zl 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench push_decoder
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.9 GiB |
| CPU user | 369.9s |
| CPU sys | 1.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 250.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.9 GiB |
| CPU user | 244.4s |
| CPU sys | 0.8s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5944634999:finish:f1bcabc059da0e4065a583a01250a8c8 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5944634999
run benchmark push_decoder
```yaml
env:
BENCH_FILTER: scan_plan
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6040190549
run benchmark arrow_reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940186940
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940184708)
**Target:** arrow_reader
**Sharding factor: 1** (1 workers).
Comparing claude/push-decoder-peek-59 (0d91bca5e174821c56fc6529c6bfb6d6a52e8d81) to 76829d5 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/76829d5ce4562616a69ea8b2696e96160514482f..0d91bca5e174821c56fc6529c6bfb6d6a52e8d81)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5940184708:start:4c7c4e064dc60eb375e9bfd6ec1d37aa -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#discussion_r4160254352
##########
parquet/benches/push_decoder.rs:
##########
@@ -166,5 +173,106 @@ fn bench_nbuf(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_1buf, bench_nbuf);
+/// Write a Parquet file with `num_columns` columns and one row group of
+/// 1,000 rows, with `pages_per_column` data pages per column chunk.
+fn make_paged_test_file(num_columns: usize, pages_per_column: usize) -> Bytes {
+ let num_rows = 1_000;
+ let schema = make_wide_schema(num_columns);
+ let columns: Vec<Arc<dyn arrow_array::Array>> = (0..num_columns)
+ .map(|_| {
+ Arc::new(Float32Array::from_iter_values(
+ (0..num_rows).map(|v| v as f32),
+ )) as _
+ })
+ .collect();
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let page_rows = num_rows / pages_per_column;
+ let mut buf = Vec::new();
+ let props = WriterProperties::builder()
+ .set_max_row_group_row_count(Some(num_rows))
+ .set_data_page_row_count_limit(page_rows)
+ // Page limits are checked between write batches.
+ .set_write_batch_size(page_rows)
+ .set_dictionary_enabled(false)
+ .build();
+ let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buf)
+}
+
+type BuilderFn<'a> = Box<dyn Fn() -> ParquetPushDecoderBuilder + 'a>;
+
+/// Plan a wide file with a page index: the first range (what a read-ahead
+/// caller waits for) and the whole scan, which is one row group. For
+/// comparison, `first_reader` builds a decoder, pushes the whole file and
+/// builds the first row group reader, which includes the decoder's own
+/// per-row-group setup.
+///
+/// `all` reads every column and row. `selection` keeps 10 rows of every 100,
+/// so each column chunk reads every other page. `narrow` reads 10 columns.
+fn bench_scan_plan(c: &mut Criterion) {
+ let mut group = c.benchmark_group("push_decoder/scan_plan");
+
+ for (num_cols, pages) in [(100, 10), (1_000, 10), (10_000, 10), (1_000, 100)] {
+ let file_data = make_paged_test_file(num_cols, pages);
+ let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
+ let metadata = ArrowReaderMetadata::load(&file_data, options).unwrap();
+ let selection = RowSelection::from(
+ (0..10)
+ .flat_map(|_| [RowSelector::select(10), RowSelector::skip(90)])
+ .collect::<Vec<_>>(),
+ );
+ let narrow = ProjectionMask::leaves(metadata.parquet_schema(), 0..10);
+ let variants: [(&str, BuilderFn); 3] = [
+ (
+ "all",
+ Box::new(|| ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())),
Review Comment:
do you need the function? I think cloning an existing builder would be simpler and easier to understand
##########
parquet/benches/push_decoder.rs:
##########
@@ -166,5 +173,106 @@ fn bench_nbuf(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_1buf, bench_nbuf);
+/// Write a Parquet file with `num_columns` columns and one row group of
+/// 1,000 rows, with `pages_per_column` data pages per column chunk.
+fn make_paged_test_file(num_columns: usize, pages_per_column: usize) -> Bytes {
+ let num_rows = 1_000;
+ let schema = make_wide_schema(num_columns);
+ let columns: Vec<Arc<dyn arrow_array::Array>> = (0..num_columns)
+ .map(|_| {
+ Arc::new(Float32Array::from_iter_values(
+ (0..num_rows).map(|v| v as f32),
+ )) as _
+ })
+ .collect();
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let page_rows = num_rows / pages_per_column;
+ let mut buf = Vec::new();
+ let props = WriterProperties::builder()
+ .set_max_row_group_row_count(Some(num_rows))
+ .set_data_page_row_count_limit(page_rows)
+ // Page limits are checked between write batches.
+ .set_write_batch_size(page_rows)
+ .set_dictionary_enabled(false)
+ .build();
+ let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buf)
+}
+
+type BuilderFn<'a> = Box<dyn Fn() -> ParquetPushDecoderBuilder + 'a>;
+
+/// Plan a wide file with a page index: the first range (what a read-ahead
+/// caller waits for) and the whole scan, which is one row group. For
+/// comparison, `first_reader` builds a decoder, pushes the whole file and
+/// builds the first row group reader, which includes the decoder's own
+/// per-row-group setup.
+///
+/// `all` reads every column and row. `selection` keeps 10 rows of every 100,
+/// so each column chunk reads every other page. `narrow` reads 10 columns.
+fn bench_scan_plan(c: &mut Criterion) {
+ let mut group = c.benchmark_group("push_decoder/scan_plan");
+
+ for (num_cols, pages) in [(100, 10), (1_000, 10), (10_000, 10), (1_000, 100)] {
+ let file_data = make_paged_test_file(num_cols, pages);
+ let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
+ let metadata = ArrowReaderMetadata::load(&file_data, options).unwrap();
+ let selection = RowSelection::from(
+ (0..10)
+ .flat_map(|_| [RowSelector::select(10), RowSelector::skip(90)])
+ .collect::<Vec<_>>(),
+ );
+ let narrow = ProjectionMask::leaves(metadata.parquet_schema(), 0..10);
+ let variants: [(&str, BuilderFn); 3] = [
+ (
+ "all",
+ Box::new(|| ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())),
+ ),
+ (
+ "selection",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_row_selection(selection.clone())
+ }),
+ ),
+ (
+ "narrow",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_projection(narrow.clone())
+ }),
+ ),
+ ];
+
+ for (variant, builder) in &variants {
+ let decoder = builder().build().unwrap();
+ let id = format!("{variant}/{num_cols}cols_{pages}pages");
+
+ group.bench_function(BenchmarkId::new("first_range", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().next()))
+ });
+ group.bench_function(BenchmarkId::new("whole_scan", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().count()))
Review Comment:
maybe it is worth a comment here that the scan plan is lazily evaluated (so counting forces it to be evaluated)?
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -822,6 +753,14 @@ impl RowGroupReaderBuilder {
Ok(result)
}
+ /// A [`ScanPlanBuilder`] that plans the same ranges as this builder, for
Review Comment:
could the same thing be accomplished via `self.clone().into_builder()` instead of a new API? Since this is crate local I think it is fine, I am just asking
##########
parquet/benches/push_decoder.rs:
##########
@@ -166,5 +173,106 @@ fn bench_nbuf(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_1buf, bench_nbuf);
+/// Write a Parquet file with `num_columns` columns and one row group of
+/// 1,000 rows, with `pages_per_column` data pages per column chunk.
+fn make_paged_test_file(num_columns: usize, pages_per_column: usize) -> Bytes {
+ let num_rows = 1_000;
+ let schema = make_wide_schema(num_columns);
+ let columns: Vec<Arc<dyn arrow_array::Array>> = (0..num_columns)
+ .map(|_| {
+ Arc::new(Float32Array::from_iter_values(
+ (0..num_rows).map(|v| v as f32),
+ )) as _
+ })
+ .collect();
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let page_rows = num_rows / pages_per_column;
+ let mut buf = Vec::new();
+ let props = WriterProperties::builder()
+ .set_max_row_group_row_count(Some(num_rows))
+ .set_data_page_row_count_limit(page_rows)
+ // Page limits are checked between write batches.
+ .set_write_batch_size(page_rows)
+ .set_dictionary_enabled(false)
+ .build();
+ let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buf)
+}
+
+type BuilderFn<'a> = Box<dyn Fn() -> ParquetPushDecoderBuilder + 'a>;
+
+/// Plan a wide file with a page index: the first range (what a read-ahead
+/// caller waits for) and the whole scan, which is one row group. For
+/// comparison, `first_reader` builds a decoder, pushes the whole file and
+/// builds the first row group reader, which includes the decoder's own
+/// per-row-group setup.
+///
+/// `all` reads every column and row. `selection` keeps 10 rows of every 100,
+/// so each column chunk reads every other page. `narrow` reads 10 columns.
+fn bench_scan_plan(c: &mut Criterion) {
+ let mut group = c.benchmark_group("push_decoder/scan_plan");
+
+ for (num_cols, pages) in [(100, 10), (1_000, 10), (10_000, 10), (1_000, 100)] {
+ let file_data = make_paged_test_file(num_cols, pages);
+ let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
+ let metadata = ArrowReaderMetadata::load(&file_data, options).unwrap();
+ let selection = RowSelection::from(
+ (0..10)
+ .flat_map(|_| [RowSelector::select(10), RowSelector::skip(90)])
+ .collect::<Vec<_>>(),
+ );
+ let narrow = ProjectionMask::leaves(metadata.parquet_schema(), 0..10);
+ let variants: [(&str, BuilderFn); 3] = [
+ (
+ "all",
+ Box::new(|| ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())),
+ ),
+ (
+ "selection",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_row_selection(selection.clone())
+ }),
+ ),
+ (
+ "narrow",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_projection(narrow.clone())
+ }),
+ ),
+ ];
+
+ for (variant, builder) in &variants {
+ let decoder = builder().build().unwrap();
+ let id = format!("{variant}/{num_cols}cols_{pages}pages");
+
+ group.bench_function(BenchmarkId::new("first_range", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().next()))
+ });
+ group.bench_function(BenchmarkId::new("whole_scan", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().count()))
+ });
+ group.bench_function(BenchmarkId::new("first_reader", &id), |b| {
+ b.iter(|| {
Review Comment:
this benchmark seems to have nothing to do with scan_plan. Why included it?
##########
parquet/src/arrow/push_decoder/scan_plan/frontier.rs:
##########
@@ -0,0 +1,494 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Which row groups a scan reads next, with their selections and budget.
+
+use crate::arrow::arrow_reader::{RowGroupPlan, RowGroupSelection, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::ParquetMetaData;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use super::budget::RowBudget;
+
+/// Plan for the next queued row group after row-selection slicing.
+#[derive(Debug)]
+enum QueuedRowGroupDecision {
+ /// Hand this row group to the builder.
+ Read(NextRowGroup),
+ /// Skip this row group, and keep scanning with the updated budget.
+ Skip { remaining_budget: RowBudget },
+}
+
+/// Work item handed from [`RowGroupFrontier`] to [`RowGroupReaderBuilder`](crate::arrow::push_decoder::reader_builder::RowGroupReaderBuilder).
+#[derive(Debug, Clone)]
+pub(crate) struct NextRowGroup {
Review Comment:
these are just moved to a new file as I undersand
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
Review Comment:
👍
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
Review Comment:
what does `plan again` mean here? Does this mean that if you rebuild the decoder, the previous scan plan may be invalidated and you should call `scan_plan` again? If so, perhaps we can just say that explicityl
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
Review Comment:
Maybe we could:
1. Link to `NeedsData`
2. maybe change this to be more readable like "A ScanPlan is an estimate of the ranges that may be needed, whereas the returned `NeedsData` from `try_decode` are actually needed
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
Review Comment:
this seems like a comment that should go on `push_ranges` (you could link it here if you want)
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
Review Comment:
```suggestion
/// and one range for the entire column chunk if not.
```
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
Review Comment:
It might help here to have a fallback / comment if data wasn't in the cache
maybe like
```rust
/// DecodeResult::NeedsData(ranges) => {
/// // In a real system if the data isn't cached, you would fetch it here
/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
```
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
Review Comment:
What does this mean? How could a row group be invalid? Is this trying to say that if the decoder errors not all of the data from the plan may be needed? That seems somewhat obvious to me, but it is probably ok to leave
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
+/// decoder.push_ranges(ranges, data).unwrap();
+/// }
+/// DecodeResult::Data(batch) => println!("{} rows", batch.num_rows()),
+/// DecodeResult::Finished => break,
+/// }
+/// }
+/// ```
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+/// [`DecodeResult::NeedsData`]: crate::DecodeResult::NeedsData
+/// [`ParquetPushDecoder::scan_plan`]: super::ParquetPushDecoder::scan_plan
+/// [`ParquetPushDecoder::into_builder`]: super::ParquetPushDecoder::into_builder
+/// [`ParquetPushDecoderBuilder::build`]: super::ParquetPushDecoderBuilder::build
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// `None` for an empty plan, for example of a finished decoder.
+ planner: Option<Box<Planner>>,
+}
+
+impl ScanPlan {
+ /// A plan with no ranges.
+ pub(crate) fn empty() -> Self {
+ Self { planner: None }
+ }
+}
+
+/// The state of a [`ScanPlan`] that is not empty.
+#[derive(Debug, Clone)]
+struct Planner {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the plan was made.
+ frontier: RowGroupFrontier,
+ /// The row group that the decoder was fetching when the plan was made.
+ /// It is planned first.
+ active_row_group: Option<NextRowGroup>,
+ /// The decoder's projection, predicates and batch size.
+ columns: Arc<StageColumns>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// The row group being planned, if any.
+ current: Option<RowGroupRanges>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+/// The columns each decoding stage reads, as [`RowGroupReaderBuilder`]
+/// uses them to decide which bytes a row group needs.
+///
+/// [`RowGroupReaderBuilder`]: super::reader_builder::RowGroupReaderBuilder
+#[derive(Debug)]
+struct StageColumns {
+ /// The output batch size, which aligns cached predicate reads.
+ batch_size: usize,
+ /// Columns in the output.
+ projection: ProjectionMask,
+ /// Columns each [`RowFilter`] predicate reads, in evaluation order.
+ ///
+ /// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+ predicate_projections: Vec<ProjectionMask>,
+ /// Predicate columns whose decoded values are cached for the output, if
+ /// any. Their selection is expanded to batch boundaries when fetched.
+ cache_projection: Option<ProjectionMask>,
+}
+
+/// Builds a [`ScanPlan`] from a decoder's row-group frontier and the columns
+/// each decoding stage reads.
+#[derive(Debug)]
+pub(crate) struct ScanPlanBuilder {
+ frontier: RowGroupFrontier,
+ active_row_group: Option<NextRowGroup>,
+ columns: StageColumns,
+}
+
+impl ScanPlanBuilder {
+ /// Plan the row groups in `frontier`, decoding `projection` in batches of
+ /// `batch_size` rows, with no predicates.
+ pub(crate) fn new(
+ frontier: RowGroupFrontier,
+ batch_size: usize,
+ projection: ProjectionMask,
+ ) -> Self {
+ Self {
+ frontier,
+ active_row_group: None,
+ columns: StageColumns {
+ batch_size,
+ projection,
+ predicate_projections: vec![],
+ cache_projection: None,
+ },
+ }
+ }
+
+ /// Set the columns each predicate reads, in evaluation order.
+ pub(crate) fn with_predicate_projections(
+ mut self,
+ predicate_projections: Vec<ProjectionMask>,
+ ) -> Self {
+ self.columns.predicate_projections = predicate_projections;
+ self
+ }
+
+ /// Set the predicate columns whose decoded values are cached for the
+ /// output.
+ pub(crate) fn with_cache_projection(
+ mut self,
+ cache_projection: Option<ProjectionMask>,
+ ) -> Self {
+ self.columns.cache_projection = cache_projection;
+ self
+ }
+
+ /// Plan `active_row_group` first: the row group that the decoder is
+ /// fetching, which the frontier has already handed over.
+ pub(crate) fn with_active_row_group(mut self, active_row_group: Option<NextRowGroup>) -> Self {
+ self.active_row_group = active_row_group;
+ self
+ }
+
+ pub(crate) fn build(self) -> ScanPlan {
+ let Self {
+ frontier,
+ active_row_group,
+ columns,
+ } = self;
+ let has_limit = frontier.budget.limit().is_some();
+ let planner = Planner {
+ frontier,
+ active_row_group,
+ columns: Arc::new(columns),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ current: None,
+ done: false,
+ };
+ ScanPlan {
+ planner: Some(Box::new(planner)),
+ }
+ }
+}
+
+impl Planner {
+ /// Start planning the next row group the decoder will read.
+ ///
+ /// Returns `Ok(false)` when no row group remains. A row group that the
+ /// offset/limit budget removes is planned with no ranges.
+ fn plan_next_row_group(&mut self) -> Result<bool, ParquetError> {
Review Comment:
this seems like it is duplicating logic that is elsewhere in the decoder and will have to be kept in sync
I see you have equality tests to keep it consistent, but it would be nice to avoid it.
I wonder if you have contemplated using the ScanPlan directly in the ParquetDecoder itself -- that way `ParquetDecoder::scan_plan` would return `self.scan_plan.clone()` and the calculation of the next row group / needed ranges would be in the same logic
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
+/// decoder.push_ranges(ranges, data).unwrap();
+/// }
+/// DecodeResult::Data(batch) => println!("{} rows", batch.num_rows()),
+/// DecodeResult::Finished => break,
+/// }
+/// }
+/// ```
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+/// [`DecodeResult::NeedsData`]: crate::DecodeResult::NeedsData
+/// [`ParquetPushDecoder::scan_plan`]: super::ParquetPushDecoder::scan_plan
+/// [`ParquetPushDecoder::into_builder`]: super::ParquetPushDecoder::into_builder
+/// [`ParquetPushDecoderBuilder::build`]: super::ParquetPushDecoderBuilder::build
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// `None` for an empty plan, for example of a finished decoder.
+ planner: Option<Box<Planner>>,
+}
+
+impl ScanPlan {
+ /// A plan with no ranges.
+ pub(crate) fn empty() -> Self {
+ Self { planner: None }
+ }
+}
+
+/// The state of a [`ScanPlan`] that is not empty.
+#[derive(Debug, Clone)]
+struct Planner {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the plan was made.
+ frontier: RowGroupFrontier,
+ /// The row group that the decoder was fetching when the plan was made.
+ /// It is planned first.
+ active_row_group: Option<NextRowGroup>,
+ /// The decoder's projection, predicates and batch size.
+ columns: Arc<StageColumns>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// The row group being planned, if any.
+ current: Option<RowGroupRanges>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+/// The columns each decoding stage reads, as [`RowGroupReaderBuilder`]
+/// uses them to decide which bytes a row group needs.
+///
+/// [`RowGroupReaderBuilder`]: super::reader_builder::RowGroupReaderBuilder
+#[derive(Debug)]
+struct StageColumns {
+ /// The output batch size, which aligns cached predicate reads.
+ batch_size: usize,
+ /// Columns in the output.
+ projection: ProjectionMask,
+ /// Columns each [`RowFilter`] predicate reads, in evaluation order.
+ ///
+ /// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+ predicate_projections: Vec<ProjectionMask>,
+ /// Predicate columns whose decoded values are cached for the output, if
+ /// any. Their selection is expanded to batch boundaries when fetched.
+ cache_projection: Option<ProjectionMask>,
+}
+
+/// Builds a [`ScanPlan`] from a decoder's row-group frontier and the columns
+/// each decoding stage reads.
+#[derive(Debug)]
+pub(crate) struct ScanPlanBuilder {
+ frontier: RowGroupFrontier,
+ active_row_group: Option<NextRowGroup>,
+ columns: StageColumns,
+}
+
+impl ScanPlanBuilder {
+ /// Plan the row groups in `frontier`, decoding `projection` in batches of
+ /// `batch_size` rows, with no predicates.
+ pub(crate) fn new(
+ frontier: RowGroupFrontier,
+ batch_size: usize,
+ projection: ProjectionMask,
+ ) -> Self {
+ Self {
+ frontier,
+ active_row_group: None,
+ columns: StageColumns {
+ batch_size,
+ projection,
+ predicate_projections: vec![],
+ cache_projection: None,
+ },
+ }
+ }
+
+ /// Set the columns each predicate reads, in evaluation order.
+ pub(crate) fn with_predicate_projections(
+ mut self,
+ predicate_projections: Vec<ProjectionMask>,
+ ) -> Self {
+ self.columns.predicate_projections = predicate_projections;
+ self
+ }
+
+ /// Set the predicate columns whose decoded values are cached for the
+ /// output.
+ pub(crate) fn with_cache_projection(
+ mut self,
+ cache_projection: Option<ProjectionMask>,
+ ) -> Self {
+ self.columns.cache_projection = cache_projection;
+ self
+ }
+
+ /// Plan `active_row_group` first: the row group that the decoder is
+ /// fetching, which the frontier has already handed over.
+ pub(crate) fn with_active_row_group(mut self, active_row_group: Option<NextRowGroup>) -> Self {
+ self.active_row_group = active_row_group;
+ self
+ }
+
+ pub(crate) fn build(self) -> ScanPlan {
+ let Self {
+ frontier,
+ active_row_group,
+ columns,
+ } = self;
+ let has_limit = frontier.budget.limit().is_some();
+ let planner = Planner {
+ frontier,
+ active_row_group,
+ columns: Arc::new(columns),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ current: None,
+ done: false,
+ };
+ ScanPlan {
+ planner: Some(Box::new(planner)),
+ }
+ }
+}
+
+impl Planner {
+ /// Start planning the next row group the decoder will read.
+ ///
+ /// Returns `Ok(false)` when no row group remains. A row group that the
+ /// offset/limit budget removes is planned with no ranges.
+ fn plan_next_row_group(&mut self) -> Result<bool, ParquetError> {
+ // Use the decoder's own row-group walk, so row groups that the
+ // selection or the budget removes are skipped here too.
+ let next_row_group = match self.active_row_group.take() {
+ Some(active) => Some(active),
+ None => self.frontier.next_readable_row_group()?,
+ };
+ let Some(NextRowGroup {
+ row_group_idx,
+ row_count,
+ selection,
+ budget,
+ }) = next_row_group
+ else {
+ return Ok(false);
+ };
+
+ let filtered = self.frontier.has_predicates;
+ let selection = if filtered {
+ // Predicates see every selected row. Offset and limit apply to
+ // their output, so they cannot be applied before decoding.
+ selection
+ } else {
+ // Apply offset and limit as the decoder does before it requests
+ // the output columns.
+ let plan_builder =
+ ReadPlanBuilder::new(self.columns.batch_size).with_selection(selection);
+ let BudgetedReadPlan {
+ plan_builder,
+ rows_after_budget,
+ remaining_budget,
+ ..
+ } = budget.apply_to_plan(plan_builder, row_count);
+ self.frontier
+ .update_budget_after_row_group(remaining_budget);
+ if rows_after_budget == 0 {
+ self.current = None;
+ return Ok(true);
+ }
+ plan_builder.selection().cloned()
+ };
+
+ let rows = SelectedRows::new(selection.as_ref(), row_count);
+ let first_row = self.next_row;
+ self.next_row += rows.selected_before(row_count);
+
+ let metadata = &self.frontier.parquet_metadata;
+ let page_index = metadata
+ .page_index()
+ .filter(|page_index| page_index.has_offset_indexes())
+ .cloned();
+ let row_group = metadata.row_group(row_group_idx);
+
+ // Predicate columns that are cached for the output are fetched with the
+ // selection expanded to batch boundaries. See `fetch_ranges`.
+ let expanded_selection = match (&selection, &self.columns.cache_projection) {
+ (Some(selection), Some(_)) if filtered => {
+ Some(selection.expand_to_batch_boundaries(self.columns.batch_size, row_count))
+ }
+ _ => None,
+ };
+
+ let stages = self
+ .columns
+ .predicate_projections
+ .iter()
+ .enumerate()
+ .map(|(idx, mask)| (ScanStage::Predicate(idx), mask))
+ .chain(std::iter::once((
+ ScanStage::Projection,
+ &self.columns.projection,
+ )));
+
+ let mut planned_columns = vec![false; row_group.columns().len()];
+ let mut stage_plans = vec![];
+ for (stage, mask) in stages {
+ let conditional = filtered
+ && (stage != ScanStage::Predicate(0)
+ || (self.has_limit && !self.at_first_row_group));
+ let mut columns = vec![];
+ for (column_idx, chunk) in row_group.columns().iter().enumerate() {
+ // The decoder reuses a column that an earlier stage read.
+ if !mask.leaf_included(column_idx) || planned_columns[column_idx] {
+ continue;
+ }
+ planned_columns[column_idx] = true;
+ let is_cached = self
+ .columns
+ .cache_projection
+ .as_ref()
+ .is_some_and(|cache| cache.leaf_included(column_idx));
+ let (chunk_start, chunk_len) = chunk.byte_range();
+ columns.push(StageColumn {
+ column_idx,
+ chunk: chunk_start..chunk_start + chunk_len,
+ expanded: matches!(stage, ScanStage::Predicate(_))
+ && is_cached
+ && expanded_selection.is_some(),
+ });
+ }
+ stage_plans.push(StagePlan {
+ stage,
+ conditional,
+ columns,
+ });
+ }
+ self.at_first_row_group = false;
+
+ self.current = Some(RowGroupRanges {
+ row_group: RowGroupContext {
+ row_group_idx,
+ row_count,
+ first_row,
+ rows,
+ selection,
+ expanded_selection,
+ page_index,
+ },
+ stages: stage_plans.into_iter(),
+ stage: None,
+ });
+ Ok(true)
+ }
+}
+
+impl Iterator for ScanPlan {
+ type Item = PlannedRange;
+
+ fn next(&mut self) -> Option<PlannedRange> {
+ self.planner.as_mut()?.next()
+ }
+}
+
+impl Planner {
+ fn next(&mut self) -> Option<PlannedRange> {
+ loop {
+ if let Some(range) = self.current.as_mut().and_then(RowGroupRanges::next) {
+ return Some(range);
+ }
+ if self.done {
+ return None;
+ }
+ match self.plan_next_row_group() {
+ Ok(true) => {}
+ // The decoder reports the same error when it reaches this row
+ // group. The plan is advisory, so it ends here.
+ Ok(false) | Err(_) => {
+ self.current = None;
+ self.done = true;
+ }
+ }
+ }
+ }
+}
+
+/// Once [`ScanPlan::next`] returns `None`, it always returns `None`.
+impl FusedIterator for ScanPlan {}
+
+/// The columns that one decoding stage of a row group reads first.
+#[derive(Debug, Clone)]
+struct StagePlan {
+ stage: ScanStage,
+ /// See [`PlannedRange::conditional`].
+ conditional: bool,
+ columns: Vec<StageColumn>,
+}
+
+/// A column chunk that a decoding stage reads.
+#[derive(Debug, Clone)]
+struct StageColumn {
Review Comment:
I can't help by think this looks very similar to the actual decoder's state machine
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
+/// decoder.push_ranges(ranges, data).unwrap();
+/// }
+/// DecodeResult::Data(batch) => println!("{} rows", batch.num_rows()),
+/// DecodeResult::Finished => break,
+/// }
+/// }
+/// ```
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+/// [`DecodeResult::NeedsData`]: crate::DecodeResult::NeedsData
+/// [`ParquetPushDecoder::scan_plan`]: super::ParquetPushDecoder::scan_plan
+/// [`ParquetPushDecoder::into_builder`]: super::ParquetPushDecoder::into_builder
+/// [`ParquetPushDecoderBuilder::build`]: super::ParquetPushDecoderBuilder::build
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// `None` for an empty plan, for example of a finished decoder.
+ planner: Option<Box<Planner>>,
+}
+
+impl ScanPlan {
+ /// A plan with no ranges.
+ pub(crate) fn empty() -> Self {
+ Self { planner: None }
+ }
+}
+
+/// The state of a [`ScanPlan`] that is not empty.
+#[derive(Debug, Clone)]
+struct Planner {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the plan was made.
+ frontier: RowGroupFrontier,
+ /// The row group that the decoder was fetching when the plan was made.
+ /// It is planned first.
+ active_row_group: Option<NextRowGroup>,
+ /// The decoder's projection, predicates and batch size.
+ columns: Arc<StageColumns>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// The row group being planned, if any.
+ current: Option<RowGroupRanges>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+/// The columns each decoding stage reads, as [`RowGroupReaderBuilder`]
+/// uses them to decide which bytes a row group needs.
+///
+/// [`RowGroupReaderBuilder`]: super::reader_builder::RowGroupReaderBuilder
+#[derive(Debug)]
+struct StageColumns {
+ /// The output batch size, which aligns cached predicate reads.
+ batch_size: usize,
+ /// Columns in the output.
+ projection: ProjectionMask,
+ /// Columns each [`RowFilter`] predicate reads, in evaluation order.
+ ///
+ /// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+ predicate_projections: Vec<ProjectionMask>,
+ /// Predicate columns whose decoded values are cached for the output, if
+ /// any. Their selection is expanded to batch boundaries when fetched.
+ cache_projection: Option<ProjectionMask>,
+}
+
+/// Builds a [`ScanPlan`] from a decoder's row-group frontier and the columns
+/// each decoding stage reads.
+#[derive(Debug)]
+pub(crate) struct ScanPlanBuilder {
+ frontier: RowGroupFrontier,
+ active_row_group: Option<NextRowGroup>,
+ columns: StageColumns,
+}
+
+impl ScanPlanBuilder {
+ /// Plan the row groups in `frontier`, decoding `projection` in batches of
+ /// `batch_size` rows, with no predicates.
+ pub(crate) fn new(
+ frontier: RowGroupFrontier,
+ batch_size: usize,
+ projection: ProjectionMask,
+ ) -> Self {
+ Self {
+ frontier,
+ active_row_group: None,
+ columns: StageColumns {
+ batch_size,
+ projection,
+ predicate_projections: vec![],
+ cache_projection: None,
+ },
+ }
+ }
+
+ /// Set the columns each predicate reads, in evaluation order.
+ pub(crate) fn with_predicate_projections(
+ mut self,
+ predicate_projections: Vec<ProjectionMask>,
+ ) -> Self {
+ self.columns.predicate_projections = predicate_projections;
+ self
+ }
+
+ /// Set the predicate columns whose decoded values are cached for the
+ /// output.
+ pub(crate) fn with_cache_projection(
+ mut self,
+ cache_projection: Option<ProjectionMask>,
+ ) -> Self {
+ self.columns.cache_projection = cache_projection;
+ self
+ }
+
+ /// Plan `active_row_group` first: the row group that the decoder is
+ /// fetching, which the frontier has already handed over.
+ pub(crate) fn with_active_row_group(mut self, active_row_group: Option<NextRowGroup>) -> Self {
+ self.active_row_group = active_row_group;
+ self
+ }
+
+ pub(crate) fn build(self) -> ScanPlan {
+ let Self {
+ frontier,
+ active_row_group,
+ columns,
+ } = self;
+ let has_limit = frontier.budget.limit().is_some();
+ let planner = Planner {
+ frontier,
+ active_row_group,
+ columns: Arc::new(columns),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ current: None,
+ done: false,
+ };
+ ScanPlan {
+ planner: Some(Box::new(planner)),
+ }
+ }
+}
+
+impl Planner {
+ /// Start planning the next row group the decoder will read.
+ ///
+ /// Returns `Ok(false)` when no row group remains. A row group that the
+ /// offset/limit budget removes is planned with no ranges.
+ fn plan_next_row_group(&mut self) -> Result<bool, ParquetError> {
Review Comment:
maybe as a follow on / cleanup PR
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
Review Comment:
if these are pub crate -- why include them at all? It is probably fine, I am just wondering
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
Review Comment:
this is a really nice example
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
Review Comment:
I think another important property to mention is that the ranges are generated on demand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5941929180
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940184173)
**Target:** arrow_reader
Comparing claude/push-decoder-peek-59 (0d91bca5e174821c56fc6529c6bfb6d6a52e8d81) to 76829d5 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/76829d5ce4562616a69ea8b2696e96160514482f..0d91bca5e174821c56fc6529c6bfb6d6a52e8d81)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 850.7±16.37µs ? ?/sec 1.00 854.7±8.92µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 951.8±4.82µs ? ?/sec 1.00 953.4±7.24µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 850.7±12.62µs ? ?/sec 1.00 847.4±8.98µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.00 285.5±0.44µs ? ?/sec 1.00 285.1±0.57µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs, large values 1.09 50.1±0.96ms ? ?/sec 1.00 46.1±3.23ms ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.00 391.9±0.71µs ? ?/sec 1.00 391.1±1.08µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.00 285.7±0.60µs ? ?/sec 1.01 288.4±0.36µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.00 408.0±2.17µs ? ?/sec 1.00 407.5±6.04µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.00 471.4±4.76µs ? ?/sec 1.00 469.4±4.14µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.00 413.1±3.23µs ? ?/sec 1.00 413.5±4.58µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.4±0.03µs ? ?/sec 1.00 80.5±0.09µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.00 111.7±0.14µs ? ?/sec 1.00 111.2±0.14µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 84.3±0.06µs ? ?/sec 1.00 84.1±0.12µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.00 148.1±0.58µs ? ?/sec 1.00 147.7±0.89µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.11 157.1±3.79µs ? ?/sec 1.00 141.7±0.52µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.00 149.8±0.32µs ? ?/sec 1.00 149.3±0.63µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.01 154.9±0.44µs ? ?/sec 1.00 153.3±0.58µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, mandatory, no NULLs 1.00 227.4±0.28µs ? ?/sec 1.00 226.6±0.37µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, half NULLs 1.00 243.5±1.92µs ? ?/sec 1.00 243.8±1.58µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, no NULLs 1.00 230.2±0.23µs ? ?/sec 1.00 230.1±0.37µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.11 1078.6±4.61µs ? ?/sec 1.00 969.0±1.24µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 672.6±0.51µs ? ?/sec 1.00 674.1±1.64µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.05 1029.4±22.81µs ? ?/sec 1.00 975.9±16.52µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 191.2±0.15µs ? ?/sec 1.00 189.9±0.29µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 293.8±0.18µs ? ?/sec 1.01 295.9±0.59µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.01 195.4±0.28µs ? ?/sec 1.00 194.4±0.25µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 126.4±0.39µs ? ?/sec 1.02 128.9±0.34µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 183.6±0.15µs ? ?/sec 1.01 184.9±0.82µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 131.2±0.22µs ? ?/sec 1.01 133.1±0.28µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 60.1±0.05µs ? ?/sec 1.02 61.2±0.08µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 146.7±0.12µs ? ?/sec 1.01 148.2±0.12µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 63.2±0.03µs ? ?/sec 1.02 64.2±0.04µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 803.8±0.71µs ? ?/sec 1.00 803.3±0.78µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 511.2±0.31µs ? ?/sec 1.00 511.9±0.43µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.01 812.1±2.06µs ? ?/sec 1.00 807.8±0.93µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.05 24.2±0.05µs ? ?/sec 1.00 23.1±0.04µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.00 131.2±0.14µs ? ?/sec 1.01 132.2±0.11µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.02 28.6±0.06µs ? ?/sec 1.00 28.0±0.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.00 73.5±0.31µs ? ?/sec 1.03 75.6±0.24µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.01 129.7±0.14µs ? ?/sec 1.00 128.8±0.29µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.00 79.2±0.15µs ? ?/sec 1.00 79.1±0.24µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.04 5.4±0.03µs ? ?/sec 1.00 5.2±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.01 93.3±0.08µs ? ?/sec 1.00 92.6±0.19µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.03 8.9±0.03µs ? ?/sec 1.00 8.7±0.07µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.00 145.2±0.38µs ? ?/sec 1.01 146.5±0.37µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.01 201.2±0.17µs ? ?/sec 1.00 198.5±0.28µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.00 150.3±1.63µs ? ?/sec 1.01 151.8±0.53µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.00 7.6±0.02µs ? ?/sec 1.00 7.6±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.01 128.7±0.08µs ? ?/sec 1.00 127.6±0.21µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.02 11.3±0.03µs ? ?/sec 1.00 11.1±0.07µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.01 303.0±0.79µs ? ?/sec 1.00 301.2±0.96µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.01 270.1±0.32µs ? ?/sec 1.00 268.6±0.48µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.00 310.0±0.88µs ? ?/sec 1.05 326.5±13.95µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.01 12.1±0.03µs ? ?/sec 1.00 12.0±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.00 124.0±0.13µs ? ?/sec 1.01 124.8±0.13µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.03 16.3±0.03µs ? ?/sec 1.00 15.9±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed increasing value 1.00 87.5±0.61µs ? ?/sec 1.01 88.7±0.66µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed single value 1.00 79.6±0.33µs ? ?/sec 1.00 79.5±0.70µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value 1.00 50.5±0.40µs ? ?/sec 1.01 50.8±0.39µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value 1.00 46.0±0.18µs ? ?/sec 1.00 46.2±0.37µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value 1.00 74.8±0.27µs ? ?/sec 1.00 74.9±0.25µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 89.3±0.99µs ? ?/sec 1.00 89.8±1.01µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.00 87.4±0.49µs ? ?/sec 1.01 88.0±0.51µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.00 91.4±0.99µs ? ?/sec 1.00 91.6±0.98µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed stepped increasing value 1.00 108.0±0.23µs ? ?/sec 1.00 108.2±0.19µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.00 126.5±0.83µs ? ?/sec 1.00 126.9±0.82µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.00 144.5±0.41µs ? ?/sec 1.00 144.7±0.47µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.00 130.1±0.80µs ? ?/sec 1.00 130.6±0.81µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 54.0±0.08µs ? ?/sec 1.00 54.2±0.11µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 107.2±0.15µs ? ?/sec 1.00 107.3±0.13µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 57.6±0.06µs ? ?/sec 1.00 57.8±0.15µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 84.3±0.10µs ? ?/sec 1.00 84.6±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 125.1±0.20µs ? ?/sec 1.00 125.2±0.39µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 88.0±0.06µs ? ?/sec 1.00 88.1±0.08µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 47.2±0.28µs ? ?/sec 1.01 47.5±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.00 104.4±0.12µs ? ?/sec 1.00 104.6±0.15µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.00 50.7±0.05µs ? ?/sec 1.00 50.8±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed increasing value 1.00 78.3±0.16µs ? ?/sec 1.00 78.3±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed single value 1.01 75.3±0.16µs ? ?/sec 1.00 74.9±0.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value 1.00 43.3±0.07µs ? ?/sec 1.00 43.3±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value 1.00 41.5±0.06µs ? ?/sec 1.00 41.4±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value 1.01 68.2±0.05µs ? ?/sec 1.00 67.9±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 79.7±0.06µs ? ?/sec 1.00 79.6±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.00 83.4±0.15µs ? ?/sec 1.00 83.3±0.14µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.00 81.7±0.07µs ? ?/sec 1.00 81.6±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed stepped increasing value 1.00 100.9±0.09µs ? ?/sec 1.00 100.4±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 114.8±0.12µs ? ?/sec 1.00 114.9±0.17µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 140.4±0.21µs ? ?/sec 1.00 140.9±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.00 118.7±0.13µs ? ?/sec 1.00 118.5±0.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.01 83.4±0.07µs ? ?/sec 1.00 83.0±0.19µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 124.4±0.14µs ? ?/sec 1.00 124.3±0.22µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 87.7±0.06µs ? ?/sec 1.00 87.6±0.19µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 86.8±0.05µs ? ?/sec 1.00 86.9±0.09µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 127.7±0.28µs ? ?/sec 1.00 127.9±0.19µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 90.5±0.04µs ? ?/sec 1.00 90.3±0.14µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 54.7±0.05µs ? ?/sec 1.00 54.2±0.03µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 109.6±0.13µs ? ?/sec 1.00 109.6±0.14µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.00 58.8±0.04µs ? ?/sec 1.00 58.5±0.04µs ? ?/sec
arrow_array_reader/Int16Array/binary packed increasing value 1.00 54.8±0.50µs ? ?/sec 1.00 55.0±0.60µs ? ?/sec
arrow_array_reader/Int16Array/binary packed single value 1.00 46.1±0.58µs ? ?/sec 1.00 46.1±0.66µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value 1.00 33.5±0.30µs ? ?/sec 1.01 33.8±0.34µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value 1.00 28.8±0.34µs ? ?/sec 1.00 28.9±0.38µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value 1.00 57.6±0.26µs ? ?/sec 1.01 58.2±0.30µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.00 72.2±1.03µs ? ?/sec 1.00 72.2±1.04µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.00 70.6±0.48µs ? ?/sec 1.00 70.7±0.52µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.00 74.2±1.04µs ? ?/sec 1.00 74.3±1.00µs ? ?/sec
arrow_array_reader/Int16Array/binary packed stepped increasing value 1.00 74.8±0.16µs ? ?/sec 1.01 75.2±0.28µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.00 92.0±0.37µs ? ?/sec 1.01 92.5±0.39µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.00 110.7±0.18µs ? ?/sec 1.00 110.8±0.31µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.00 95.6±0.32µs ? ?/sec 1.00 95.7±0.34µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 21.0±0.04µs ? ?/sec 1.01 21.2±0.05µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.01 74.9±0.11µs ? ?/sec 1.00 74.1±0.17µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.6±0.04µs ? ?/sec 1.00 24.7±0.06µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.00 51.3±0.04µs ? ?/sec 1.00 51.3±0.07µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.00 92.0±0.15µs ? ?/sec 1.00 91.8±0.15µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.00 55.2±0.04µs ? ?/sec 1.00 55.0±0.12µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.00 13.9±0.04µs ? ?/sec 1.00 13.9±0.04µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.01 71.5±0.14µs ? ?/sec 1.00 71.0±0.19µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.01 17.7±0.08µs ? ?/sec 1.00 17.6±0.06µs ? ?/sec
arrow_array_reader/Int32Array/binary packed increasing value 1.00 50.9±0.46µs ? ?/sec 1.01 51.3±0.60µs ? ?/sec
arrow_array_reader/Int32Array/binary packed single value 1.00 43.0±0.34µs ? ?/sec 1.02 43.8±0.39µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value 1.00 31.4±0.27µs ? ?/sec 1.01 31.8±0.32µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value 1.00 27.2±0.19µs ? ?/sec 1.02 27.7±0.24µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value 1.00 55.7±0.28µs ? ?/sec 1.01 56.0±0.30µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.00 70.4±1.01µs ? ?/sec 1.00 70.7±0.96µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.00 69.2±0.49µs ? ?/sec 1.01 69.7±0.54µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.00 72.5±0.99µs ? ?/sec 1.00 72.8±0.95µs ? ?/sec
arrow_array_reader/Int32Array/binary packed stepped increasing value 1.00 70.8±0.20µs ? ?/sec 1.00 71.1±0.23µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.00 89.6±0.82µs ? ?/sec 1.00 89.3±0.81µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.00 108.2±0.39µs ? ?/sec 1.01 109.0±0.53µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.00 93.1±0.78µs ? ?/sec 1.00 93.3±0.79µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.4±0.05µs ? ?/sec 1.00 17.4±0.05µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.01 71.4±0.16µs ? ?/sec 1.00 70.6±0.23µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.01 21.1±0.10µs ? ?/sec 1.00 21.0±0.05µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.00 47.6±0.06µs ? ?/sec 1.01 47.8±0.10µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.00 89.1±0.19µs ? ?/sec 1.00 88.7±0.28µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.00 51.1±0.05µs ? ?/sec 1.01 51.5±0.22µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.00 9.9±0.03µs ? ?/sec 1.00 10.0±0.07µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.00 68.1±0.40µs ? ?/sec 1.00 68.2±0.16µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.00 13.6±0.06µs ? ?/sec 1.00 13.6±0.06µs ? ?/sec
arrow_array_reader/Int64Array/binary packed increasing value 1.00 41.4±0.17µs ? ?/sec 1.00 41.3±0.15µs ? ?/sec
arrow_array_reader/Int64Array/binary packed single value 1.01 38.7±0.12µs ? ?/sec 1.00 38.4±0.14µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value 1.00 24.6±0.07µs ? ?/sec 1.00 24.5±0.06µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value 1.01 22.9±0.06µs ? ?/sec 1.00 22.7±0.05µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value 1.00 49.0±0.05µs ? ?/sec 1.00 48.9±0.15µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.00 60.7±0.05µs ? ?/sec 1.00 60.8±0.05µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.00 64.8±0.12µs ? ?/sec 1.00 64.9±0.12µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.00 62.7±0.04µs ? ?/sec 1.00 62.7±0.04µs ? ?/sec
arrow_array_reader/Int64Array/binary packed stepped increasing value 1.01 63.7±0.10µs ? ?/sec 1.00 63.4±0.13µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.00 78.0±0.14µs ? ?/sec 1.00 78.0±0.23µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.00 103.7±0.18µs ? ?/sec 1.00 103.9±0.22µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.00 81.9±0.17µs ? ?/sec 1.00 82.1±0.21µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.00 45.8±0.03µs ? ?/sec 1.00 45.9±0.18µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.6±0.10µs ? ?/sec 1.00 87.7±0.19µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.6±0.04µs ? ?/sec 1.00 49.6±0.06µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.00 50.0±0.05µs ? ?/sec 1.00 49.9±0.05µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 91.2±0.15µs ? ?/sec 1.00 91.4±0.25µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.00 53.4±0.04µs ? ?/sec 1.00 53.4±0.08µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.03 16.7±0.03µs ? ?/sec 1.00 16.3±0.03µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.00 72.3±0.11µs ? ?/sec 1.01 73.0±0.14µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.00 20.1±0.06µs ? ?/sec 1.00 20.0±0.04µs ? ?/sec
arrow_array_reader/Int8Array/binary packed increasing value 1.00 54.2±0.55µs ? ?/sec 1.01 54.9±0.51µs ? ?/sec
arrow_array_reader/Int8Array/binary packed single value 1.00 46.1±0.31µs ? ?/sec 1.01 46.6±0.79µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value 1.00 33.2±0.29µs ? ?/sec 1.01 33.7±0.31µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value 1.00 29.0±0.14µs ? ?/sec 1.00 29.0±0.43µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value 1.00 57.7±0.25µs ? ?/sec 1.00 57.9±0.23µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.00 71.9±0.79µs ? ?/sec 1.01 72.4±0.70µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.00 70.1±0.42µs ? ?/sec 1.01 70.8±0.39µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.00 73.9±0.79µs ? ?/sec 1.00 74.2±0.66µs ? ?/sec
arrow_array_reader/Int8Array/binary packed stepped increasing value 1.00 74.4±0.17µs ? ?/sec 1.00 74.7±0.14µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.00 91.9±0.21µs ? ?/sec 1.01 92.5±0.22µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.00 110.1±0.24µs ? ?/sec 1.01 110.9±0.16µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 95.3±0.19µs ? ?/sec 1.00 95.4±0.09µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.9±0.04µs ? ?/sec 1.01 21.1±0.04µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.00 74.0±0.09µs ? ?/sec 1.00 74.3±0.17µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.01 24.6±0.03µs ? ?/sec 1.00 24.4±0.04µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.00 51.3±0.03µs ? ?/sec 1.00 51.1±0.04µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.00 91.9±0.11µs ? ?/sec 1.00 91.7±0.18µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.00 55.0±0.05µs ? ?/sec 1.00 54.8±0.05µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.00 13.8±0.05µs ? ?/sec 1.00 13.8±0.04µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.00 70.9±0.11µs ? ?/sec 1.01 71.3±0.24µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.01 17.5±0.07µs ? ?/sec 1.00 17.3±0.06µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/90pct NULLs 1.01 882.6±2.60µs ? ?/sec 1.00 875.6±0.85µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/99pct NULLs 1.01 451.4±5.21µs ? ?/sec 1.00 448.5±4.64µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/half NULLs 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.00ms ? ?/sec
arrow_array_reader/ListArray/Fixed32List/no NULLs 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
arrow_array_reader/ListArray/Int32List/90pct NULLs 1.01 869.4±1.07µs ? ?/sec 1.00 863.3±1.25µs ? ?/sec
arrow_array_reader/ListArray/Int32List/99pct NULLs 1.00 448.7±5.12µs ? ?/sec 1.01 451.2±4.69µs ? ?/sec
arrow_array_reader/ListArray/Int32List/half NULLs 1.00 1964.6±3.27µs ? ?/sec 1.00 1958.2±4.72µs ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs 1.01 2.7±0.00ms ? ?/sec 1.00 2.7±0.00ms ? ?/sec
arrow_array_reader/ListArray/StringList/90pct NULLs 1.01 936.9±2.22µs ? ?/sec 1.00 932.0±1.39µs ? ?/sec
arrow_array_reader/ListArray/StringList/99pct NULLs 1.00 453.0±4.28µs ? ?/sec 1.01 457.2±5.40µs ? ?/sec
arrow_array_reader/ListArray/StringList/half NULLs 1.01 3.0±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
arrow_array_reader/ListArray/StringList/no NULLs 1.00 4.9±0.03ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
arrow_array_reader/PlainStringToDictionary/high cardinality 1.00 1088.4±8.32µs ? ?/sec 1.00 1093.6±12.28µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/low cardinality 1.00 1024.3±5.74µs ? ?/sec 1.00 1021.3±10.42µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 1 1.00 1088.8±8.34µs ? ?/sec 1.00 1091.6±11.99µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 2 1.00 1038.7±6.16µs ? ?/sec 1.00 1036.0±10.83µs ? ?/sec
arrow_array_reader/StringArray/const delta byte array encoded, mandatory, no NULLs 1.00 547.5±2.00µs ? ?/sec 1.01 552.7±2.78µs ? ?/sec
arrow_array_reader/StringArray/const delta length byte array encoded, mandatory, no NULLs 1.03 224.3±1.12µs ? ?/sec 1.00 217.1±1.25µs ? ?/sec
arrow_array_reader/StringArray/const prefix delta byte array encoded, mandatory, no NULLs 1.02 799.0±34.72µs ? ?/sec 1.00 780.2±39.76µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.01 285.6±0.38µs ? ?/sec 1.00 284.1±0.27µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.00 389.9±1.17µs ? ?/sec 1.00 391.7±0.72µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.00 290.1±0.44µs ? ?/sec 1.00 288.8±0.65µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.00 444.3±5.89µs ? ?/sec 1.02 452.8±6.05µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.00 492.3±3.01µs ? ?/sec 1.01 498.7±2.72µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.00 445.1±5.20µs ? ?/sec 1.03 457.0±5.46µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.00 193.4±0.49µs ? ?/sec 1.00 194.3±0.71µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.00 244.7±0.43µs ? ?/sec 1.00 244.3±0.62µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.00 197.3±0.52µs ? ?/sec 1.00 198.2±0.79µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.4±0.08µs ? ?/sec 1.00 80.5±0.05µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.00 111.7±0.14µs ? ?/sec 1.00 111.5±0.19µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 84.3±0.06µs ? ?/sec 1.00 84.3±0.04µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.00 221.7±0.31µs ? ?/sec 1.00 221.5±0.75µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.01 187.6±0.23µs ? ?/sec 1.00 186.6±1.22µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.00 230.4±0.51µs ? ?/sec 1.00 230.2±0.77µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed increasing value 1.00 54.4±0.49µs ? ?/sec 1.01 54.6±0.67µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed single value 1.00 46.4±0.34µs ? ?/sec 1.00 46.5±0.35µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value 1.00 33.3±0.29µs ? ?/sec 1.01 33.5±0.36µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value 1.00 28.9±0.16µs ? ?/sec 1.01 29.1±0.19µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value 1.00 57.6±0.26µs ? ?/sec 1.01 58.0±0.23µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.00 75.5±0.20µs ? ?/sec 1.01 75.9±0.15µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.00 72.7±0.14µs ? ?/sec 1.00 73.0±0.18µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 77.8±0.15µs ? ?/sec 1.00 78.1±0.14µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed stepped increasing value 1.00 74.5±0.18µs ? ?/sec 1.00 74.8±0.16µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.00 97.4±0.16µs ? ?/sec 1.01 98.1±0.15µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.00 114.2±0.19µs ? ?/sec 1.00 114.3±0.16µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 100.9±0.10µs ? ?/sec 1.00 101.2±0.14µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 21.0±0.04µs ? ?/sec 1.00 21.0±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.00 74.3±0.12µs ? ?/sec 1.00 74.1±0.15µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.6±0.05µs ? ?/sec 1.00 24.5±0.05µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.00 51.1±0.04µs ? ?/sec 1.01 51.4±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.00 92.0±0.15µs ? ?/sec 1.00 91.9±0.14µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.00 54.8±0.04µs ? ?/sec 1.00 55.0±0.05µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.00 13.8±0.05µs ? ?/sec 1.00 13.8±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.01 71.6±0.11µs ? ?/sec 1.00 71.1±0.27µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.00 17.6±0.07µs ? ?/sec 1.00 17.5±0.09µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed increasing value 1.00 51.6±0.55µs ? ?/sec 1.00 51.4±0.67µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed single value 1.00 43.0±0.63µs ? ?/sec 1.01 43.2±0.34µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value 1.00 31.9±0.34µs ? ?/sec 1.00 31.8±0.37µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value 1.00 27.1±0.34µs ? ?/sec 1.01 27.4±0.18µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value 1.00 55.8±0.25µs ? ?/sec 1.01 56.3±0.23µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.00 70.9±0.99µs ? ?/sec 1.00 71.2±0.94µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.00 69.1±0.47µs ? ?/sec 1.00 69.5±0.47µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.00 72.6±0.92µs ? ?/sec 1.00 72.9±0.94µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed stepped increasing value 1.00 71.1±0.28µs ? ?/sec 1.00 71.3±0.14µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.00 89.7±0.82µs ? ?/sec 1.00 89.5±0.81µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.00 108.3±0.33µs ? ?/sec 1.00 108.4±0.42µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.00 93.3±0.78µs ? ?/sec 1.00 93.3±0.79µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.4±0.01µs ? ?/sec 1.01 17.5±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.00 70.9±0.09µs ? ?/sec 1.01 71.4±0.10µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.00 20.7±0.04µs ? ?/sec 1.02 21.0±0.04µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.00 47.9±0.02µs ? ?/sec 1.00 47.9±0.05µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.00 88.5±0.14µs ? ?/sec 1.00 88.8±0.16µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.00 51.5±0.06µs ? ?/sec 1.00 51.3±0.08µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.00 10.1±0.04µs ? ?/sec 1.00 10.2±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.00 68.3±0.13µs ? ?/sec 1.00 68.3±0.13µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.00 13.8±0.06µs ? ?/sec 1.01 13.9±0.05µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed increasing value 1.00 41.6±0.18µs ? ?/sec 1.00 41.5±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed single value 1.01 38.9±0.10µs ? ?/sec 1.00 38.5±0.21µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value 1.00 24.7±0.06µs ? ?/sec 1.00 24.6±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value 1.01 22.9±0.05µs ? ?/sec 1.00 22.8±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value 1.00 49.2±0.06µs ? ?/sec 1.00 49.1±0.10µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.00 60.7±0.07µs ? ?/sec 1.00 60.9±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.00 64.9±0.09µs ? ?/sec 1.00 64.8±0.20µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.00 62.7±0.06µs ? ?/sec 1.00 62.6±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed stepped increasing value 1.00 63.8±0.11µs ? ?/sec 1.00 63.6±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 78.1±0.20µs ? ?/sec 1.00 78.0±0.26µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.00 103.9±0.19µs ? ?/sec 1.00 103.9±0.26µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.00 81.9±0.20µs ? ?/sec 1.00 81.8±0.32µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 46.0±0.03µs ? ?/sec 1.00 45.8±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.3±0.10µs ? ?/sec 1.00 87.7±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.01 49.8±0.05µs ? ?/sec 1.00 49.5±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.00 50.1±0.03µs ? ?/sec 1.00 50.3±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.00 90.9±0.14µs ? ?/sec 1.00 91.1±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.00 53.5±0.06µs ? ?/sec 1.03 55.1±0.10µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.01 16.9±0.05µs ? ?/sec 1.00 16.6±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.00 72.4±0.18µs ? ?/sec 1.01 72.9±0.20µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.00 20.1±0.09µs ? ?/sec 1.00 20.0±0.10µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed increasing value 1.00 54.5±0.46µs ? ?/sec 1.00 54.4±0.64µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed single value 1.00 45.6±0.57µs ? ?/sec 1.03 47.0±0.48µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value 1.00 33.4±0.32µs ? ?/sec 1.00 33.4±0.34µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value 1.00 28.7±0.33µs ? ?/sec 1.03 29.6±0.24µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value 1.00 57.5±0.27µs ? ?/sec 1.01 57.9±0.29µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.00 75.2±0.35µs ? ?/sec 1.00 75.2±0.36µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.00 72.1±0.27µs ? ?/sec 1.00 72.3±0.24µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.00 77.5±0.35µs ? ?/sec 1.00 77.3±0.38µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed stepped increasing value 1.00 74.2±0.20µs ? ?/sec 1.01 74.6±0.27µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.00 96.7±0.14µs ? ?/sec 1.00 96.5±0.14µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.00 113.0±0.24µs ? ?/sec 1.00 112.6±0.37µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.00 100.2±0.22µs ? ?/sec 1.00 99.9±0.14µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.8±0.02µs ? ?/sec 1.02 21.2±0.10µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.00 73.7±0.14µs ? ?/sec 1.00 74.0±0.24µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.2±0.04µs ? ?/sec 1.02 24.6±0.07µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.00 51.1±0.04µs ? ?/sec 1.00 51.1±0.14µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.00 91.8±0.19µs ? ?/sec 1.00 91.6±0.33µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.00 54.5±0.05µs ? ?/sec 1.01 54.9±0.18µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.00 13.6±0.06µs ? ?/sec 1.00 13.6±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.01 71.5±0.10µs ? ?/sec 1.00 70.7±0.29µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.01 17.5±0.07µs ? ?/sec 1.00 17.3±0.06µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, half NULLs 1.00 68.5±0.19µs ? ?/sec 1.00 68.8±0.15µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, no NULLs 1.02 13.9±0.03µs ? ?/sec 1.00 13.6±0.02µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, half NULLs 1.00 128.6±0.18µs ? ?/sec 1.01 130.0±0.18µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, no NULLs 1.00 65.7±0.04µs ? ?/sec 1.02 67.1±0.12µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>arrow_reader — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-35u357fo-e3f52970-pd9z
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5940184173-3013-h6hkm 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2990.6s |
| Peak memory | 191.0 MiB |
| Avg memory | 39.2 MiB |
| CPU user | 2972.4s |
| CPU sys | 12.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 2985.7s |
| Peak memory | 185.9 MiB |
| Avg memory | 43.0 MiB |
| CPU user | 2966.8s |
| CPU sys | 13.8s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5940184173:finish:855dd9901e2a3c75ffb59e60127eb815 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#discussion_r4105824153
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +617,124 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Returns the byte ranges this scan may read, in the order decoding needs
+ /// them.
+ ///
+ /// Use this to schedule I/O ahead of [`DecodeResult::NeedsData`]: for
+ /// example, to fetch the next few megabytes in the background while the
+ /// current row group decodes, or to release fetched bytes once decoding
+ /// has passed them. [`DecodeResult::NeedsData`] stays the exact and
+ /// authoritative request. This plan is an upper bound on it, for
+ /// speculation.
+ ///
+ /// # Contract
+ ///
+ /// * **Buffer independent.** The plan does not depend on the data pushed
+ /// to the decoder. It lists ranges the decoder already has, so the caller
+ /// decides how to satisfy each range: from decoder buffers, from its own
+ /// cache, or from storage.
+ /// * **Stable.** The plan covers the whole scan as configured when the
+ /// decoder was built, including row groups already decoded. It does not
+ /// change as decoding advances. [`Self::into_builder`] and
+ /// [`ParquetPushDecoderBuilder::build`] give a new decoder, whose plan
+ /// covers only the row groups that remain.
+ /// * **Same planning as decoding.** The plan uses the decoder's own
+ /// row-group order, row selections, offset/limit and projection. Per row
+ /// group, the planned bytes of a scan without a [`RowFilter`] are the
+ /// bytes that [`DecodeResult::NeedsData`] requests. With a
+ /// [`RowFilter`], they are a superset; see [`PlannedRange::conditional`].
+ /// * **Lazy.** The iterator plans one row group at a time when the caller
Review Comment:
lazy is a nice property -- though I think this comment is mostly a comment that should go on `ScanPlan`'s docs itself.
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -169,17 +173,17 @@ impl RowBudget {
}
#[derive(Debug)]
-struct BudgetedReadPlan {
+pub(crate) struct BudgetedReadPlan {
Review Comment:
perhaps as a prep PR, we could move the `BudgetedReadPlan` and `RowGroupFrontier` to their own module (we could make `scan_plan.rs` to start)
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +617,124 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Returns the byte ranges this scan may read, in the order decoding needs
+ /// them.
+ ///
+ /// Use this to schedule I/O ahead of [`DecodeResult::NeedsData`]: for
+ /// example, to fetch the next few megabytes in the background while the
+ /// current row group decodes, or to release fetched bytes once decoding
+ /// has passed them. [`DecodeResult::NeedsData`] stays the exact and
+ /// authoritative request. This plan is an upper bound on it, for
+ /// speculation.
+ ///
+ /// # Contract
+ ///
+ /// * **Buffer independent.** The plan does not depend on the data pushed
+ /// to the decoder. It lists ranges the decoder already has, so the caller
+ /// decides how to satisfy each range: from decoder buffers, from its own
+ /// cache, or from storage.
+ /// * **Stable.** The plan covers the whole scan as configured when the
+ /// decoder was built, including row groups already decoded. It does not
Review Comment:
I thought the scan did change 🤔 Or maybe this is referring to the fact that this API returns a snapshot of the plan?
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +617,124 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Returns the byte ranges this scan may read, in the order decoding needs
+ /// them.
+ ///
+ /// Use this to schedule I/O ahead of [`DecodeResult::NeedsData`]: for
+ /// example, to fetch the next few megabytes in the background while the
+ /// current row group decodes, or to release fetched bytes once decoding
+ /// has passed them. [`DecodeResult::NeedsData`] stays the exact and
+ /// authoritative request. This plan is an upper bound on it, for
+ /// speculation.
+ ///
+ /// # Contract
+ ///
+ /// * **Buffer independent.** The plan does not depend on the data pushed
+ /// to the decoder. It lists ranges the decoder already has, so the caller
+ /// decides how to satisfy each range: from decoder buffers, from its own
+ /// cache, or from storage.
+ /// * **Stable.** The plan covers the whole scan as configured when the
+ /// decoder was built, including row groups already decoded. It does not
+ /// change as decoding advances. [`Self::into_builder`] and
+ /// [`ParquetPushDecoderBuilder::build`] give a new decoder, whose plan
+ /// covers only the row groups that remain.
+ /// * **Same planning as decoding.** The plan uses the decoder's own
+ /// row-group order, row selections, offset/limit and projection. Per row
Review Comment:
details about what the decoder is doing is likely too much detail here -- just saying it is the same as what the decoder uses is probably good enough
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -199,6 +203,21 @@ pub(crate) enum RowGroupBuildResult {
},
}
+/// What [`RowGroupReaderBuilder`] uses to decide which bytes a row group
+/// needs, captured so a scan can be planned without decoding it.
+#[derive(Debug, Clone)]
+pub(crate) struct ScanPlanConfig {
Review Comment:
maybe this could be `ScanPlanBuilder` rather than a config. Als it would be nice to put this in the same file as `ScanPlan`
##########
parquet/src/arrow/push_decoder/scan_plan.rs:
##########
@@ -0,0 +1,890 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges a push decoder scan may read, in the order
+//! decoding needs them.
+//!
+//! See [`ParquetPushDecoder::scan_plan`](super::ParquetPushDecoder::scan_plan).
+
+use std::cmp::Reverse;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::RowGroupPageIndex;
+
+use super::reader_builder::{BudgetedReadPlan, ScanPlanConfig};
+use super::remaining::{NextRowGroup, RowGroupFrontier};
+
+/// A byte range that a scan may read, with the rows and stage it serves.
+///
+/// Returned by [`ParquetPushDecoder::scan_plan`](super::ParquetPushDecoder::scan_plan).
+///
+/// # Rows
+///
+/// `first_row..last_row` are positions in the rows the decoder *plans* to
+/// read: row selections and, for scans without a [`RowFilter`], offset and
+/// limit are applied. Row 0 is the first planned row of the first planned row
+/// group, and positions continue across row groups.
+///
+/// For a scan without a [`RowFilter`], planned row `n` is output row `n`. A
+/// caller that has received `n` rows from the decoder can therefore release
+/// every range whose `last_row <= n`.
+///
+/// With a [`RowFilter`], positions count the rows the first predicate sees.
+/// Offset and limit apply after the predicates, so they are not applied here.
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// First planned row this range serves.
+ pub first_row: u64,
+ /// One past the last planned row this range serves.
+ ///
+ /// This can equal `first_row` for a page that the decoder reads only to
+ /// complete a batch for its predicate cache.
+ pub last_row: u64,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// The decoding stage that first reads this range.
+ pub stage: ScanStage,
+ /// `true` if the decoder may not read this range, depending on predicate
+ /// results.
+ ///
+ /// Always `false` for a scan without a [`RowFilter`]. With a [`RowFilter`],
+ /// ranges for later predicates and for the output columns are
+ /// conditional, because earlier predicates can remove every row they
+ /// serve. Ranges for the first predicate are conditional only if a limit is
+ /// set and the range is not in the first planned row group, because the
+ /// limit can be reached first.
+ ///
+ /// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+ pub conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk. It serves every planned row in
+ /// the column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder scan may read, in the order decoding
+/// needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`](super::ParquetPushDecoder::scan_plan),
+/// which documents the contract.
+///
+/// The iterator plans one row group at a time, when the caller asks for its
+/// first range. It owns its state, so the caller can keep it while it pushes
+/// data to the decoder.
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the decoder was built.
+ frontier: RowGroupFrontier,
+ /// The decoder's projection, predicates and batch size.
+ config: Arc<ScanPlanConfig>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// Planned ranges of the current row group not yet returned.
+ pending: std::vec::IntoIter<PlannedRange>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+impl ScanPlan {
+ pub(super) fn new(frontier: RowGroupFrontier, config: ScanPlanConfig) -> Self {
+ let has_limit = frontier.budget.limit().is_some();
+ Self {
+ frontier,
+ config: Arc::new(config),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ pending: Vec::new().into_iter(),
+ done: false,
+ }
+ }
+
+ /// Plan the next row group the decoder will read.
+ ///
+ /// Returns `Ok(None)` when no row group remains. A row group that the
+ /// offset/limit budget removes gives an empty `Vec`.
+ fn plan_next_row_group(&mut self) -> Result<Option<Vec<PlannedRange>>, ParquetError> {
+ // Use the decoder's own row-group walk, so row groups that the
+ // selection or the budget removes are skipped here too.
+ let Some(NextRowGroup {
+ row_group_idx,
+ row_count,
+ selection,
+ budget,
+ }) = self.frontier.next_readable_row_group()?
+ else {
+ return Ok(None);
+ };
+
+ let filtered = self.frontier.has_predicates;
+ let selection = if filtered {
+ // Predicates see every selected row. Offset and limit apply to
+ // their output, so they cannot be applied before decoding.
+ selection
+ } else {
+ // Apply offset and limit as the decoder does before it requests
+ // the output columns.
+ let plan_builder =
+ ReadPlanBuilder::new(self.config.batch_size).with_selection(selection);
+ let BudgetedReadPlan {
+ plan_builder,
+ rows_after_budget,
+ remaining_budget,
+ ..
+ } = budget.apply_to_plan(plan_builder, row_count);
+ self.frontier
+ .update_budget_after_row_group(remaining_budget);
+ if rows_after_budget == 0 {
+ return Ok(Some(vec![]));
+ }
+ plan_builder.selection().cloned()
+ };
+
+ let rows = SelectedRows::new(selection.as_ref(), row_count);
+ let first_row = self.next_row;
+ self.next_row += rows.selected_before(row_count);
+
+ let metadata = Arc::clone(&self.frontier.parquet_metadata);
+ let page_index = metadata
+ .page_index()
+ .is_some_and(|page_index| page_index.has_offset_indexes())
+ .then(|| metadata.page_index_for_row_group(row_group_idx));
+ let row_group = metadata.row_group(row_group_idx);
+
+ // Predicate columns that are cached for the output are fetched with the
+ // selection expanded to batch boundaries. See `fetch_ranges`.
+ let expanded_selection = match (&selection, &self.config.cache_projection) {
+ (Some(selection), Some(_)) if filtered => {
+ Some(selection.expand_to_batch_boundaries(self.config.batch_size, row_count))
+ }
+ _ => None,
+ };
+
+ let columns = RowGroupColumns {
+ row_group_idx,
+ row_count,
+ first_row,
+ rows: &rows,
+ page_index: page_index.as_ref(),
+ };
+ let stages = self
+ .config
+ .predicate_projections
+ .iter()
+ .enumerate()
+ .map(|(idx, mask)| (ScanStage::Predicate(idx), mask))
+ .chain(std::iter::once((
+ ScanStage::Projection,
+ &self.config.projection,
+ )));
+
+ let mut planned_columns = vec![false; row_group.columns().len()];
+ let mut ranges = vec![];
+ for (stage, mask) in stages {
+ let conditional = filtered
+ && (stage != ScanStage::Predicate(0)
+ || (self.has_limit && !self.at_first_row_group));
+ let stage_start = ranges.len();
+ for (column_idx, chunk) in row_group.columns().iter().enumerate() {
+ // The decoder reuses a column that an earlier stage read.
+ if !mask.leaf_included(column_idx) || planned_columns[column_idx] {
+ continue;
+ }
+ planned_columns[column_idx] = true;
+ let is_cached = self
+ .config
+ .cache_projection
+ .as_ref()
+ .is_some_and(|cache| cache.leaf_included(column_idx));
+ let fetch_selection = match (stage, &expanded_selection) {
+ (ScanStage::Predicate(_), Some(expanded)) if is_cached => Some(expanded),
+ _ => selection.as_ref(),
+ };
+ let (chunk_start, chunk_len) = chunk.byte_range();
+ columns.plan(
+ column_idx,
+ chunk_start..chunk_start + chunk_len,
+ fetch_selection,
+ stage,
+ conditional,
+ &mut ranges,
+ );
+ }
+ // Order by when decoding needs each range. For equal first rows,
+ // the wider span (a dictionary page) comes first, so any prefix of
+ // the plan can be decoded.
+ ranges[stage_start..]
+ .sort_by_key(|p| (p.first_row, Reverse(p.last_row), p.range.start));
+ }
+ self.at_first_row_group = false;
+ Ok(Some(ranges))
+ }
+}
+
+impl Iterator for ScanPlan {
+ type Item = PlannedRange;
+
+ fn next(&mut self) -> Option<PlannedRange> {
Review Comment:
this seems pretty fine grained to me (it is per range when the requests are typically requested per column or per row group). On the other hand, this API basically adds enough visibilty for the callers to figure out where row group boundaries are, etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5941934761
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940184708)
**Target:** arrow_reader
Comparing claude/push-decoder-peek-59 (0d91bca5e174821c56fc6529c6bfb6d6a52e8d81) to 76829d5 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/76829d5ce4562616a69ea8b2696e96160514482f..0d91bca5e174821c56fc6529c6bfb6d6a52e8d81)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 842.3±12.73µs ? ?/sec 1.00 842.1±10.00µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 951.9±6.37µs ? ?/sec 1.00 955.5±8.49µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 846.4±12.89µs ? ?/sec 1.00 847.3±10.12µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.00 279.9±0.21µs ? ?/sec 1.02 284.6±0.24µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs, large values 1.03 38.9±0.37ms ? ?/sec 1.00 37.6±0.20ms ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.00 387.6±0.45µs ? ?/sec 1.01 392.0±1.31µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.00 288.9±0.17µs ? ?/sec 1.00 288.4±0.38µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.00 407.2±5.02µs ? ?/sec 1.01 409.2±6.82µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.01 468.5±3.86µs ? ?/sec 1.00 462.4±2.78µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.00 410.7±3.07µs ? ?/sec 1.00 409.2±2.54µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.3±0.05µs ? ?/sec 1.00 80.3±0.05µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.00 111.3±0.12µs ? ?/sec 1.00 111.1±0.15µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 84.3±0.05µs ? ?/sec 1.00 84.1±0.05µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.00 147.0±0.54µs ? ?/sec 1.00 147.2±0.59µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.06 150.1±3.35µs ? ?/sec 1.00 141.9±0.64µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.02 150.1±0.58µs ? ?/sec 1.00 147.4±0.58µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.02 154.0±0.22µs ? ?/sec 1.00 151.3±0.47µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, mandatory, no NULLs 1.00 226.3±0.27µs ? ?/sec 1.00 226.2±0.47µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, half NULLs 1.00 244.4±1.97µs ? ?/sec 1.00 243.3±1.71µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, no NULLs 1.00 230.2±0.26µs ? ?/sec 1.00 230.0±0.36µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 967.7±2.25µs ? ?/sec 1.00 968.5±0.81µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 672.2±0.54µs ? ?/sec 1.00 673.8±0.54µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 972.6±2.28µs ? ?/sec 1.00 971.5±0.76µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 189.2±0.22µs ? ?/sec 1.00 188.2±0.33µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 294.3±0.19µs ? ?/sec 1.00 295.4±0.41µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 193.4±0.22µs ? ?/sec 1.00 192.9±0.31µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.01 130.1±0.31µs ? ?/sec 1.00 128.9±0.24µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 183.1±0.22µs ? ?/sec 1.01 184.0±0.36µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 132.6±0.61µs ? ?/sec 1.00 132.7±0.36µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 60.1±0.04µs ? ?/sec 1.02 61.1±0.03µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 146.7±0.10µs ? ?/sec 1.01 147.9±0.67µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 63.1±0.05µs ? ?/sec 1.02 64.0±0.05µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 802.6±0.44µs ? ?/sec 1.01 807.5±1.14µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 509.5±0.37µs ? ?/sec 1.01 512.2±0.50µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.00 807.4±0.84µs ? ?/sec 1.00 808.2±0.66µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.00 22.5±0.08µs ? ?/sec 1.02 23.0±0.03µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.00 130.5±0.09µs ? ?/sec 1.01 132.3±0.12µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.01 28.0±0.04µs ? ?/sec 1.00 27.7±0.07µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.02 75.2±0.36µs ? ?/sec 1.00 73.8±0.20µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.00 129.0±0.16µs ? ?/sec 1.00 128.7±0.15µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.01 79.4±0.19µs ? ?/sec 1.00 78.9±0.44µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.00 5.4±0.03µs ? ?/sec 1.00 5.4±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.00 92.7±0.09µs ? ?/sec 1.00 92.7±0.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.00 8.9±0.04µs ? ?/sec 1.00 8.9±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.00 153.3±10.92µs ? ?/sec 1.03 158.1±14.65µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.01 200.5±0.52µs ? ?/sec 1.00 199.1±0.23µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.00 151.3±0.38µs ? ?/sec 1.00 151.9±0.42µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.00 7.6±0.02µs ? ?/sec 1.00 7.6±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.00 128.2±0.07µs ? ?/sec 1.00 127.6±0.10µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.00 11.2±0.02µs ? ?/sec 1.00 11.2±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.00 299.6±0.83µs ? ?/sec 1.00 300.9±0.97µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.00 269.6±1.40µs ? ?/sec 1.00 269.5±0.40µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.01 306.8±0.94µs ? ?/sec 1.00 305.2±1.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.00 12.3±0.04µs ? ?/sec 1.01 12.4±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.00 123.5±0.10µs ? ?/sec 1.01 125.1±0.16µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.00 16.1±0.04µs ? ?/sec 1.02 16.5±0.02µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed increasing value 1.00 87.5±0.63µs ? ?/sec 1.01 88.1±0.73µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed single value 1.00 79.5±0.33µs ? ?/sec 1.00 79.3±0.65µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value 1.00 50.7±0.52µs ? ?/sec 1.00 50.8±0.36µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value 1.00 46.1±0.20µs ? ?/sec 1.00 46.1±0.36µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value 1.00 74.9±0.37µs ? ?/sec 1.00 74.9±0.24µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 89.2±1.02µs ? ?/sec 1.01 89.7±0.98µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.00 87.5±0.51µs ? ?/sec 1.00 87.9±0.45µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.00 91.2±1.00µs ? ?/sec 1.00 91.7±0.92µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed stepped increasing value 1.00 107.9±0.20µs ? ?/sec 1.00 108.0±0.22µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.00 126.0±0.83µs ? ?/sec 1.00 126.5±0.92µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.00 144.5±0.45µs ? ?/sec 1.00 144.7±0.47µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.00 129.5±0.89µs ? ?/sec 1.00 129.8±0.97µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 53.9±0.12µs ? ?/sec 1.00 54.1±0.05µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 107.5±0.15µs ? ?/sec 1.00 107.6±0.16µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.01 57.9±0.05µs ? ?/sec 1.00 57.4±0.10µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 84.3±0.05µs ? ?/sec 1.00 84.3±0.39µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 125.2±0.19µs ? ?/sec 1.00 125.2±0.19µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 87.9±0.08µs ? ?/sec 1.00 87.8±0.08µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 47.1±0.30µs ? ?/sec 1.03 48.4±0.10µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.00 104.5±0.18µs ? ?/sec 1.00 104.6±0.35µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.00 50.4±0.05µs ? ?/sec 1.00 50.6±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed increasing value 1.01 78.6±0.11µs ? ?/sec 1.00 78.2±0.13µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed single value 1.01 75.3±0.12µs ? ?/sec 1.00 74.9±0.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value 1.01 43.6±0.07µs ? ?/sec 1.00 43.4±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value 1.01 41.8±0.10µs ? ?/sec 1.00 41.5±0.05µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value 1.00 68.3±0.07µs ? ?/sec 1.00 68.0±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 79.8±0.08µs ? ?/sec 1.01 80.3±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.00 83.7±0.12µs ? ?/sec 1.00 83.5±0.17µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.00 81.9±0.06µs ? ?/sec 1.00 82.0±0.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed stepped increasing value 1.01 101.1±0.21µs ? ?/sec 1.00 100.6±0.11µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 114.9±0.11µs ? ?/sec 1.00 115.4±0.65µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 141.0±0.18µs ? ?/sec 1.00 140.4±0.22µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.00 118.9±0.14µs ? ?/sec 1.00 118.5±0.16µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 83.2±0.06µs ? ?/sec 1.00 83.2±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 124.7±0.19µs ? ?/sec 1.00 124.7±0.18µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 87.7±0.07µs ? ?/sec 1.00 87.4±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 87.0±0.23µs ? ?/sec 1.00 86.7±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 127.8±0.17µs ? ?/sec 1.00 128.4±0.19µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 90.6±0.13µs ? ?/sec 1.00 90.3±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.02 54.7±0.06µs ? ?/sec 1.00 53.9±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 109.5±0.15µs ? ?/sec 1.00 109.5±0.18µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.01 58.7±0.06µs ? ?/sec 1.00 58.2±0.06µs ? ?/sec
arrow_array_reader/Int16Array/binary packed increasing value 1.00 54.6±0.64µs ? ?/sec 1.00 54.6±0.53µs ? ?/sec
arrow_array_reader/Int16Array/binary packed single value 1.00 45.9±0.64µs ? ?/sec 1.03 47.3±0.42µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value 1.00 33.5±0.35µs ? ?/sec 1.00 33.5±0.35µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value 1.00 28.8±0.35µs ? ?/sec 1.03 29.5±0.23µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value 1.00 57.5±0.24µs ? ?/sec 1.01 58.0±0.28µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.00 72.1±1.03µs ? ?/sec 1.00 72.3±0.99µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.00 70.5±0.52µs ? ?/sec 1.00 70.7±0.53µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.00 74.1±1.00µs ? ?/sec 1.00 74.2±0.90µs ? ?/sec
arrow_array_reader/Int16Array/binary packed stepped increasing value 1.00 74.7±0.15µs ? ?/sec 1.00 74.7±0.18µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.00 92.0±0.33µs ? ?/sec 1.00 91.8±0.36µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.00 110.6±0.19µs ? ?/sec 1.00 110.5±0.27µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.00 95.2±0.33µs ? ?/sec 1.00 95.7±0.39µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.9±0.02µs ? ?/sec 1.01 21.1±0.03µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.01 74.9±0.12µs ? ?/sec 1.00 74.0±0.15µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.02µs ? ?/sec 1.01 24.6±0.04µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.00 51.1±0.05µs ? ?/sec 1.00 51.2±0.05µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.00 91.8±0.21µs ? ?/sec 1.00 92.0±0.17µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.00 55.0±0.04µs ? ?/sec 1.00 55.2±0.05µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.00 13.8±0.03µs ? ?/sec 1.00 13.8±0.05µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.00 71.4±0.13µs ? ?/sec 1.00 71.5±0.15µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.00 17.5±0.02µs ? ?/sec 1.01 17.7±0.01µs ? ?/sec
arrow_array_reader/Int32Array/binary packed increasing value 1.00 50.8±0.70µs ? ?/sec 1.01 51.2±0.61µs ? ?/sec
arrow_array_reader/Int32Array/binary packed single value 1.00 42.9±0.38µs ? ?/sec 1.02 43.7±0.43µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value 1.00 31.4±0.34µs ? ?/sec 1.01 31.7±0.39µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value 1.00 27.1±0.18µs ? ?/sec 1.02 27.6±0.24µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value 1.00 55.7±0.25µs ? ?/sec 1.01 56.0±0.28µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.00 70.4±0.99µs ? ?/sec 1.00 70.6±0.93µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.00 69.6±0.49µs ? ?/sec 1.00 69.8±0.55µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.00 72.5±1.02µs ? ?/sec 1.00 72.7±0.93µs ? ?/sec
arrow_array_reader/Int32Array/binary packed stepped increasing value 1.00 70.9±0.17µs ? ?/sec 1.00 71.0±0.18µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.00 89.5±0.84µs ? ?/sec 1.00 89.2±0.81µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.00 108.2±0.39µs ? ?/sec 1.01 109.0±0.48µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.00 93.1±0.82µs ? ?/sec 1.00 93.1±0.80µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.3±0.02µs ? ?/sec 1.01 17.5±0.03µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.01 71.3±0.14µs ? ?/sec 1.00 70.7±0.13µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.00 21.0±0.03µs ? ?/sec 1.00 20.9±0.02µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.00 47.6±0.02µs ? ?/sec 1.00 47.8±0.03µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.01 89.2±0.22µs ? ?/sec 1.00 88.6±0.16µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.00 51.1±0.04µs ? ?/sec 1.00 51.3±0.06µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.00 9.9±0.03µs ? ?/sec 1.00 9.9±0.02µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.00 68.2±0.15µs ? ?/sec 1.00 68.1±0.16µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.00 13.5±0.02µs ? ?/sec 1.00 13.6±0.03µs ? ?/sec
arrow_array_reader/Int64Array/binary packed increasing value 1.00 41.5±0.13µs ? ?/sec 1.00 41.3±0.33µs ? ?/sec
arrow_array_reader/Int64Array/binary packed single value 1.00 38.4±0.09µs ? ?/sec 1.00 38.6±0.36µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value 1.01 24.7±0.06µs ? ?/sec 1.00 24.6±0.34µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value 1.00 22.8±0.05µs ? ?/sec 1.00 22.9±0.37µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value 1.00 49.0±0.06µs ? ?/sec 1.00 49.0±0.19µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.00 60.8±0.06µs ? ?/sec 1.00 61.1±0.23µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.00 65.0±0.11µs ? ?/sec 1.00 64.8±0.17µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.00 62.6±0.05µs ? ?/sec 1.01 63.1±0.25µs ? ?/sec
arrow_array_reader/Int64Array/binary packed stepped increasing value 1.00 63.6±0.09µs ? ?/sec 1.00 63.4±0.20µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.00 77.9±0.13µs ? ?/sec 1.00 77.9±0.25µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.00 103.8±0.20µs ? ?/sec 1.00 103.8±0.18µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.00 81.7±0.15µs ? ?/sec 1.00 81.8±0.23µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.00 45.8±0.04µs ? ?/sec 1.00 45.7±0.07µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.2±0.10µs ? ?/sec 1.00 87.5±0.14µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.4±0.03µs ? ?/sec 1.00 49.4±0.04µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.00 49.9±0.02µs ? ?/sec 1.00 49.9±0.05µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 90.5±0.19µs ? ?/sec 1.00 90.7±0.15µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.00 53.6±0.03µs ? ?/sec 1.00 53.4±0.04µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.04 16.9±0.03µs ? ?/sec 1.00 16.2±0.03µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.00 72.8±0.11µs ? ?/sec 1.01 73.2±0.12µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.00 20.0±0.02µs ? ?/sec 1.02 20.4±0.03µs ? ?/sec
arrow_array_reader/Int8Array/binary packed increasing value 1.00 54.1±0.69µs ? ?/sec 1.00 54.2±0.61µs ? ?/sec
arrow_array_reader/Int8Array/binary packed single value 1.00 46.0±0.36µs ? ?/sec 1.01 46.5±0.32µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value 1.00 33.2±0.35µs ? ?/sec 1.00 33.4±0.39µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value 1.00 28.9±0.16µs ? ?/sec 1.00 29.0±0.19µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value 1.00 57.6±0.24µs ? ?/sec 1.00 57.7±0.29µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.00 71.7±0.80µs ? ?/sec 1.00 72.0±0.72µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.00 70.1±0.41µs ? ?/sec 1.01 70.8±0.40µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.00 73.9±0.80µs ? ?/sec 1.00 74.2±0.70µs ? ?/sec
arrow_array_reader/Int8Array/binary packed stepped increasing value 1.00 74.4±0.19µs ? ?/sec 1.00 74.5±0.17µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.00 91.7±0.23µs ? ?/sec 1.00 92.1±0.17µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.00 109.9±0.16µs ? ?/sec 1.01 110.9±0.17µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 95.1±0.22µs ? ?/sec 1.01 95.6±0.18µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.8±0.04µs ? ?/sec 1.01 20.9±0.04µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.00 74.1±0.13µs ? ?/sec 1.00 74.2±0.13µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.01 24.5±0.05µs ? ?/sec 1.00 24.3±0.04µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.00 51.2±0.05µs ? ?/sec 1.00 51.4±0.05µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.00 91.5±0.18µs ? ?/sec 1.00 91.6±0.17µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.01 54.9±0.05µs ? ?/sec 1.00 54.6±0.06µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.01 13.7±0.03µs ? ?/sec 1.00 13.6±0.05µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.00 71.1±0.15µs ? ?/sec 1.01 71.7±0.15µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.01 17.5±0.02µs ? ?/sec 1.00 17.3±0.04µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/90pct NULLs 1.00 882.5±1.36µs ? ?/sec 1.00 879.0±1.29µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/99pct NULLs 1.00 449.2±4.60µs ? ?/sec 1.00 449.6±4.27µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/half NULLs 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
arrow_array_reader/ListArray/Fixed32List/no NULLs 1.04 3.1±0.02ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
arrow_array_reader/ListArray/Int32List/90pct NULLs 1.01 868.2±0.93µs ? ?/sec 1.00 862.9±1.05µs ? ?/sec
arrow_array_reader/ListArray/Int32List/99pct NULLs 1.00 449.6±4.84µs ? ?/sec 1.00 449.0±3.87µs ? ?/sec
arrow_array_reader/ListArray/Int32List/half NULLs 1.01 1963.6±5.80µs ? ?/sec 1.00 1953.0±3.58µs ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs 1.00 2.7±0.00ms ? ?/sec 1.00 2.7±0.00ms ? ?/sec
arrow_array_reader/ListArray/StringList/90pct NULLs 1.01 935.0±1.51µs ? ?/sec 1.00 924.3±1.92µs ? ?/sec
arrow_array_reader/ListArray/StringList/99pct NULLs 1.00 455.7±4.49µs ? ?/sec 1.00 454.5±3.38µs ? ?/sec
arrow_array_reader/ListArray/StringList/half NULLs 1.01 3.0±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
arrow_array_reader/ListArray/StringList/no NULLs 1.02 4.9±0.03ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
arrow_array_reader/PlainStringToDictionary/high cardinality 1.00 1077.7±14.77µs ? ?/sec 1.00 1078.5±15.73µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/low cardinality 1.01 1009.3±13.74µs ? ?/sec 1.00 1003.3±15.43µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 1 1.01 1077.3±14.64µs ? ?/sec 1.00 1071.2±15.80µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 2 1.01 1029.9±13.91µs ? ?/sec 1.00 1017.5±16.01µs ? ?/sec
arrow_array_reader/StringArray/const delta byte array encoded, mandatory, no NULLs 1.00 542.8±0.61µs ? ?/sec 1.00 544.5±3.19µs ? ?/sec
arrow_array_reader/StringArray/const delta length byte array encoded, mandatory, no NULLs 1.03 223.6±1.78µs ? ?/sec 1.00 216.3±0.62µs ? ?/sec
arrow_array_reader/StringArray/const prefix delta byte array encoded, mandatory, no NULLs 1.00 777.2±43.96µs ? ?/sec 1.00 778.6±42.83µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.00 284.7±0.24µs ? ?/sec 1.00 283.6±0.19µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.00 390.9±0.53µs ? ?/sec 1.00 391.1±1.42µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.00 288.6±0.15µs ? ?/sec 1.00 289.2±0.25µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.00 441.3±4.71µs ? ?/sec 1.01 447.7±3.42µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.00 490.5±2.96µs ? ?/sec 1.01 495.7±2.86µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.00 447.5±4.43µs ? ?/sec 1.03 461.1±3.08µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.00 193.7±0.56µs ? ?/sec 1.00 193.8±1.36µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.01 244.9±0.70µs ? ?/sec 1.00 243.6±1.21µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.00 197.6±0.68µs ? ?/sec 1.00 197.4±1.41µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.3±0.04µs ? ?/sec 1.00 80.4±0.06µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.00 111.6±0.13µs ? ?/sec 1.00 111.4±0.12µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 84.3±0.04µs ? ?/sec 1.00 84.2±0.06µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.02 222.4±0.40µs ? ?/sec 1.00 218.8±0.30µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.00 185.3±0.27µs ? ?/sec 1.00 184.9±0.30µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.01 228.8±0.36µs ? ?/sec 1.00 226.4±0.38µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed increasing value 1.00 54.3±0.64µs ? ?/sec 1.00 54.4±0.63µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed single value 1.00 46.2±0.35µs ? ?/sec 1.00 46.4±0.27µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value 1.00 33.3±0.36µs ? ?/sec 1.01 33.5±0.39µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value 1.00 29.0±0.19µs ? ?/sec 1.00 29.1±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value 1.00 57.6±0.25µs ? ?/sec 1.01 58.0±0.28µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.00 75.4±0.19µs ? ?/sec 1.00 75.7±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.00 72.6±0.17µs ? ?/sec 1.00 72.9±0.18µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 77.7±0.17µs ? ?/sec 1.00 78.0±0.16µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed stepped increasing value 1.00 74.5±0.18µs ? ?/sec 1.00 74.7±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.00 97.3±0.13µs ? ?/sec 1.01 97.8±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.00 114.1±0.24µs ? ?/sec 1.00 114.2±0.22µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 100.7±0.12µs ? ?/sec 1.01 101.3±0.16µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.9±0.03µs ? ?/sec 1.00 20.9±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.00 74.1±0.13µs ? ?/sec 1.00 74.3±0.24µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.03µs ? ?/sec 1.01 24.6±0.02µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.01 51.6±0.05µs ? ?/sec 1.00 51.3±0.05µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.00 92.1±0.17µs ? ?/sec 1.00 91.9±0.12µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.00 54.8±0.05µs ? ?/sec 1.00 54.8±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.00 13.7±0.04µs ? ?/sec 1.00 13.8±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.01 71.5±0.18µs ? ?/sec 1.00 70.9±0.15µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.00 17.6±0.05µs ? ?/sec 1.00 17.6±0.01µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed increasing value 1.00 51.4±0.59µs ? ?/sec 1.00 51.6±0.63µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed single value 1.00 42.8±0.60µs ? ?/sec 1.01 43.2±0.30µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value 1.00 31.7±0.33µs ? ?/sec 1.00 31.8±0.39µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value 1.00 27.0±0.33µs ? ?/sec 1.02 27.5±0.20µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value 1.00 55.8±0.24µs ? ?/sec 1.01 56.3±0.30µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.00 70.8±0.99µs ? ?/sec 1.01 71.1±1.00µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.00 69.0±0.50µs ? ?/sec 1.01 69.4±0.50µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.00 72.6±0.94µs ? ?/sec 1.00 72.9±0.99µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed stepped increasing value 1.00 70.8±0.12µs ? ?/sec 1.01 71.4±0.17µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.00 89.6±0.84µs ? ?/sec 1.00 89.4±0.91µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.00 108.0±0.37µs ? ?/sec 1.00 108.3±0.45µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.00 93.2±0.81µs ? ?/sec 1.00 93.2±0.91µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.4±0.01µs ? ?/sec 1.01 17.6±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.00 70.7±0.12µs ? ?/sec 1.00 71.0±0.14µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.00 21.0±0.07µs ? ?/sec 1.00 21.0±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.00 47.8±0.02µs ? ?/sec 1.00 48.0±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.00 89.0±0.19µs ? ?/sec 1.00 88.8±0.14µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.00 51.3±0.05µs ? ?/sec 1.00 51.4±0.05µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.00 10.1±0.01µs ? ?/sec 1.00 10.0±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.00 68.6±0.25µs ? ?/sec 1.00 68.8±0.14µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.00 13.7±0.04µs ? ?/sec 1.01 13.9±0.01µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed increasing value 1.01 41.9±0.12µs ? ?/sec 1.00 41.5±0.31µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed single value 1.00 38.8±0.09µs ? ?/sec 1.00 38.7±0.31µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value 1.00 24.7±0.06µs ? ?/sec 1.00 24.6±0.33µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value 1.01 23.2±0.05µs ? ?/sec 1.00 23.0±0.34µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value 1.00 49.1±0.05µs ? ?/sec 1.00 49.0±0.18µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.00 60.6±0.07µs ? ?/sec 1.01 61.0±0.20µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.01 65.4±0.12µs ? ?/sec 1.00 65.0±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.00 63.0±0.04µs ? ?/sec 1.00 63.0±0.22µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed stepped increasing value 1.00 63.7±0.08µs ? ?/sec 1.00 63.4±0.19µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 77.9±0.16µs ? ?/sec 1.00 77.9±0.26µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.00 104.0±0.18µs ? ?/sec 1.00 104.3±0.19µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.00 81.8±0.13µs ? ?/sec 1.00 81.8±0.25µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.00 45.9±0.03µs ? ?/sec 1.00 45.8±0.03µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.6±0.11µs ? ?/sec 1.00 87.6±0.14µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.7±0.04µs ? ?/sec 1.00 49.6±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.00 50.0±0.07µs ? ?/sec 1.00 50.0±0.03µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.00 91.0±0.21µs ? ?/sec 1.00 91.0±0.14µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.00 53.5±0.05µs ? ?/sec 1.00 53.5±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.04 17.0±0.02µs ? ?/sec 1.00 16.3±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.00 73.4±0.13µs ? ?/sec 1.00 73.3±0.15µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.00 20.0±0.03µs ? ?/sec 1.01 20.3±0.08µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed increasing value 1.00 54.4±0.66µs ? ?/sec 1.00 54.3±0.63µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed single value 1.00 45.6±0.63µs ? ?/sec 1.03 46.9±0.43µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value 1.00 33.3±0.36µs ? ?/sec 1.00 33.4±0.37µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value 1.00 28.6±0.33µs ? ?/sec 1.03 29.4±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value 1.00 57.5±0.25µs ? ?/sec 1.00 57.8±0.28µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.00 75.1±0.36µs ? ?/sec 1.00 75.0±0.33µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.00 72.0±0.25µs ? ?/sec 1.00 72.1±0.25µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.00 77.3±0.36µs ? ?/sec 1.00 77.2±0.32µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed stepped increasing value 1.00 74.2±0.16µs ? ?/sec 1.00 74.5±0.20µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.00 96.5±0.12µs ? ?/sec 1.00 96.3±0.14µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.00 113.2±0.17µs ? ?/sec 1.00 112.8±0.20µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.00 100.1±0.09µs ? ?/sec 1.00 99.7±0.13µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.7±0.03µs ? ?/sec 1.01 20.9±0.06µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.01 74.1±0.26µs ? ?/sec 1.00 73.6±0.13µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.1±0.08µs ? ?/sec 1.00 24.2±0.04µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.00 51.0±0.03µs ? ?/sec 1.00 51.1±0.06µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.00 91.9±0.18µs ? ?/sec 1.00 91.9±0.18µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.00 54.3±0.05µs ? ?/sec 1.01 54.8±0.03µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.00 13.5±0.03µs ? ?/sec 1.01 13.6±0.03µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.01 71.4±0.13µs ? ?/sec 1.00 70.7±0.16µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.00 17.3±0.03µs ? ?/sec 1.00 17.2±0.02µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, half NULLs 1.00 68.3±0.17µs ? ?/sec 1.01 68.9±0.11µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, no NULLs 1.00 13.9±0.08µs ? ?/sec 1.00 13.9±0.03µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, half NULLs 1.00 128.4±0.16µs ? ?/sec 1.01 130.2±0.13µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, no NULLs 1.00 65.5±0.07µs ? ?/sec 1.02 67.0±0.07µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>arrow_reader — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-35u357fo-edc6f289-v5vl
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5940184708-3014-rnbr2 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 3005.7s |
| Peak memory | 192.2 MiB |
| Avg memory | 39.4 MiB |
| CPU user | 2988.9s |
| CPU sys | 11.4s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 3015.6s |
| Peak memory | 190.8 MiB |
| Avg memory | 42.7 MiB |
| CPU user | 2999.8s |
| CPU sys | 11.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5940184708:finish:3664ff2e6f04adb32374e6e21eab9a54 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946213654
run benchmark push_decoder
```yaml
env:
BENCH_FILTER: scan_plan
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946424091
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946300658)
**Target:** push_decoder
Comparing claude/push-decoder-peek-59 (ec3b0d2c50ee1d475fc773b1bdb71752854669cc) to 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81 [diff](https://github.com/apache/arrow-rs/compare/0d91bca5e174821c56fc6529c6bfb6d6a52e8d81..ec3b0d2c50ee1d475fc773b1bdb71752854669cc)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
env:
BENCH_FILTER: scan_plan
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
push_decoder/scan_plan/first_range/all/10000cols_10pages 1.00 368.8±1.35µs ? ?/sec 1.06 389.5±2.10µs ? ?/sec
push_decoder/scan_plan/first_range/all/1000cols_100pages 1.00 36.0±0.21µs ? ?/sec 1.00 35.9±0.14µs ? ?/sec
push_decoder/scan_plan/first_range/all/1000cols_10pages 1.00 35.6±0.16µs ? ?/sec 1.04 37.1±0.16µs ? ?/sec
push_decoder/scan_plan/first_range/all/100cols_10pages 1.00 4.1±0.03µs ? ?/sec 1.02 4.2±0.02µs ? ?/sec
push_decoder/scan_plan/first_range/narrow/10000cols_10pages 1.53 9.5±0.37µs ? ?/sec 1.00 6.2±0.01µs ? ?/sec
push_decoder/scan_plan/first_range/narrow/1000cols_100pages 1.16 1574.8±4.73ns ? ?/sec 1.00 1357.2±7.64ns ? ?/sec
push_decoder/scan_plan/first_range/narrow/1000cols_10pages 1.16 1604.1±4.32ns ? ?/sec 1.00 1388.3±11.35ns ? ?/sec
push_decoder/scan_plan/first_range/narrow/100cols_10pages 1.04 963.3±5.95ns ? ?/sec 1.00 927.4±7.38ns ? ?/sec
push_decoder/scan_plan/first_range/selection/10000cols_10pages 1.00 2.6±0.01ms ? ?/sec 1.04 2.7±0.02ms ? ?/sec
push_decoder/scan_plan/first_range/selection/1000cols_100pages 1.00 374.8±1.86µs ? ?/sec 1.06 397.1±0.47µs ? ?/sec
push_decoder/scan_plan/first_range/selection/1000cols_10pages 1.00 238.8±1.12µs ? ?/sec 1.00 238.0±1.18µs ? ?/sec
push_decoder/scan_plan/first_range/selection/100cols_10pages 1.01 24.9±0.10µs ? ?/sec 1.00 24.7±0.11µs ? ?/sec
push_decoder/scan_plan/first_reader/all/10000cols_10pages 1.00 4.4±0.04ms ? ?/sec
push_decoder/scan_plan/first_reader/all/1000cols_100pages 1.00 610.6±1.46µs ? ?/sec
push_decoder/scan_plan/first_reader/all/1000cols_10pages 1.00 428.1±1.56µs ? ?/sec
push_decoder/scan_plan/first_reader/all/100cols_10pages 1.00 43.1±0.09µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/10000cols_10pages 1.00 195.1±5.33µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/1000cols_100pages 1.00 22.5±0.02µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/1000cols_10pages 1.00 21.6±0.02µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/100cols_10pages 1.00 6.4±0.01µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/10000cols_10pages 1.00 9.9±0.10ms ? ?/sec
push_decoder/scan_plan/first_reader/selection/1000cols_100pages 1.00 1229.4±4.77µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/1000cols_10pages 1.00 952.3±4.46µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/100cols_10pages 1.00 93.6±0.38µs ? ?/sec
push_decoder/scan_plan/whole_scan/all/10000cols_10pages 1.00 7.0±0.01ms ? ?/sec 1.01 7.0±0.02ms ? ?/sec
push_decoder/scan_plan/whole_scan/all/1000cols_100pages 1.00 6.4±0.01ms ? ?/sec 1.00 6.5±0.02ms ? ?/sec
push_decoder/scan_plan/whole_scan/all/1000cols_10pages 1.01 622.0±1.52µs ? ?/sec 1.00 614.7±1.61µs ? ?/sec
push_decoder/scan_plan/whole_scan/all/100cols_10pages 1.01 33.2±0.05µs ? ?/sec 1.00 32.8±0.05µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/10000cols_10pages 1.15 10.7±0.02µs ? ?/sec 1.00 9.3±0.41µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/1000cols_100pages 1.03 27.3±0.04µs ? ?/sec 1.00 26.4±0.04µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/1000cols_10pages 1.08 4.2±0.01µs ? ?/sec 1.00 3.8±0.01µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/100cols_10pages 1.02 3.5±0.01µs ? ?/sec 1.00 3.4±0.01µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/10000cols_10pages 1.00 9.9±0.02ms ? ?/sec 1.02 10.1±0.02ms ? ?/sec
push_decoder/scan_plan/whole_scan/selection/1000cols_100pages 1.00 1052.2±2.12µs ? ?/sec 1.02 1074.0±2.58µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/1000cols_10pages 1.00 894.9±1.64µs ? ?/sec 1.00 899.4±2.17µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/100cols_10pages 1.04 61.4±0.19µs ? ?/sec 1.00 58.8±0.10µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>push_decoder — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-anhwsxxa-057cef0d-x2cz
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5946300658-3019-b8w4f 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench push_decoder
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.9 GiB |
| CPU user | 372.2s |
| CPU sys | 0.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 250.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.9 GiB |
| CPU user | 243.5s |
| CPU sys | 0.8s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5946300658:finish:be3218712c31477851aa14a70c2d1aa3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6040192219
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-6040190549)
**Target:** arrow_reader
**Sharding factor: 1** (1 workers).
Comparing claude/push-decoder-peek-59 (a3b72c1c1d3341b5d20a3844d50c65069183718e) to 02accce (merge-base) [diff](https://github.com/apache/arrow-rs/compare/02accced63a992a20c91cbb5ac24261863531751..a3b72c1c1d3341b5d20a3844d50c65069183718e)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6040190549:start:0181a7721d6e17f04122c584cfbacade -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6082279502
🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946353229
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946213654)
**Target:** push_decoder
Comparing claude/push-decoder-peek-59 (7dae14104f80fdcb42506ebfc94e211f5b6f583d) to 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81 [diff](https://github.com/apache/arrow-rs/compare/0d91bca5e174821c56fc6529c6bfb6d6a52e8d81..7dae14104f80fdcb42506ebfc94e211f5b6f583d)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
env:
BENCH_FILTER: scan_plan
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
push_decoder/scan_plan/first_range/all/10000cols_10pages 1.00 375.1±1.03µs ? ?/sec 1.06 397.3±14.65µs ? ?/sec
push_decoder/scan_plan/first_range/all/1000cols_100pages 1.00 35.0±0.11µs ? ?/sec 1.12 39.1±1.25µs ? ?/sec
push_decoder/scan_plan/first_range/all/1000cols_10pages 1.00 35.8±0.06µs ? ?/sec 1.11 39.6±1.36µs ? ?/sec
push_decoder/scan_plan/first_range/all/100cols_10pages 1.00 4.1±0.02µs ? ?/sec 1.10 4.5±0.14µs ? ?/sec
push_decoder/scan_plan/first_range/narrow/10000cols_10pages 1.31 8.2±0.02µs ? ?/sec 1.00 6.2±0.03µs ? ?/sec
push_decoder/scan_plan/first_range/narrow/1000cols_100pages 1.08 1565.7±4.44ns ? ?/sec 1.00 1443.3±26.14ns ? ?/sec
push_decoder/scan_plan/first_range/narrow/1000cols_10pages 1.11 1566.7±5.98ns ? ?/sec 1.00 1417.3±26.55ns ? ?/sec
push_decoder/scan_plan/first_range/narrow/100cols_10pages 1.00 961.9±5.02ns ? ?/sec 1.05 1009.5±26.14ns ? ?/sec
push_decoder/scan_plan/first_range/selection/10000cols_10pages 1.05 2.6±0.01ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
push_decoder/scan_plan/first_range/selection/1000cols_100pages 1.00 376.2±1.45µs ? ?/sec 1.03 386.9±2.32µs ? ?/sec
push_decoder/scan_plan/first_range/selection/1000cols_10pages 1.00 241.5±0.96µs ? ?/sec 1.00 240.5±2.78µs ? ?/sec
push_decoder/scan_plan/first_range/selection/100cols_10pages 1.00 25.0±0.11µs ? ?/sec 1.00 24.9±0.28µs ? ?/sec
push_decoder/scan_plan/first_reader/all/10000cols_10pages 1.00 4.3±0.02ms ? ?/sec
push_decoder/scan_plan/first_reader/all/1000cols_100pages 1.00 594.4±1.43µs ? ?/sec
push_decoder/scan_plan/first_reader/all/1000cols_10pages 1.00 430.6±2.64µs ? ?/sec
push_decoder/scan_plan/first_reader/all/100cols_10pages 1.00 43.2±0.08µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/10000cols_10pages 1.00 179.8±3.77µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/1000cols_100pages 1.00 22.4±0.03µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/1000cols_10pages 1.00 21.6±0.03µs ? ?/sec
push_decoder/scan_plan/first_reader/narrow/100cols_10pages 1.00 6.4±0.02µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/10000cols_10pages 1.00 9.7±0.07ms ? ?/sec
push_decoder/scan_plan/first_reader/selection/1000cols_100pages 1.00 1214.1±6.37µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/1000cols_10pages 1.00 952.9±3.88µs ? ?/sec
push_decoder/scan_plan/first_reader/selection/100cols_10pages 1.00 93.7±0.34µs ? ?/sec
push_decoder/scan_plan/whole_scan/all/10000cols_10pages 1.00 7.0±0.01ms ? ?/sec 1.00 7.0±0.02ms ? ?/sec
push_decoder/scan_plan/whole_scan/all/1000cols_100pages 1.02 6.5±0.01ms ? ?/sec 1.00 6.4±0.01ms ? ?/sec
push_decoder/scan_plan/whole_scan/all/1000cols_10pages 1.01 628.7±1.14µs ? ?/sec 1.00 623.0±1.68µs ? ?/sec
push_decoder/scan_plan/whole_scan/all/100cols_10pages 1.26 42.0±1.06µs ? ?/sec 1.00 33.4±0.18µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/10000cols_10pages 1.21 10.7±0.03µs ? ?/sec 1.00 8.9±0.23µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/1000cols_100pages 1.01 27.3±0.04µs ? ?/sec 1.00 27.1±0.09µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/1000cols_10pages 1.03 4.1±0.00µs ? ?/sec 1.00 4.0±0.04µs ? ?/sec
push_decoder/scan_plan/whole_scan/narrow/100cols_10pages 1.00 3.5±0.01µs ? ?/sec 1.01 3.6±0.04µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/10000cols_10pages 1.01 9.8±0.01ms ? ?/sec 1.00 9.7±0.04ms ? ?/sec
push_decoder/scan_plan/whole_scan/selection/1000cols_100pages 1.00 1048.0±1.87µs ? ?/sec 1.01 1061.5±3.67µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/1000cols_10pages 1.00 897.6±1.34µs ? ?/sec 1.00 898.1±4.00µs ? ?/sec
push_decoder/scan_plan/whole_scan/selection/100cols_10pages 1.00 61.5±0.31µs ? ?/sec 1.03 63.2±0.45µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>push_decoder — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-anhwsxxa-0503ffe1-4wwm
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5946213654-3017-shlqg 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench push_decoder
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.9 GiB |
| CPU user | 371.2s |
| CPU sys | 0.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 250.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.8 GiB |
| CPU user | 243.5s |
| CPU sys | 0.8s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5946213654:finish:d690cd1086e528986ab6363c20c7720b -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940094058
checking it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946214276
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946213654)
**Target:** push_decoder
**Sharding factor: 1** (1 workers).
Comparing claude/push-decoder-peek-59 (7dae14104f80fdcb42506ebfc94e211f5b6f583d) to 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81 [diff](https://github.com/apache/arrow-rs/compare/0d91bca5e174821c56fc6529c6bfb6d6a52e8d81..7dae14104f80fdcb42506ebfc94e211f5b6f583d)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
env:
BENCH_FILTER: scan_plan
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5946213654:start:cdb72b700c07d6562ae49741777b5f47 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946300658
run benchmark push_decoder
```yaml
env:
BENCH_FILTER: scan_plan
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946301283
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5946300658)
**Target:** push_decoder
**Sharding factor: 1** (1 workers).
Comparing claude/push-decoder-peek-59 (ec3b0d2c50ee1d475fc773b1bdb71752854669cc) to 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81 [diff](https://github.com/apache/arrow-rs/compare/0d91bca5e174821c56fc6529c6bfb6d6a52e8d81..ec3b0d2c50ee1d475fc773b1bdb71752854669cc)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
env:
BENCH_FILTER: scan_plan
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5946300658:start:9c16c76f0f2448e7d50e45ab227dfd5b -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#discussion_r4210301263
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
+/// decoder.push_ranges(ranges, data).unwrap();
+/// }
+/// DecodeResult::Data(batch) => println!("{} rows", batch.num_rows()),
+/// DecodeResult::Finished => break,
+/// }
+/// }
+/// ```
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+/// [`DecodeResult::NeedsData`]: crate::DecodeResult::NeedsData
+/// [`ParquetPushDecoder::scan_plan`]: super::ParquetPushDecoder::scan_plan
+/// [`ParquetPushDecoder::into_builder`]: super::ParquetPushDecoder::into_builder
+/// [`ParquetPushDecoderBuilder::build`]: super::ParquetPushDecoderBuilder::build
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// `None` for an empty plan, for example of a finished decoder.
+ planner: Option<Box<Planner>>,
+}
+
+impl ScanPlan {
+ /// A plan with no ranges.
+ pub(crate) fn empty() -> Self {
+ Self { planner: None }
+ }
+}
+
+/// The state of a [`ScanPlan`] that is not empty.
+#[derive(Debug, Clone)]
+struct Planner {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the plan was made.
+ frontier: RowGroupFrontier,
+ /// The row group that the decoder was fetching when the plan was made.
+ /// It is planned first.
+ active_row_group: Option<NextRowGroup>,
+ /// The decoder's projection, predicates and batch size.
+ columns: Arc<StageColumns>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// The row group being planned, if any.
+ current: Option<RowGroupRanges>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+/// The columns each decoding stage reads, as [`RowGroupReaderBuilder`]
+/// uses them to decide which bytes a row group needs.
+///
+/// [`RowGroupReaderBuilder`]: super::reader_builder::RowGroupReaderBuilder
+#[derive(Debug)]
+struct StageColumns {
+ /// The output batch size, which aligns cached predicate reads.
+ batch_size: usize,
+ /// Columns in the output.
+ projection: ProjectionMask,
+ /// Columns each [`RowFilter`] predicate reads, in evaluation order.
+ ///
+ /// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+ predicate_projections: Vec<ProjectionMask>,
+ /// Predicate columns whose decoded values are cached for the output, if
+ /// any. Their selection is expanded to batch boundaries when fetched.
+ cache_projection: Option<ProjectionMask>,
+}
+
+/// Builds a [`ScanPlan`] from a decoder's row-group frontier and the columns
+/// each decoding stage reads.
+#[derive(Debug)]
+pub(crate) struct ScanPlanBuilder {
+ frontier: RowGroupFrontier,
+ active_row_group: Option<NextRowGroup>,
+ columns: StageColumns,
+}
+
+impl ScanPlanBuilder {
+ /// Plan the row groups in `frontier`, decoding `projection` in batches of
+ /// `batch_size` rows, with no predicates.
+ pub(crate) fn new(
+ frontier: RowGroupFrontier,
+ batch_size: usize,
+ projection: ProjectionMask,
+ ) -> Self {
+ Self {
+ frontier,
+ active_row_group: None,
+ columns: StageColumns {
+ batch_size,
+ projection,
+ predicate_projections: vec![],
+ cache_projection: None,
+ },
+ }
+ }
+
+ /// Set the columns each predicate reads, in evaluation order.
+ pub(crate) fn with_predicate_projections(
+ mut self,
+ predicate_projections: Vec<ProjectionMask>,
+ ) -> Self {
+ self.columns.predicate_projections = predicate_projections;
+ self
+ }
+
+ /// Set the predicate columns whose decoded values are cached for the
+ /// output.
+ pub(crate) fn with_cache_projection(
+ mut self,
+ cache_projection: Option<ProjectionMask>,
+ ) -> Self {
+ self.columns.cache_projection = cache_projection;
+ self
+ }
+
+ /// Plan `active_row_group` first: the row group that the decoder is
+ /// fetching, which the frontier has already handed over.
+ pub(crate) fn with_active_row_group(mut self, active_row_group: Option<NextRowGroup>) -> Self {
+ self.active_row_group = active_row_group;
+ self
+ }
+
+ pub(crate) fn build(self) -> ScanPlan {
+ let Self {
+ frontier,
+ active_row_group,
+ columns,
+ } = self;
+ let has_limit = frontier.budget.limit().is_some();
+ let planner = Planner {
+ frontier,
+ active_row_group,
+ columns: Arc::new(columns),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ current: None,
+ done: false,
+ };
+ ScanPlan {
+ planner: Some(Box::new(planner)),
+ }
+ }
+}
+
+impl Planner {
+ /// Start planning the next row group the decoder will read.
+ ///
+ /// Returns `Ok(false)` when no row group remains. A row group that the
+ /// offset/limit budget removes is planned with no ranges.
+ fn plan_next_row_group(&mut self) -> Result<bool, ParquetError> {
Review Comment:
Specifically, this logic seems very similar to the code in
https://github.com/apache/arrow-rs/blob/e1450a346b51f2020c480d7c76c91afc32c63ffe/parquet/src/arrow/push_decoder/reader_builder/mod.rs#L396-L724
I think we should plan over time to try and unify the two paths. I'll file a follow on ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6044289322
Thank you @alamb for the careful review. I know this code is hard to verify, your incredible attention to detail is invaluable.
I filed issues for the open follow-ups from this review:
- https://github.com/apache/arrow-rs/issues/11429: the predicate stage projection should have one source, `ColumnFetch` should fetch pages by index, and there are test and benchmark gaps (@viirya)
- https://github.com/apache/arrow-rs/issues/11430: with a `RowFilter`, `LIMIT` does not shorten the plan (@viirya)
- https://github.com/apache/arrow-rs/issues/11431: document the read-ahead buffering contract, and deprecate `is_at_row_group_boundary` / `row_groups_remaining` (@viirya, @alamb)
https://github.com/apache/arrow-rs/issues/11428 already tracks the duplicated logic in the planner and the decoder. https://github.com/apache/arrow-rs/pull/11393 removes the placeholder `StageSchedule`.
I will help to work through these followups 🙏🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940185922
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940184173)
**Target:** arrow_reader
**Sharding factor: 1** (1 workers).
Comparing claude/push-decoder-peek-59 (0d91bca5e174821c56fc6529c6bfb6d6a52e8d81) to 76829d5 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/76829d5ce4562616a69ea8b2696e96160514482f..0d91bca5e174821c56fc6529c6bfb6d6a52e8d81)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5940184173:start:78efdabca641936a40f50086a2f2ee20 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#discussion_r4191552246
##########
parquet/benches/push_decoder.rs:
##########
@@ -166,5 +173,106 @@ fn bench_nbuf(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_1buf, bench_nbuf);
+/// Write a Parquet file with `num_columns` columns and one row group of
+/// 1,000 rows, with `pages_per_column` data pages per column chunk.
+fn make_paged_test_file(num_columns: usize, pages_per_column: usize) -> Bytes {
+ let num_rows = 1_000;
+ let schema = make_wide_schema(num_columns);
+ let columns: Vec<Arc<dyn arrow_array::Array>> = (0..num_columns)
+ .map(|_| {
+ Arc::new(Float32Array::from_iter_values(
+ (0..num_rows).map(|v| v as f32),
+ )) as _
+ })
+ .collect();
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let page_rows = num_rows / pages_per_column;
+ let mut buf = Vec::new();
+ let props = WriterProperties::builder()
+ .set_max_row_group_row_count(Some(num_rows))
+ .set_data_page_row_count_limit(page_rows)
+ // Page limits are checked between write batches.
+ .set_write_batch_size(page_rows)
+ .set_dictionary_enabled(false)
+ .build();
+ let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buf)
+}
+
+type BuilderFn<'a> = Box<dyn Fn() -> ParquetPushDecoderBuilder + 'a>;
+
+/// Plan a wide file with a page index: the first range (what a read-ahead
+/// caller waits for) and the whole scan, which is one row group. For
+/// comparison, `first_reader` builds a decoder, pushes the whole file and
+/// builds the first row group reader, which includes the decoder's own
+/// per-row-group setup.
+///
+/// `all` reads every column and row. `selection` keeps 10 rows of every 100,
+/// so each column chunk reads every other page. `narrow` reads 10 columns.
+fn bench_scan_plan(c: &mut Criterion) {
+ let mut group = c.benchmark_group("push_decoder/scan_plan");
+
+ for (num_cols, pages) in [(100, 10), (1_000, 10), (10_000, 10), (1_000, 100)] {
+ let file_data = make_paged_test_file(num_cols, pages);
+ let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
+ let metadata = ArrowReaderMetadata::load(&file_data, options).unwrap();
+ let selection = RowSelection::from(
+ (0..10)
+ .flat_map(|_| [RowSelector::select(10), RowSelector::skip(90)])
+ .collect::<Vec<_>>(),
+ );
+ let narrow = ProjectionMask::leaves(metadata.parquet_schema(), 0..10);
+ let variants: [(&str, BuilderFn); 3] = [
+ (
+ "all",
+ Box::new(|| ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())),
Review Comment:
Handled in rebase
##########
parquet/benches/push_decoder.rs:
##########
@@ -166,5 +173,106 @@ fn bench_nbuf(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_1buf, bench_nbuf);
+/// Write a Parquet file with `num_columns` columns and one row group of
+/// 1,000 rows, with `pages_per_column` data pages per column chunk.
+fn make_paged_test_file(num_columns: usize, pages_per_column: usize) -> Bytes {
+ let num_rows = 1_000;
+ let schema = make_wide_schema(num_columns);
+ let columns: Vec<Arc<dyn arrow_array::Array>> = (0..num_columns)
+ .map(|_| {
+ Arc::new(Float32Array::from_iter_values(
+ (0..num_rows).map(|v| v as f32),
+ )) as _
+ })
+ .collect();
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let page_rows = num_rows / pages_per_column;
+ let mut buf = Vec::new();
+ let props = WriterProperties::builder()
+ .set_max_row_group_row_count(Some(num_rows))
+ .set_data_page_row_count_limit(page_rows)
+ // Page limits are checked between write batches.
+ .set_write_batch_size(page_rows)
+ .set_dictionary_enabled(false)
+ .build();
+ let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buf)
+}
+
+type BuilderFn<'a> = Box<dyn Fn() -> ParquetPushDecoderBuilder + 'a>;
+
+/// Plan a wide file with a page index: the first range (what a read-ahead
+/// caller waits for) and the whole scan, which is one row group. For
+/// comparison, `first_reader` builds a decoder, pushes the whole file and
+/// builds the first row group reader, which includes the decoder's own
+/// per-row-group setup.
+///
+/// `all` reads every column and row. `selection` keeps 10 rows of every 100,
+/// so each column chunk reads every other page. `narrow` reads 10 columns.
+fn bench_scan_plan(c: &mut Criterion) {
+ let mut group = c.benchmark_group("push_decoder/scan_plan");
+
+ for (num_cols, pages) in [(100, 10), (1_000, 10), (10_000, 10), (1_000, 100)] {
+ let file_data = make_paged_test_file(num_cols, pages);
+ let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
+ let metadata = ArrowReaderMetadata::load(&file_data, options).unwrap();
+ let selection = RowSelection::from(
+ (0..10)
+ .flat_map(|_| [RowSelector::select(10), RowSelector::skip(90)])
+ .collect::<Vec<_>>(),
+ );
+ let narrow = ProjectionMask::leaves(metadata.parquet_schema(), 0..10);
+ let variants: [(&str, BuilderFn); 3] = [
+ (
+ "all",
+ Box::new(|| ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())),
+ ),
+ (
+ "selection",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_row_selection(selection.clone())
+ }),
+ ),
+ (
+ "narrow",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_projection(narrow.clone())
+ }),
+ ),
+ ];
+
+ for (variant, builder) in &variants {
+ let decoder = builder().build().unwrap();
+ let id = format!("{variant}/{num_cols}cols_{pages}pages");
+
+ group.bench_function(BenchmarkId::new("first_range", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().next()))
+ });
+ group.bench_function(BenchmarkId::new("whole_scan", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().count()))
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The ranges are generated on demand, as you iterate. The plan does not
+/// calculate a range until you ask for it.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. If you
+/// rebuild the decoder with [`ParquetPushDecoder::into_builder`], the old
+/// plan is not valid for the new decoder. Call `scan_plan` again on the new
+/// decoder.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the entire column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. Its state does not grow with the number of pages.
+/// * If the decoder will return an error for a row group, the plan ends
+/// before that row group. An example is a row selection with more rows
+/// than the row group.
+///
+/// Keep the fetched ranges in your own cache, and answer each `NeedsData`
+/// with exactly the requested ranges. See [`ParquetPushDecoder::push_ranges`].
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// // `read` takes the data from the cache. If a range is not in
+/// // the cache, it fetches it here.
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
+/// decoder.push_ranges(ranges, data).unwrap();
+/// }
+/// DecodeResult::Data(batch) => println!("{} rows", batch.num_rows()),
+/// DecodeResult::Finished => break,
+/// }
+/// }
+/// ```
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+/// [`DecodeResult::NeedsData`]: crate::DecodeResult::NeedsData
+/// [`ParquetPushDecoder::scan_plan`]: super::ParquetPushDecoder::scan_plan
+/// [`ParquetPushDecoder::into_builder`]: super::ParquetPushDecoder::into_builder
+/// [`ParquetPushDecoder::try_decode`]: super::ParquetPushDecoder::try_decode
+/// [`ParquetPushDecoder::push_ranges`]: super::ParquetPushDecoder::push_ranges
+/// [`ParquetPushDecoderBuilder::build`]: super::ParquetPushDecoderBuilder::build
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// `None` for an empty plan, for example of a finished decoder.
+ planner: Option<Box<Planner>>,
+}
+
+impl ScanPlan {
+ /// A plan with no ranges.
+ pub(crate) fn empty() -> Self {
+ Self { planner: None }
+ }
+}
+
+/// The state of a [`ScanPlan`] that is not empty.
+#[derive(Debug, Clone)]
+struct Planner {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the plan was made.
+ frontier: RowGroupFrontier,
+ /// The row group that the decoder was fetching when the plan was made.
+ /// It is planned first.
+ active_row_group: Option<NextRowGroup>,
+ /// The decoder's projection, predicates and batch size.
+ columns: Arc<StageColumns>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// The row group being planned, if any.
+ current: Option<RowGroupRanges>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+/// The decoder's batch size and decoding stages.
+#[derive(Debug)]
+struct StageColumns {
+ /// The output batch size, which aligns cached predicate reads.
+ batch_size: usize,
+ /// What each decoding stage fetches, shared with the decoder.
+ stages: Arc<StageSchedule>,
+}
+
+/// Builds a [`ScanPlan`] from a decoder's row-group frontier and the columns
+/// each decoding stage reads.
+#[derive(Debug)]
+pub(crate) struct ScanPlanBuilder {
+ frontier: RowGroupFrontier,
+ active_row_group: Option<NextRowGroup>,
+ columns: StageColumns,
+}
+
+impl ScanPlanBuilder {
+ /// Plan the row groups in `frontier`, decoding in batches of
+ /// `batch_size` rows with the decoding stages `stages`.
+ pub(crate) fn new(
+ frontier: RowGroupFrontier,
+ batch_size: usize,
+ stages: Arc<StageSchedule>,
+ ) -> Self {
+ Self {
+ frontier,
+ active_row_group: None,
+ columns: StageColumns { batch_size, stages },
+ }
+ }
+
+ /// Plan `active_row_group` first: the row group that the decoder is
+ /// fetching, which the frontier has already handed over.
+ pub(crate) fn with_active_row_group(mut self, active_row_group: Option<NextRowGroup>) -> Self {
+ self.active_row_group = active_row_group;
+ self
+ }
+
+ pub(crate) fn build(self) -> ScanPlan {
+ let Self {
+ frontier,
+ active_row_group,
+ columns,
+ } = self;
+ let has_limit = frontier.budget.limit().is_some();
+ let planner = Planner {
+ frontier,
+ active_row_group,
+ columns: Arc::new(columns),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ current: None,
+ done: false,
+ };
+ ScanPlan {
+ planner: Some(Box::new(planner)),
+ }
+ }
+}
+
+impl Planner {
+ /// Start planning the next row group the decoder will read.
+ ///
+ /// Returns `Ok(false)` when no row group remains. A row group that the
+ /// offset/limit budget removes is planned with no ranges.
+ fn plan_next_row_group(&mut self) -> Result<bool, ParquetError> {
+ // Use the decoder's own row-group walk, so row groups that the
+ // selection or the budget removes are skipped here too.
+ let next_row_group = match self.active_row_group.take() {
+ Some(active) => Some(active),
+ None => self.frontier.next_readable_row_group()?,
+ };
+ let Some(NextRowGroup {
+ row_group_idx,
+ row_count,
+ selection,
+ budget,
+ }) = next_row_group
+ else {
+ return Ok(false);
+ };
+
+ let filtered = self.frontier.has_predicates;
+ let selection = if filtered {
+ // Predicates see every selected row. Offset and limit apply to
+ // their output, so they cannot be applied before decoding.
+ selection
+ } else {
+ // Apply offset and limit as the decoder does before it requests
+ // the output columns.
+ let plan_builder =
+ ReadPlanBuilder::new(self.columns.batch_size).with_selection(selection);
+ let BudgetedReadPlan {
+ plan_builder,
+ rows_after_budget,
+ remaining_budget,
+ ..
+ } = budget.apply_to_plan(plan_builder, row_count);
+ self.frontier
+ .update_budget_after_row_group(remaining_budget);
+ if rows_after_budget == 0 {
+ self.current = None;
+ return Ok(true);
+ }
+ plan_builder.selection().cloned()
+ };
+
+ let rows = SelectedRows::new(selection.as_ref(), row_count);
+ let first_row = self.next_row;
+ self.next_row += rows.selected_before(row_count);
+
+ let metadata = &self.frontier.parquet_metadata;
+ let page_index = metadata
+ .page_index()
+ .filter(|page_index| page_index.has_offset_indexes())
+ .cloned();
+ let row_group = metadata.row_group(row_group_idx);
+
+ // Predicate columns that are cached for the output are fetched with the
+ // selection expanded to batch boundaries. See `fetch_ranges`.
+ let stages = &self.columns.stages;
+ let expanded_selection = match (&selection, stages.cache_projection()) {
+ (Some(selection), Some(_)) => {
+ Some(selection.expand_to_batch_boundaries(self.columns.batch_size, row_count))
+ }
+ _ => None,
+ };
+
+ let num_columns = row_group.columns().len();
+ let mut planned_columns = vec![false; num_columns];
+ let mut stage_plans = vec![];
+ for (stage, fetch) in stages.stages() {
+ let conditional = filtered
+ && (stage != ScanStage::Predicate(0)
+ || (self.has_limit && !self.at_first_row_group));
+ // The decoder reuses a column that an earlier stage read.
+ // `columns_to_fetch` filters, so it gives no size hint. Reserve
+ // for every column to avoid growing the vector one column at a time.
+ let mut columns = Vec::with_capacity(num_columns);
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/10555/commits/80202f2adf46b188b12d0333efdd82c4ea3a0755
##########
parquet/benches/push_decoder.rs:
##########
@@ -166,5 +173,106 @@ fn bench_nbuf(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_1buf, bench_nbuf);
+/// Write a Parquet file with `num_columns` columns and one row group of
+/// 1,000 rows, with `pages_per_column` data pages per column chunk.
+fn make_paged_test_file(num_columns: usize, pages_per_column: usize) -> Bytes {
+ let num_rows = 1_000;
+ let schema = make_wide_schema(num_columns);
+ let columns: Vec<Arc<dyn arrow_array::Array>> = (0..num_columns)
+ .map(|_| {
+ Arc::new(Float32Array::from_iter_values(
+ (0..num_rows).map(|v| v as f32),
+ )) as _
+ })
+ .collect();
+ let batch = RecordBatch::try_new(schema.clone(), columns).unwrap();
+
+ let page_rows = num_rows / pages_per_column;
+ let mut buf = Vec::new();
+ let props = WriterProperties::builder()
+ .set_max_row_group_row_count(Some(num_rows))
+ .set_data_page_row_count_limit(page_rows)
+ // Page limits are checked between write batches.
+ .set_write_batch_size(page_rows)
+ .set_dictionary_enabled(false)
+ .build();
+ let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buf)
+}
+
+type BuilderFn<'a> = Box<dyn Fn() -> ParquetPushDecoderBuilder + 'a>;
+
+/// Plan a wide file with a page index: the first range (what a read-ahead
+/// caller waits for) and the whole scan, which is one row group. For
+/// comparison, `first_reader` builds a decoder, pushes the whole file and
+/// builds the first row group reader, which includes the decoder's own
+/// per-row-group setup.
+///
+/// `all` reads every column and row. `selection` keeps 10 rows of every 100,
+/// so each column chunk reads every other page. `narrow` reads 10 columns.
+fn bench_scan_plan(c: &mut Criterion) {
+ let mut group = c.benchmark_group("push_decoder/scan_plan");
+
+ for (num_cols, pages) in [(100, 10), (1_000, 10), (10_000, 10), (1_000, 100)] {
+ let file_data = make_paged_test_file(num_cols, pages);
+ let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required);
+ let metadata = ArrowReaderMetadata::load(&file_data, options).unwrap();
+ let selection = RowSelection::from(
+ (0..10)
+ .flat_map(|_| [RowSelector::select(10), RowSelector::skip(90)])
+ .collect::<Vec<_>>(),
+ );
+ let narrow = ProjectionMask::leaves(metadata.parquet_schema(), 0..10);
+ let variants: [(&str, BuilderFn); 3] = [
+ (
+ "all",
+ Box::new(|| ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())),
+ ),
+ (
+ "selection",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_row_selection(selection.clone())
+ }),
+ ),
+ (
+ "narrow",
+ Box::new(|| {
+ ParquetPushDecoderBuilder::new_with_metadata(metadata.clone())
+ .with_projection(narrow.clone())
+ }),
+ ),
+ ];
+
+ for (variant, builder) in &variants {
+ let decoder = builder().build().unwrap();
+ let id = format!("{variant}/{num_cols}cols_{pages}pages");
+
+ group.bench_function(BenchmarkId::new("first_range", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().next()))
+ });
+ group.bench_function(BenchmarkId::new("whole_scan", &id), |b| {
+ b.iter(|| black_box(decoder.scan_plan().count()))
+ });
+ group.bench_function(BenchmarkId::new("first_reader", &id), |b| {
+ b.iter(|| {
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
+/// reports the error.
+///
+/// Keep the fetched ranges in your own cache, and push exactly the ranges
+/// that `NeedsData` requests. The decoder does not use a requested range
+/// that is split over two pushed buffers, and it releases a pushed buffer
+/// only if its range is equal to a requested range.
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/frontier.rs:
##########
@@ -0,0 +1,494 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Which row groups a scan reads next, with their selections and budget.
+
+use crate::arrow::arrow_reader::{RowGroupPlan, RowGroupSelection, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::ParquetMetaData;
+use std::collections::VecDeque;
+use std::sync::Arc;
+
+use super::budget::RowBudget;
+
+/// Plan for the next queued row group after row-selection slicing.
+#[derive(Debug)]
+enum QueuedRowGroupDecision {
+ /// Hand this row group to the builder.
+ Read(NextRowGroup),
+ /// Skip this row group, and keep scanning with the updated budget.
+ Skip { remaining_budget: RowBudget },
+}
+
+/// Work item handed from [`RowGroupFrontier`] to [`RowGroupReaderBuilder`](crate::arrow::push_decoder::reader_builder::RowGroupReaderBuilder).
+#[derive(Debug, Clone)]
+pub(crate) struct NextRowGroup {
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. After
+/// [`ParquetPushDecoder::into_builder`], plan again.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. It is lazy, and its state does not grow with the number
+/// of pages.
+/// * If a row group is not valid, the plan ends before it. The decoder
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/10555/commits/ab833917627563fc46da2a17b6c8777088b2ded6
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub(crate) enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// [`DecodeResult::NeedsData`] asks for it. `NeedsData` stays the exact
+/// request.
+///
+/// * The plan contains every range that the decoder can request after
Review Comment:
Handled in rebase
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The ranges are generated on demand, as you iterate. The plan does not
+/// calculate a range until you ask for it.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. If you
+/// rebuild the decoder with [`ParquetPushDecoder::into_builder`], the old
+/// plan is not valid for the new decoder. Call `scan_plan` again on the new
+/// decoder.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the entire column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. Its state does not grow with the number of pages.
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/10555/commits/ab833917627563fc46da2a17b6c8777088b2ded6
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/10555/commits/7904ea8b826400b447f062a9d2a212eac6c20f45
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#discussion_r4126858940
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +617,124 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Returns the byte ranges this scan may read, in the order decoding needs
+ /// them.
+ ///
+ /// Use this to schedule I/O ahead of [`DecodeResult::NeedsData`]: for
+ /// example, to fetch the next few megabytes in the background while the
+ /// current row group decodes, or to release fetched bytes once decoding
+ /// has passed them. [`DecodeResult::NeedsData`] stays the exact and
+ /// authoritative request. This plan is an upper bound on it, for
+ /// speculation.
+ ///
+ /// # Contract
+ ///
+ /// * **Buffer independent.** The plan does not depend on the data pushed
+ /// to the decoder. It lists ranges the decoder already has, so the caller
+ /// decides how to satisfy each range: from decoder buffers, from its own
+ /// cache, or from storage.
+ /// * **Stable.** The plan covers the whole scan as configured when the
+ /// decoder was built, including row groups already decoded. It does not
+ /// change as decoding advances. [`Self::into_builder`] and
+ /// [`ParquetPushDecoderBuilder::build`] give a new decoder, whose plan
+ /// covers only the row groups that remain.
+ /// * **Same planning as decoding.** The plan uses the decoder's own
+ /// row-group order, row selections, offset/limit and projection. Per row
Review Comment:
Agreed, removed now
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -169,17 +173,17 @@ impl RowBudget {
}
#[derive(Debug)]
-struct BudgetedReadPlan {
+pub(crate) struct BudgetedReadPlan {
Review Comment:
Done as a pure move to `push_decoder/scan_plan/{budget,frontier}.rs`. I can open it as a separate PR if you want.
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +617,124 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Returns the byte ranges this scan may read, in the order decoding needs
+ /// them.
+ ///
+ /// Use this to schedule I/O ahead of [`DecodeResult::NeedsData`]: for
+ /// example, to fetch the next few megabytes in the background while the
+ /// current row group decodes, or to release fetched bytes once decoding
+ /// has passed them. [`DecodeResult::NeedsData`] stays the exact and
+ /// authoritative request. This plan is an upper bound on it, for
+ /// speculation.
+ ///
+ /// # Contract
+ ///
+ /// * **Buffer independent.** The plan does not depend on the data pushed
+ /// to the decoder. It lists ranges the decoder already has, so the caller
+ /// decides how to satisfy each range: from decoder buffers, from its own
+ /// cache, or from storage.
+ /// * **Stable.** The plan covers the whole scan as configured when the
+ /// decoder was built, including row groups already decoded. It does not
+ /// change as decoding advances. [`Self::into_builder`] and
+ /// [`ParquetPushDecoderBuilder::build`] give a new decoder, whose plan
+ /// covers only the row groups that remain.
+ /// * **Same planning as decoding.** The plan uses the decoder's own
+ /// row-group order, row selections, offset/limit and projection. Per row
+ /// group, the planned bytes of a scan without a [`RowFilter`] are the
+ /// bytes that [`DecodeResult::NeedsData`] requests. With a
+ /// [`RowFilter`], they are a superset; see [`PlannedRange::conditional`].
+ /// * **Lazy.** The iterator plans one row group at a time when the caller
Review Comment:
Done. I moved it to the `ScanPlan` docs.
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -605,6 +617,124 @@ impl ParquetPushDecoder {
self.state.peek_next_row_group()
}
+ /// Returns the byte ranges this scan may read, in the order decoding needs
+ /// them.
+ ///
+ /// Use this to schedule I/O ahead of [`DecodeResult::NeedsData`]: for
+ /// example, to fetch the next few megabytes in the background while the
+ /// current row group decodes, or to release fetched bytes once decoding
+ /// has passed them. [`DecodeResult::NeedsData`] stays the exact and
+ /// authoritative request. This plan is an upper bound on it, for
+ /// speculation.
+ ///
+ /// # Contract
+ ///
+ /// * **Buffer independent.** The plan does not depend on the data pushed
+ /// to the decoder. It lists ranges the decoder already has, so the caller
+ /// decides how to satisfy each range: from decoder buffers, from its own
+ /// cache, or from storage.
+ /// * **Stable.** The plan covers the whole scan as configured when the
+ /// decoder was built, including row groups already decoded. It does not
Review Comment:
Good catch, I updated so that the plan now changes as the decoder advances. `scan_plan()` now plans from the current state of the decoder. Before (when you reviewed last), it gave a snapshot of the full scan. A snapshot cannot show progress, for example with batch decoding (https://github.com/apache/arrow-rs/pull/11223). The plan is now a superset of the ranges that the decoder can still request. To get the full scan, callers can hit `scan_plan()` one time after `build()`.
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -199,6 +203,21 @@ pub(crate) enum RowGroupBuildResult {
},
}
+/// What [`RowGroupReaderBuilder`] uses to decide which bytes a row group
+/// needs, captured so a scan can be planned without decoding it.
+#[derive(Debug, Clone)]
+pub(crate) struct ScanPlanConfig {
Review Comment:
Done. `ScanPlanBuilder` is now in the same file as `ScanPlan`.
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,890 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges a push decoder scan may read, in the order
+//! decoding needs them.
+//!
+//! See [`ParquetPushDecoder::scan_plan`](super::ParquetPushDecoder::scan_plan).
+
+use std::cmp::Reverse;
+use std::ops::Range;
+use std::sync::Arc;
+
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::RowGroupPageIndex;
+
+use super::reader_builder::{BudgetedReadPlan, ScanPlanConfig};
+use super::remaining::{NextRowGroup, RowGroupFrontier};
+
+/// A byte range that a scan may read, with the rows and stage it serves.
+///
+/// Returned by [`ParquetPushDecoder::scan_plan`](super::ParquetPushDecoder::scan_plan).
+///
+/// # Rows
+///
+/// `first_row..last_row` are positions in the rows the decoder *plans* to
+/// read: row selections and, for scans without a [`RowFilter`], offset and
+/// limit are applied. Row 0 is the first planned row of the first planned row
+/// group, and positions continue across row groups.
+///
+/// For a scan without a [`RowFilter`], planned row `n` is output row `n`. A
+/// caller that has received `n` rows from the decoder can therefore release
+/// every range whose `last_row <= n`.
+///
+/// With a [`RowFilter`], positions count the rows the first predicate sees.
+/// Offset and limit apply after the predicates, so they are not applied here.
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// First planned row this range serves.
+ pub first_row: u64,
+ /// One past the last planned row this range serves.
+ ///
+ /// This can equal `first_row` for a page that the decoder reads only to
+ /// complete a batch for its predicate cache.
+ pub last_row: u64,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// The decoding stage that first reads this range.
+ pub stage: ScanStage,
+ /// `true` if the decoder may not read this range, depending on predicate
+ /// results.
+ ///
+ /// Always `false` for a scan without a [`RowFilter`]. With a [`RowFilter`],
+ /// ranges for later predicates and for the output columns are
+ /// conditional, because earlier predicates can remove every row they
+ /// serve. Ranges for the first predicate are conditional only if a limit is
+ /// set and the range is not in the first planned row group, because the
+ /// limit can be reached first.
+ ///
+ /// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+ pub conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk. It serves every planned row in
+ /// the column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum ScanStage {
+ /// Evaluation of the predicate at this index in the
+ /// [`RowFilter`](crate::arrow::arrow_reader::RowFilter).
+ Predicate(usize),
+ /// Decoding of the output columns.
+ Projection,
+}
+
+/// The byte ranges that a push decoder scan may read, in the order decoding
+/// needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`](super::ParquetPushDecoder::scan_plan),
+/// which documents the contract.
+///
+/// The iterator plans one row group at a time, when the caller asks for its
+/// first range. It owns its state, so the caller can keep it while it pushes
+/// data to the decoder.
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the decoder was built.
+ frontier: RowGroupFrontier,
+ /// The decoder's projection, predicates and batch size.
+ config: Arc<ScanPlanConfig>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// Planned ranges of the current row group not yet returned.
+ pending: std::vec::IntoIter<PlannedRange>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+impl ScanPlan {
+ pub(super) fn new(frontier: RowGroupFrontier, config: ScanPlanConfig) -> Self {
+ let has_limit = frontier.budget.limit().is_some();
+ Self {
+ frontier,
+ config: Arc::new(config),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ pending: Vec::new().into_iter(),
+ done: false,
+ }
+ }
+
+ /// Plan the next row group the decoder will read.
+ ///
+ /// Returns `Ok(None)` when no row group remains. A row group that the
+ /// offset/limit budget removes gives an empty `Vec`.
+ fn plan_next_row_group(&mut self) -> Result<Option<Vec<PlannedRange>>, ParquetError> {
+ // Use the decoder's own row-group walk, so row groups that the
+ // selection or the budget removes are skipped here too.
+ let Some(NextRowGroup {
+ row_group_idx,
+ row_count,
+ selection,
+ budget,
+ }) = self.frontier.next_readable_row_group()?
+ else {
+ return Ok(None);
+ };
+
+ let filtered = self.frontier.has_predicates;
+ let selection = if filtered {
+ // Predicates see every selected row. Offset and limit apply to
+ // their output, so they cannot be applied before decoding.
+ selection
+ } else {
+ // Apply offset and limit as the decoder does before it requests
+ // the output columns.
+ let plan_builder =
+ ReadPlanBuilder::new(self.config.batch_size).with_selection(selection);
+ let BudgetedReadPlan {
+ plan_builder,
+ rows_after_budget,
+ remaining_budget,
+ ..
+ } = budget.apply_to_plan(plan_builder, row_count);
+ self.frontier
+ .update_budget_after_row_group(remaining_budget);
+ if rows_after_budget == 0 {
+ return Ok(Some(vec![]));
+ }
+ plan_builder.selection().cloned()
+ };
+
+ let rows = SelectedRows::new(selection.as_ref(), row_count);
+ let first_row = self.next_row;
+ self.next_row += rows.selected_before(row_count);
+
+ let metadata = Arc::clone(&self.frontier.parquet_metadata);
+ let page_index = metadata
+ .page_index()
+ .is_some_and(|page_index| page_index.has_offset_indexes())
+ .then(|| metadata.page_index_for_row_group(row_group_idx));
+ let row_group = metadata.row_group(row_group_idx);
+
+ // Predicate columns that are cached for the output are fetched with the
+ // selection expanded to batch boundaries. See `fetch_ranges`.
+ let expanded_selection = match (&selection, &self.config.cache_projection) {
+ (Some(selection), Some(_)) if filtered => {
+ Some(selection.expand_to_batch_boundaries(self.config.batch_size, row_count))
+ }
+ _ => None,
+ };
+
+ let columns = RowGroupColumns {
+ row_group_idx,
+ row_count,
+ first_row,
+ rows: &rows,
+ page_index: page_index.as_ref(),
+ };
+ let stages = self
+ .config
+ .predicate_projections
+ .iter()
+ .enumerate()
+ .map(|(idx, mask)| (ScanStage::Predicate(idx), mask))
+ .chain(std::iter::once((
+ ScanStage::Projection,
+ &self.config.projection,
+ )));
+
+ let mut planned_columns = vec![false; row_group.columns().len()];
+ let mut ranges = vec![];
+ for (stage, mask) in stages {
+ let conditional = filtered
+ && (stage != ScanStage::Predicate(0)
+ || (self.has_limit && !self.at_first_row_group));
+ let stage_start = ranges.len();
+ for (column_idx, chunk) in row_group.columns().iter().enumerate() {
+ // The decoder reuses a column that an earlier stage read.
+ if !mask.leaf_included(column_idx) || planned_columns[column_idx] {
+ continue;
+ }
+ planned_columns[column_idx] = true;
+ let is_cached = self
+ .config
+ .cache_projection
+ .as_ref()
+ .is_some_and(|cache| cache.leaf_included(column_idx));
+ let fetch_selection = match (stage, &expanded_selection) {
+ (ScanStage::Predicate(_), Some(expanded)) if is_cached => Some(expanded),
+ _ => selection.as_ref(),
+ };
+ let (chunk_start, chunk_len) = chunk.byte_range();
+ columns.plan(
+ column_idx,
+ chunk_start..chunk_start + chunk_len,
+ fetch_selection,
+ stage,
+ conditional,
+ &mut ranges,
+ );
+ }
+ // Order by when decoding needs each range. For equal first rows,
+ // the wider span (a dictionary page) comes first, so any prefix of
+ // the plan can be decoded.
+ ranges[stage_start..]
+ .sort_by_key(|p| (p.first_row, Reverse(p.last_row), p.range.start));
+ }
+ self.at_first_row_group = false;
+ Ok(Some(ranges))
+ }
+}
+
+impl Iterator for ScanPlan {
+ type Item = PlannedRange;
+
+ fn next(&mut self) -> Option<PlannedRange> {
Review Comment:
Yes, it's page-level by design. With batch decoding (https://github.com/apache/arrow-rs/pull/11223), the decoder requests single pages. A caller can group entries by `row_group` and `column`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940184173
run benchmark arrow_reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5940184708
run benchmark arrow_reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5944635613
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-5944634999)
**Target:** push_decoder
**Sharding factor: 1** (1 workers).
Comparing claude/push-decoder-peek-59 (a1a2c7e383cdf90c89b1b85ef57e0822445fa2c0) to 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81 [diff](https://github.com/apache/arrow-rs/compare/0d91bca5e174821c56fc6529c6bfb6d6a52e8d81..a1a2c7e383cdf90c89b1b85ef57e0822445fa2c0)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 0d91bca5e174821c56fc6529c6bfb6d6a52e8d81
env:
BENCH_FILTER: scan_plan
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5944634999:start:13f07bd6c93754033f16035f4e6c7162 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#discussion_r4163563942
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -317,7 +232,25 @@ impl RowGroupReaderBuilder {
row_selection_policy,
state: Some(RowGroupDecoderState::Finished),
buffers,
- }
+ stages: Arc::new(StageSchedule::new(ProjectionMask::all(), vec![], None)),
Review Comment:
nit: building a placeholder `StageSchedule` and replacing it a few lines later reads a bit oddly. Computing `predicate_projections` and `cache_projection` before constructing `Self` would avoid it. `compute_cache_projection_inner` needs only `max_predicate_cache_size`, `projection` and the schema, so it could become a free function or take those as arguments.
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -551,13 +486,13 @@ impl RowGroupReaderBuilder {
row_count,
self.batch_size,
&self.metadata,
- predicate.projection(), // use the predicate's projection
+ fetch.projection, // use the predicate's projection
Review Comment:
This fetches with the `StageSchedule` snapshot taken in `new`, but a few lines below, `try_into_in_memory_row_group` (L545) and `build_array_reader` (L555) still use the live `predicate.projection()`. `fill_column_chunks` requires the fetch and fill projections to be the same mask.
They match today, but `ArrowPredicate::evaluate` takes `&mut self`, and nothing in the trait says `projection()` must stay constant. Before this change, both sides read it live. Could the decode side use `self.stages.fetch(..).projection` too, so there is one source?
The same applies to the cache: `Start` still recomputes it with `compute_cache_projection` (L437), while the fetch here uses `stages.cache_projection`.
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
Review Comment:
The derived `PartialEq` also compares `first_row`, `last_row`, `stage` and `conditional`, which callers can't see. For example, the same page of row group 1 planned right after `build()` and planned again at the row-group boundary has identical public fields, but compares unequal (`first_row` is 200 in one and 0 in the other). Could equality cover only the public fields, or could the derive be dropped? `plan_at_boundary_is_rebuilt_decoder_plan` could compare the public fields explicitly.
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
Review Comment:
"Contains every range" reads as if each request appears as an item of the plan. Without a selection, the decoder requests whole column chunks while the plan lists the dictionary and each page, so the guarantee (and what the tests check with `covers`) is that every requested byte is in the plan. Maybe: "The plan covers every byte that the decoder can request after the plan is made."
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The ranges are generated on demand, as you iterate. The plan does not
+/// calculate a range until you ask for it.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. If you
+/// rebuild the decoder with [`ParquetPushDecoder::into_builder`], the old
+/// plan is not valid for the new decoder. Call `scan_plan` again on the new
+/// decoder.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the entire column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. Its state does not grow with the number of pages.
+/// * If the decoder will return an error for a row group, the plan ends
+/// before that row group. An example is a row selection with more rows
+/// than the row group.
+///
+/// Keep the fetched ranges in your own cache, and answer each `NeedsData`
+/// with exactly the requested ranges. See [`ParquetPushDecoder::push_ranges`].
+///
+/// # Example
+///
+/// ```
+/// # use std::collections::BTreeMap;
+/// # use std::ops::Range;
+/// # use bytes::Bytes;
+/// # use arrow_array::record_batch;
+/// # use parquet::DecodeResult;
+/// # use parquet::arrow::ArrowWriter;
+/// # use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};
+/// # use parquet::arrow::push_decoder::ParquetPushDecoderBuilder;
+/// # use parquet::file::metadata::PageIndexPolicy;
+/// # use parquet::file::properties::WriterProperties;
+/// # let file = {
+/// # let mut buffer = vec![];
+/// # let batch = record_batch!(("a", Int32, [1, 2, 3, 4])).unwrap();
+/// # let props = WriterProperties::builder().set_max_row_group_row_count(Some(2)).build();
+/// # let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
+/// # writer.write(&batch).unwrap();
+/// # writer.close().unwrap();
+/// # Bytes::from(buffer)
+/// # };
+/// # let fetch = |range: &Range<u64>| file.slice(range.start as usize..range.end as usize);
+/// # // Join cached ranges that cover `range`, or fetch it.
+/// # let read = |cache: &BTreeMap<u64, Bytes>, range: &Range<u64>| {
+/// # let mut data = Vec::new();
+/// # let mut position = range.start;
+/// # while position < range.end {
+/// # let Some((start, bytes)) = cache.range(..=position).next_back() else {
+/// # return fetch(range);
+/// # };
+/// # let end = (start + bytes.len() as u64).min(range.end);
+/// # if end <= position {
+/// # return fetch(range);
+/// # }
+/// # data.extend_from_slice(&bytes[(position - start) as usize..(end - start) as usize]);
+/// # position = end;
+/// # }
+/// # Bytes::from(data)
+/// # };
+/// # let options = ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Optional);
+/// # let metadata = ArrowReaderMetadata::load(&file, options).unwrap();
+/// let mut decoder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
+/// .build()
+/// .unwrap();
+///
+/// // Read ahead up to 1 MB into a cache
+/// let mut cache = BTreeMap::new();
+/// let mut cached = 0;
+/// for planned in decoder.scan_plan() {
+/// if cached + planned.len() > 1024 * 1024 {
+/// break;
+/// }
+/// cached += planned.len();
+/// cache.insert(planned.range.start, fetch(&planned.range));
+/// }
+///
+/// // Answer each request from the cache
+/// loop {
+/// match decoder.try_decode().unwrap() {
+/// DecodeResult::NeedsData(ranges) => {
+/// // `read` takes the data from the cache. If a range is not in
+/// // the cache, it fetches it here.
+/// let data = ranges.iter().map(|range| read(&cache, range)).collect();
+/// decoder.push_ranges(ranges, data).unwrap();
+/// }
+/// DecodeResult::Data(batch) => println!("{} rows", batch.num_rows()),
+/// DecodeResult::Finished => break,
+/// }
+/// }
+/// ```
+///
+/// [`RowFilter`]: crate::arrow::arrow_reader::RowFilter
+/// [`DecodeResult::NeedsData`]: crate::DecodeResult::NeedsData
+/// [`ParquetPushDecoder::scan_plan`]: super::ParquetPushDecoder::scan_plan
+/// [`ParquetPushDecoder::into_builder`]: super::ParquetPushDecoder::into_builder
+/// [`ParquetPushDecoder::try_decode`]: super::ParquetPushDecoder::try_decode
+/// [`ParquetPushDecoder::push_ranges`]: super::ParquetPushDecoder::push_ranges
+/// [`ParquetPushDecoderBuilder::build`]: super::ParquetPushDecoderBuilder::build
+#[derive(Debug, Clone)]
+pub struct ScanPlan {
+ /// `None` for an empty plan, for example of a finished decoder.
+ planner: Option<Box<Planner>>,
+}
+
+impl ScanPlan {
+ /// A plan with no ranges.
+ pub(crate) fn empty() -> Self {
+ Self { planner: None }
+ }
+}
+
+/// The state of a [`ScanPlan`] that is not empty.
+#[derive(Debug, Clone)]
+struct Planner {
+ /// The decoder's row-group queue, selections and offset/limit budget, as
+ /// they were when the plan was made.
+ frontier: RowGroupFrontier,
+ /// The row group that the decoder was fetching when the plan was made.
+ /// It is planned first.
+ active_row_group: Option<NextRowGroup>,
+ /// The decoder's projection, predicates and batch size.
+ columns: Arc<StageColumns>,
+ /// Whether an output limit is set.
+ has_limit: bool,
+ /// Whether no row group has been planned yet.
+ at_first_row_group: bool,
+ /// Planned rows before the next row group.
+ next_row: u64,
+ /// The row group being planned, if any.
+ current: Option<RowGroupRanges>,
+ /// Whether planning has ended.
+ done: bool,
+}
+
+/// The decoder's batch size and decoding stages.
+#[derive(Debug)]
+struct StageColumns {
+ /// The output batch size, which aligns cached predicate reads.
+ batch_size: usize,
+ /// What each decoding stage fetches, shared with the decoder.
+ stages: Arc<StageSchedule>,
+}
+
+/// Builds a [`ScanPlan`] from a decoder's row-group frontier and the columns
+/// each decoding stage reads.
+#[derive(Debug)]
+pub(crate) struct ScanPlanBuilder {
+ frontier: RowGroupFrontier,
+ active_row_group: Option<NextRowGroup>,
+ columns: StageColumns,
+}
+
+impl ScanPlanBuilder {
+ /// Plan the row groups in `frontier`, decoding in batches of
+ /// `batch_size` rows with the decoding stages `stages`.
+ pub(crate) fn new(
+ frontier: RowGroupFrontier,
+ batch_size: usize,
+ stages: Arc<StageSchedule>,
+ ) -> Self {
+ Self {
+ frontier,
+ active_row_group: None,
+ columns: StageColumns { batch_size, stages },
+ }
+ }
+
+ /// Plan `active_row_group` first: the row group that the decoder is
+ /// fetching, which the frontier has already handed over.
+ pub(crate) fn with_active_row_group(mut self, active_row_group: Option<NextRowGroup>) -> Self {
+ self.active_row_group = active_row_group;
+ self
+ }
+
+ pub(crate) fn build(self) -> ScanPlan {
+ let Self {
+ frontier,
+ active_row_group,
+ columns,
+ } = self;
+ let has_limit = frontier.budget.limit().is_some();
+ let planner = Planner {
+ frontier,
+ active_row_group,
+ columns: Arc::new(columns),
+ has_limit,
+ at_first_row_group: true,
+ next_row: 0,
+ current: None,
+ done: false,
+ };
+ ScanPlan {
+ planner: Some(Box::new(planner)),
+ }
+ }
+}
+
+impl Planner {
+ /// Start planning the next row group the decoder will read.
+ ///
+ /// Returns `Ok(false)` when no row group remains. A row group that the
+ /// offset/limit budget removes is planned with no ranges.
+ fn plan_next_row_group(&mut self) -> Result<bool, ParquetError> {
+ // Use the decoder's own row-group walk, so row groups that the
+ // selection or the budget removes are skipped here too.
+ let next_row_group = match self.active_row_group.take() {
+ Some(active) => Some(active),
+ None => self.frontier.next_readable_row_group()?,
+ };
+ let Some(NextRowGroup {
+ row_group_idx,
+ row_count,
+ selection,
+ budget,
+ }) = next_row_group
+ else {
+ return Ok(false);
+ };
+
+ let filtered = self.frontier.has_predicates;
+ let selection = if filtered {
+ // Predicates see every selected row. Offset and limit apply to
+ // their output, so they cannot be applied before decoding.
+ selection
+ } else {
+ // Apply offset and limit as the decoder does before it requests
+ // the output columns.
+ let plan_builder =
+ ReadPlanBuilder::new(self.columns.batch_size).with_selection(selection);
+ let BudgetedReadPlan {
+ plan_builder,
+ rows_after_budget,
+ remaining_budget,
+ ..
+ } = budget.apply_to_plan(plan_builder, row_count);
+ self.frontier
+ .update_budget_after_row_group(remaining_budget);
+ if rows_after_budget == 0 {
+ self.current = None;
+ return Ok(true);
+ }
+ plan_builder.selection().cloned()
+ };
+
+ let rows = SelectedRows::new(selection.as_ref(), row_count);
+ let first_row = self.next_row;
+ self.next_row += rows.selected_before(row_count);
+
+ let metadata = &self.frontier.parquet_metadata;
+ let page_index = metadata
+ .page_index()
+ .filter(|page_index| page_index.has_offset_indexes())
+ .cloned();
+ let row_group = metadata.row_group(row_group_idx);
+
+ // Predicate columns that are cached for the output are fetched with the
+ // selection expanded to batch boundaries. See `fetch_ranges`.
+ let stages = &self.columns.stages;
+ let expanded_selection = match (&selection, stages.cache_projection()) {
+ (Some(selection), Some(_)) => {
+ Some(selection.expand_to_batch_boundaries(self.columns.batch_size, row_count))
+ }
+ _ => None,
+ };
+
+ let num_columns = row_group.columns().len();
+ let mut planned_columns = vec![false; num_columns];
+ let mut stage_plans = vec![];
+ for (stage, fetch) in stages.stages() {
+ let conditional = filtered
+ && (stage != ScanStage::Predicate(0)
+ || (self.has_limit && !self.at_first_row_group));
+ // The decoder reuses a column that an earlier stage read.
+ // `columns_to_fetch` filters, so it gives no size hint. Reserve
+ // for every column to avoid growing the vector one column at a time.
+ let mut columns = Vec::with_capacity(num_columns);
Review Comment:
This reserves for every leaf column in the file, for every stage, and `StagePlan` keeps the vector for the whole row group. Reading 10 columns of a 10,000-column file reserves about 240 KB per stage. Reserving `fetch.projection`'s leaf count (or `shrink_to_fit` after `extend`) would keep the speedup without over-allocating.
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
Review Comment:
Following up on the earlier question about these `pub(crate)` fields: `stage` and `conditional` are only read by tests. Ordering uses only `first_row` / `last_row`. They also pull `has_limit`, `at_first_row_group`, and the `conditional` computation at L381 into production code. I'd either drop them, or make `conditional` public. A public `conditional` would let a caller with a `RowFilter` prefetch predicate columns eagerly and output columns lazily.
That matters because with a filter, offset and limit don't shorten the plan (L332), so a `LIMIT 10` + filter scan lists every remaining row group, and a caller with a large read-ahead budget will fetch far more than it needs.
##########
parquet/src/arrow/in_memory_row_group.rs:
##########
@@ -205,6 +175,127 @@ impl InMemoryRowGroup<'_> {
}
}
+/// The leaf columns in `projection` that are not yet read, in column order.
+///
+/// A decoding stage of a row group does not fetch a column that an earlier
+/// stage of the same row group read.
+#[inline]
+pub(crate) fn columns_to_fetch<'a>(
+ projection: &'a ProjectionMask,
+ num_columns: usize,
+ is_read: impl Fn(usize) -> bool + 'a,
+) -> impl Iterator<Item = usize> + 'a {
+ (0..num_columns).filter(move |&idx| projection.leaf_included(idx) && !is_read(idx))
+}
+
+/// The selection that [`InMemoryRowGroup::fetch_ranges`] uses to choose the
+/// pages of column `idx`: `expanded_selection` (the selection expanded to
+/// batch boundaries) if `cache_mask` includes the column, else `selection`.
+#[inline]
+pub(crate) fn column_selection<'a>(
+ selection: Option<&'a RowSelection>,
+ expanded_selection: Option<&'a RowSelection>,
+ cache_mask: Option<&ProjectionMask>,
+ idx: usize,
+) -> Option<&'a RowSelection> {
+ match expanded_selection {
+ Some(expanded) if cache_mask.is_some_and(|mask| mask.leaf_included(idx)) => Some(expanded),
+ _ => selection,
+ }
+}
+
+/// What [`InMemoryRowGroup::fetch_ranges`] fetches for one column chunk.
+///
+/// This is the one place that decides which bytes of a column chunk are
+/// requested, so other users of this rule cannot differ from the decoder.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum ColumnFetch<'a> {
+ /// The whole column chunk, as one range. This is the case if there is
+ /// no selection or the column has no offset index.
+ Chunk {
+ /// Byte range of the column chunk.
+ range: Range<u64>,
+ },
+ /// The dictionary page, if any, and the data pages that contain a
+ /// selected row.
+ Pages {
+ /// Byte range of the dictionary page, if the column chunk has one.
+ dictionary: Option<Range<u64>>,
+ /// The page locations of the column chunk.
+ locations: &'a [PageLocation],
+ /// Indexes in `locations` of the data pages to fetch, in page order.
+ pages: Vec<usize>,
+ },
+}
+
+impl<'a> ColumnFetch<'a> {
+ /// The fetch of the column chunk at byte range `chunk`, with page
+ /// locations `locations` from its offset index, if any, and row
+ /// selection `selection`, if any.
+ #[inline]
+ pub(crate) fn new(
+ chunk: Range<u64>,
+ locations: Option<&'a [PageLocation]>,
+ selection: Option<&RowSelection>,
+ ) -> Self {
+ let (Some(selection), Some(locations)) = (selection, locations) else {
+ return Self::Chunk { range: chunk };
+ };
+ // `scan_ranges` returns the ranges of the selected pages in page
+ // order. Map them back to page indexes.
+ let fetched = selection.scan_ranges(locations);
Review Comment:
`scan_ranges` builds byte ranges from page locations, and this maps them back to page indexes by matching offsets, after which `ranges()` rebuilds the same byte ranges. On the decoder's path, that is an extra `Vec<usize>` and a second pass per column per stage. A variant of `scan_ranges` that returns the selected page indexes directly would make `ColumnFetch` the source and avoid the round-trip, along with any reliance on page offsets being distinct.
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The ranges are generated on demand, as you iterate. The plan does not
+/// calculate a range until you ask for it.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. If you
+/// rebuild the decoder with [`ParquetPushDecoder::into_builder`], the old
+/// plan is not valid for the new decoder. Call `scan_plan` again on the new
+/// decoder.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the entire column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. Its state does not grow with the number of pages.
Review Comment:
"Its state does not grow with the number of pages" holds only without a selection. With one, each column cursor holds `PageSet::Selected(Vec<usize>)`, and starting a stage builds every column's `ColumnFetch` up front. An earlier revision documented the cost with a selection; it might be worth bringing that back.
##########
parquet/src/arrow/push_decoder/scan_plan/mod.rs:
##########
@@ -0,0 +1,1718 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ScanPlan`]: the byte ranges that a push decoder may still read.
+
+use std::cmp::Reverse;
+use std::collections::BinaryHeap;
+use std::iter::FusedIterator;
+use std::ops::Range;
+use std::sync::Arc;
+
+use super::reader_builder::StageSchedule;
+use crate::arrow::arrow_reader::{ReadPlanBuilder, RowSelection};
+use crate::arrow::in_memory_row_group::{
+ ColumnFetch, column_selection, columns_to_fetch, dictionary_range, page_range,
+};
+use crate::errors::ParquetError;
+use crate::file::metadata::page_index::PageIndexProvider;
+use crate::file::page_index::offset_index::PageLocation;
+
+mod budget;
+mod frontier;
+
+pub(crate) use budget::{BudgetedReadPlan, RowBudget};
+pub(crate) use frontier::{NextRowGroup, RowGroupFrontier};
+
+/// One item of a [`ScanPlan`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct PlannedRange {
+ /// Byte range in the file.
+ pub range: Range<u64>,
+ /// Row group index in the file.
+ pub row_group: usize,
+ /// Leaf column index in the file.
+ pub column: usize,
+ /// What the range contains.
+ pub kind: PageKind,
+ /// First planned row that this range serves, counted from the start of
+ /// the plan. Used only to order the plan.
+ pub(crate) first_row: u64,
+ /// One past the last planned row that this range serves.
+ pub(crate) last_row: u64,
+ /// The decoding stage that first reads this range.
+ pub(crate) stage: ScanStage,
+ /// `true` if a predicate result can make this range unnecessary.
+ pub(crate) conditional: bool,
+}
+
+impl PlannedRange {
+ /// Number of bytes in this range.
+ pub fn len(&self) -> u64 {
+ self.range.end - self.range.start
+ }
+
+ /// Returns `true` if this range contains no bytes.
+ pub fn is_empty(&self) -> bool {
+ self.range.is_empty()
+ }
+}
+
+/// What a [`PlannedRange`] contains.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum PageKind {
+ /// The dictionary page of a column chunk.
+ Dictionary,
+ /// One data page.
+ Data,
+ /// A complete column chunk. The plan uses this when the column has no
+ /// offset index, so page locations are unknown.
+ ColumnChunk,
+}
+
+/// The decoding stage that first reads a [`PlannedRange`].
+///
+/// The decoder reads a column once per row group. A column that a predicate
+/// reads is not read again for the output.
+pub(crate) use super::reader_builder::Stage as ScanStage;
+
+/// The byte ranges that a push decoder may still read, in the order that
+/// decoding needs them.
+///
+/// Created by [`ParquetPushDecoder::scan_plan`]. Use it to fetch data before
+/// the decoder asks for it.
+///
+/// A `ScanPlan` is an estimate of the ranges that the decoder may need. The
+/// ranges in [`DecodeResult::NeedsData`] from
+/// [`ParquetPushDecoder::try_decode`] are the ranges that it actually needs.
+///
+/// * The plan contains every range that the decoder can request after
+/// the plan is made. It can contain more, for example ranges that a
+/// [`RowFilter`] makes unnecessary.
+/// * The ranges are generated on demand, as you iterate. The plan does not
+/// calculate a range until you ask for it.
+/// * The plan comes from the current state of the decoder. To plan the
+/// whole scan, call [`ParquetPushDecoder::scan_plan`] once after
+/// [`ParquetPushDecoderBuilder::build`] and keep the iterator. If you
+/// rebuild the decoder with [`ParquetPushDecoder::into_builder`], the old
+/// plan is not valid for the new decoder. Call `scan_plan` again on the new
+/// decoder.
+/// * Row groups are in read order. In a row group, the columns of the
+/// [`RowFilter`] predicates come first, then the other output columns. The
+/// ranges are ordered by the first row that they serve, and the ranges of
+/// one column chunk stay in file order.
+/// * There is one range for each page if the column has an offset index,
+/// and one range for the entire column chunk if not.
+/// * The plan does not change decoding, does no I/O and does not depend
+/// on pushed data. Its state does not grow with the number of pages.
+/// * If the decoder will return an error for a row group, the plan ends
+/// before that row group. An example is a row selection with more rows
+/// than the row group.
+///
+/// Keep the fetched ranges in your own cache, and answer each `NeedsData`
Review Comment:
Two things a read-ahead caller has to work out that the docs could state:
- **Avoiding a copy.** Without a selection, a request is a whole column chunk that spans several planned ranges, so a cache keyed by planned range has to concatenate them for every request (the hidden `read` helper does exactly that). Fetching one buffer per `(row_group, column)` and answering with `Bytes::slice` is zero-copy, and the decoder still releases the buffer, because the pushed range equals the requested one.
- **When cached ranges can be dropped.** Each planned range is requested at most once per visit to its row group, and ranges a predicate prunes are never requested. Because row groups can be read out of order or more than once, eviction has to follow plan position, not the `row_group` number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-5871230043
@adriangb -- are you willing to get this PR ready for review ? I think it is a key step towards even more flexible scheduling policies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6042010155
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10555#issuecomment-6040190549)
**Target:** arrow_reader
Comparing claude/push-decoder-peek-59 (a3b72c1c1d3341b5d20a3844d50c65069183718e) to 02accce (merge-base) [diff](https://github.com/apache/arrow-rs/compare/02accced63a992a20c91cbb5ac24261863531751..a3b72c1c1d3341b5d20a3844d50c65069183718e)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_reader
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 838.1±9.36µs ? ?/sec 1.00 832.9±9.06µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 945.6±4.48µs ? ?/sec 1.00 948.6±4.28µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.01 848.4±13.13µs ? ?/sec 1.00 841.0±7.49µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.04 285.7±0.34µs ? ?/sec 1.00 275.7±0.29µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs, large values 1.07 43.7±2.01ms ? ?/sec 1.00 40.9±1.49ms ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.01 390.1±0.79µs ? ?/sec 1.00 386.0±0.61µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.00 281.7±0.29µs ? ?/sec 1.02 288.1±0.30µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.00 404.0±5.47µs ? ?/sec 1.03 417.2±6.46µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.00 470.0±3.84µs ? ?/sec 1.00 470.2±3.42µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.00 401.9±4.29µs ? ?/sec 1.03 415.7±5.42µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.4±0.07µs ? ?/sec 1.00 80.4±0.08µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.00 111.4±0.18µs ? ?/sec 1.00 111.5±0.13µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 83.9±0.10µs ? ?/sec 1.00 84.3±0.09µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.01 150.4±6.14µs ? ?/sec 1.00 149.4±5.72µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.01 142.1±0.58µs ? ?/sec 1.00 141.3±0.42µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.01 154.0±4.24µs ? ?/sec 1.00 152.9±2.83µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.00 155.5±5.43µs ? ?/sec 1.05 163.9±10.57µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, mandatory, no NULLs 1.00 213.4±1.32µs ? ?/sec 1.00 213.6±0.65µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, half NULLs 1.00 227.9±2.46µs ? ?/sec 1.01 230.8±3.23µs ? ?/sec
arrow_array_reader/BooleanArray/plain encoded, optional, no NULLs 1.00 217.9±1.22µs ? ?/sec 1.00 217.0±0.99µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 969.6±0.98µs ? ?/sec 1.00 972.1±1.65µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 673.9±0.80µs ? ?/sec 1.00 672.1±0.96µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.06 1027.0±17.44µs ? ?/sec 1.00 972.0±1.39µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 191.2±1.24µs ? ?/sec 1.00 188.8±0.72µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 295.7±0.79µs ? ?/sec 1.00 294.9±0.54µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 194.6±1.06µs ? ?/sec 1.00 193.8±0.71µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.01 130.4±0.28µs ? ?/sec 1.00 129.3±0.34µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 182.9±0.19µs ? ?/sec 1.00 183.2±0.28µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 130.3±0.27µs ? ?/sec 1.01 132.1±0.27µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 60.3±0.08µs ? ?/sec 1.00 60.4±0.09µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 147.2±0.12µs ? ?/sec 1.00 147.2±0.19µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 63.1±0.08µs ? ?/sec 1.00 63.3±0.05µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 804.3±0.84µs ? ?/sec 1.00 804.0±0.60µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 511.6±0.56µs ? ?/sec 1.00 510.2±0.66µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.00 807.0±0.89µs ? ?/sec 1.09 875.9±72.16µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.02 23.0±0.09µs ? ?/sec 1.00 22.5±0.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.00 132.1±0.32µs ? ?/sec 1.00 131.5±0.41µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.00 26.7±0.11µs ? ?/sec 1.05 28.0±0.11µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.00 74.2±0.36µs ? ?/sec 1.00 73.9±0.30µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.00 129.2±0.19µs ? ?/sec 1.00 129.0±0.13µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.00 78.0±0.17µs ? ?/sec 1.00 77.7±0.30µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.00 5.4±0.02µs ? ?/sec 1.01 5.4±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.01 93.6±0.23µs ? ?/sec 1.00 93.0±0.12µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.00 8.8±0.03µs ? ?/sec 1.01 8.9±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.19 176.7±22.20µs ? ?/sec 1.00 148.6±0.39µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.00 198.3±0.31µs ? ?/sec 1.01 199.3±0.26µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.01 152.1±0.50µs ? ?/sec 1.00 151.2±0.58µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.00 7.6±0.01µs ? ?/sec 1.02 7.7±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.00 128.1±0.13µs ? ?/sec 1.00 127.7±0.18µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.00 11.1±0.03µs ? ?/sec 1.04 11.5±0.02µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.01 304.0±1.54µs ? ?/sec 1.00 299.9±1.21µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.01 270.1±0.62µs ? ?/sec 1.00 266.6±0.50µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.00 305.5±1.19µs ? ?/sec 1.00 304.7±1.19µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.00 12.1±0.03µs ? ?/sec 1.04 12.6±0.09µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.01 125.3±0.16µs ? ?/sec 1.00 123.8±0.42µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.00 15.7±0.07µs ? ?/sec 1.01 15.8±0.05µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed increasing value 1.00 88.3±0.55µs ? ?/sec 1.02 90.2±4.50µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed single value 1.00 80.0±0.72µs ? ?/sec 1.03 82.6±4.23µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value 1.00 50.8±0.24µs ? ?/sec 1.02 51.9±2.27µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value 1.00 46.7±0.38µs ? ?/sec 1.02 47.7±2.18µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value 1.00 75.4±0.24µs ? ?/sec 1.00 75.5±0.46µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 90.0±0.99µs ? ?/sec 1.00 90.1±0.91µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.00 87.9±0.49µs ? ?/sec 1.00 88.4±0.48µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.00 91.8±0.97µs ? ?/sec 1.00 92.0±0.93µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed stepped increasing value 1.00 108.9±0.13µs ? ?/sec 1.00 109.0±0.76µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.00 127.0±0.90µs ? ?/sec 1.00 127.1±0.85µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.00 145.0±0.39µs ? ?/sec 1.01 145.9±0.46µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.00 130.4±0.93µs ? ?/sec 1.00 130.4±0.78µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 54.4±0.12µs ? ?/sec 1.00 54.5±0.10µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 107.5±0.19µs ? ?/sec 1.00 107.8±0.18µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 57.7±0.14µs ? ?/sec 1.00 57.6±0.10µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 84.9±0.07µs ? ?/sec 1.00 84.6±0.22µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 125.9±0.23µs ? ?/sec 1.00 125.9±0.27µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 87.9±0.10µs ? ?/sec 1.00 88.1±0.19µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 48.2±0.10µs ? ?/sec 1.01 48.6±0.07µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.00 104.5±0.17µs ? ?/sec 1.01 105.2±0.21µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.00 50.6±0.11µs ? ?/sec 1.00 50.8±0.11µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed increasing value 1.01 79.0±0.11µs ? ?/sec 1.00 77.8±0.25µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed single value 1.01 75.1±0.10µs ? ?/sec 1.00 74.6±0.14µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value 1.01 43.7±0.07µs ? ?/sec 1.00 43.1±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value 1.01 41.8±0.06µs ? ?/sec 1.00 41.5±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value 1.00 68.1±0.08µs ? ?/sec 1.00 67.9±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 79.4±0.09µs ? ?/sec 1.00 79.8±0.16µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.00 83.9±0.13µs ? ?/sec 1.00 83.5±0.11µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.00 81.5±0.09µs ? ?/sec 1.00 81.5±0.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed stepped increasing value 1.00 100.8±0.15µs ? ?/sec 1.00 100.6±0.16µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 114.9±0.17µs ? ?/sec 1.00 114.6±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 140.5±0.29µs ? ?/sec 1.00 140.3±0.29µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.00 118.2±0.17µs ? ?/sec 1.00 118.2±0.22µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 83.3±0.15µs ? ?/sec 1.00 83.3±0.19µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 124.9±0.32µs ? ?/sec 1.00 124.4±0.35µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 87.5±0.34µs ? ?/sec 1.01 88.1±0.16µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 86.9±0.07µs ? ?/sec 1.00 87.1±0.17µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.00 128.1±0.27µs ? ?/sec 1.01 129.0±0.27µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 90.5±0.08µs ? ?/sec 1.00 90.9±0.19µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 54.9±0.36µs ? ?/sec 1.00 54.5±0.15µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 109.7±0.37µs ? ?/sec 1.00 109.3±0.21µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.00 58.1±0.24µs ? ?/sec 1.00 58.2±0.18µs ? ?/sec
arrow_array_reader/Int16Array/binary packed increasing value 1.00 54.6±0.58µs ? ?/sec 1.00 54.4±0.69µs ? ?/sec
arrow_array_reader/Int16Array/binary packed single value 1.00 46.2±0.52µs ? ?/sec 1.03 47.4±0.36µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value 1.00 33.7±0.36µs ? ?/sec 1.00 33.6±0.28µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value 1.00 28.9±0.35µs ? ?/sec 1.02 29.6±0.17µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value 1.01 58.7±0.39µs ? ?/sec 1.00 58.4±0.25µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.00 72.4±0.96µs ? ?/sec 1.00 72.5±0.91µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.00 70.8±0.51µs ? ?/sec 1.00 70.7±0.46µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.00 74.5±0.96µs ? ?/sec 1.00 74.4±0.90µs ? ?/sec
arrow_array_reader/Int16Array/binary packed stepped increasing value 1.00 75.7±0.23µs ? ?/sec 1.00 75.5±0.15µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.00 92.3±0.29µs ? ?/sec 1.01 93.1±0.36µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.00 110.8±0.29µs ? ?/sec 1.00 111.0±0.32µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.00 95.9±0.31µs ? ?/sec 1.00 95.9±0.32µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 21.1±0.03µs ? ?/sec 1.00 21.0±0.04µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.00 74.2±0.21µs ? ?/sec 1.01 74.7±0.17µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.06µs ? ?/sec 1.00 24.6±0.05µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.00 51.5±0.04µs ? ?/sec 1.00 51.6±0.04µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.00 92.3±0.22µs ? ?/sec 1.01 92.7±0.17µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.00 55.0±0.05µs ? ?/sec 1.00 54.9±0.05µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.00 14.0±0.03µs ? ?/sec 1.00 13.9±0.03µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.01 71.9±0.17µs ? ?/sec 1.00 71.3±0.18µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.00 17.7±0.04µs ? ?/sec 1.00 17.7±0.03µs ? ?/sec
arrow_array_reader/Int32Array/binary packed increasing value 1.00 50.8±0.70µs ? ?/sec 1.00 50.9±0.70µs ? ?/sec
arrow_array_reader/Int32Array/binary packed single value 1.00 43.0±0.28µs ? ?/sec 1.03 44.2±0.39µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value 1.00 31.6±0.38µs ? ?/sec 1.00 31.7±0.31µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value 1.00 27.4±0.18µs ? ?/sec 1.01 27.7±0.24µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value 1.00 56.6±0.38µs ? ?/sec 1.00 56.4±0.22µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.00 71.1±1.03µs ? ?/sec 1.00 71.0±0.91µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.00 69.5±0.52µs ? ?/sec 1.00 69.5±0.44µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.00 73.1±1.04µs ? ?/sec 1.00 73.2±0.87µs ? ?/sec
arrow_array_reader/Int32Array/binary packed stepped increasing value 1.00 71.8±0.16µs ? ?/sec 1.00 71.6±0.16µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.00 90.0±0.88µs ? ?/sec 1.00 89.8±0.82µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.00 108.7±0.55µs ? ?/sec 1.00 109.0±0.44µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.00 93.3±0.90µs ? ?/sec 1.00 93.6±0.80µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.5±0.05µs ? ?/sec 1.00 17.4±0.06µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.00 70.8±0.16µs ? ?/sec 1.01 71.9±0.19µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.00 20.7±0.03µs ? ?/sec 1.01 21.0±0.06µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.00 47.6±0.03µs ? ?/sec 1.00 47.8±0.12µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.00 88.9±0.19µs ? ?/sec 1.00 89.3±0.24µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.00 51.2±0.06µs ? ?/sec 1.00 51.3±0.06µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.00 9.9±0.03µs ? ?/sec 1.00 9.9±0.03µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.00 68.6±0.24µs ? ?/sec 1.00 68.5±0.23µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.00 13.6±0.04µs ? ?/sec 1.00 13.6±0.06µs ? ?/sec
arrow_array_reader/Int64Array/binary packed increasing value 1.02 41.8±0.15µs ? ?/sec 1.00 41.1±0.17µs ? ?/sec
arrow_array_reader/Int64Array/binary packed single value 1.01 38.7±0.11µs ? ?/sec 1.00 38.3±0.11µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value 1.01 24.7±0.09µs ? ?/sec 1.00 24.4±0.12µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value 1.02 23.0±0.05µs ? ?/sec 1.00 22.7±0.04µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value 1.00 49.1±0.07µs ? ?/sec 1.00 49.0±0.08µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.00 60.8±0.06µs ? ?/sec 1.00 60.7±0.05µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.01 65.1±0.13µs ? ?/sec 1.00 64.7±0.14µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.00 62.5±0.06µs ? ?/sec 1.00 62.5±0.07µs ? ?/sec
arrow_array_reader/Int64Array/binary packed stepped increasing value 1.00 63.7±0.11µs ? ?/sec 1.00 63.8±0.12µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.01 78.0±0.27µs ? ?/sec 1.00 77.4±0.15µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.01 104.0±0.27µs ? ?/sec 1.00 103.2±0.24µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.00 81.8±0.20µs ? ?/sec 1.00 81.4±0.17µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 46.4±0.08µs ? ?/sec 1.00 45.9±0.07µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.01 87.6±0.18µs ? ?/sec 1.00 87.1±0.21µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.7±0.08µs ? ?/sec 1.00 49.8±0.11µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.01 50.3±0.04µs ? ?/sec 1.00 49.9±0.05µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 90.5±0.24µs ? ?/sec 1.00 90.9±0.20µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.00 53.7±0.04µs ? ?/sec 1.00 53.6±0.04µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.02 16.6±0.04µs ? ?/sec 1.00 16.2±0.05µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.00 72.8±0.25µs ? ?/sec 1.00 72.7±0.16µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.00 20.0±0.04µs ? ?/sec 1.01 20.2±0.08µs ? ?/sec
arrow_array_reader/Int8Array/binary packed increasing value 1.00 54.2±0.60µs ? ?/sec 1.01 54.8±0.60µs ? ?/sec
arrow_array_reader/Int8Array/binary packed single value 1.00 46.3±0.27µs ? ?/sec 1.00 46.5±0.65µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value 1.00 33.6±0.44µs ? ?/sec 1.01 33.8±0.23µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value 1.00 29.2±0.23µs ? ?/sec 1.00 29.2±0.36µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value 1.00 58.3±0.39µs ? ?/sec 1.00 58.2±0.24µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.00 72.0±0.88µs ? ?/sec 1.00 71.9±0.76µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.00 70.6±0.42µs ? ?/sec 1.00 70.5±0.43µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.01 74.3±0.85µs ? ?/sec 1.00 73.9±0.77µs ? ?/sec
arrow_array_reader/Int8Array/binary packed stepped increasing value 1.00 75.5±0.14µs ? ?/sec 1.00 75.2±0.14µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.00 92.1±0.30µs ? ?/sec 1.00 92.1±0.17µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.00 110.4±0.22µs ? ?/sec 1.00 110.8±0.28µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 95.7±0.28µs ? ?/sec 1.00 95.5±0.24µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.01 21.2±0.02µs ? ?/sec 1.00 20.9±0.04µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.00 74.2±0.16µs ? ?/sec 1.01 74.9±0.16µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.4±0.05µs ? ?/sec 1.00 24.3±0.04µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.00 51.3±0.04µs ? ?/sec 1.00 51.0±0.05µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.00 92.0±0.22µs ? ?/sec 1.00 92.2±0.19µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.00 54.9±0.06µs ? ?/sec 1.00 54.8±0.08µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.00 13.9±0.03µs ? ?/sec 1.00 13.9±0.05µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.00 71.8±0.16µs ? ?/sec 1.00 72.0±0.22µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.00 17.5±0.04µs ? ?/sec 1.00 17.5±0.02µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/90pct NULLs 1.01 898.7±1.40µs ? ?/sec 1.00 890.6±1.43µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/99pct NULLs 1.01 436.4±0.49µs ? ?/sec 1.00 433.6±0.50µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/half NULLs 1.00 2.2±0.04ms ? ?/sec 1.02 2.3±0.05ms ? ?/sec
arrow_array_reader/ListArray/Fixed32List/no NULLs 1.00 3.1±0.08ms ? ?/sec 1.03 3.2±0.14ms ? ?/sec
arrow_array_reader/ListArray/Int32List/90pct NULLs 1.01 883.7±0.96µs ? ?/sec 1.00 874.2±1.00µs ? ?/sec
arrow_array_reader/ListArray/Int32List/99pct NULLs 1.00 434.7±0.45µs ? ?/sec 1.00 432.8±0.71µs ? ?/sec
arrow_array_reader/ListArray/Int32List/half NULLs 1.00 1975.6±5.91µs ? ?/sec 1.00 1972.7±10.38µs ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.03ms ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs or empty lists 1.02 2.6±0.03ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
arrow_array_reader/ListArray/StringList/90pct NULLs 1.02 951.7±1.55µs ? ?/sec 1.00 937.3±1.47µs ? ?/sec
arrow_array_reader/ListArray/StringList/99pct NULLs 1.01 440.3±0.41µs ? ?/sec 1.00 436.7±0.60µs ? ?/sec
arrow_array_reader/ListArray/StringList/half NULLs 1.02 3.1±0.05ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
arrow_array_reader/ListArray/StringList/no NULLs 1.04 5.1±0.15ms ? ?/sec 1.00 4.9±0.14ms ? ?/sec
arrow_array_reader/PlainStringToDictionary/high cardinality 1.02 1114.4±15.81µs ? ?/sec 1.00 1091.2±14.95µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/low cardinality 1.00 1028.1±12.56µs ? ?/sec 1.00 1031.3±15.46µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 1 1.01 1105.1±18.04µs ? ?/sec 1.00 1090.9±17.02µs ? ?/sec
arrow_array_reader/PlainStringToDictionary/medium cardinality 2 1.01 1050.7±13.84µs ? ?/sec 1.00 1040.0±16.60µs ? ?/sec
arrow_array_reader/StringArray/const delta byte array encoded, mandatory, no NULLs 1.00 545.5±1.45µs ? ?/sec 1.01 549.3±1.44µs ? ?/sec
arrow_array_reader/StringArray/const delta length byte array encoded, mandatory, no NULLs 1.00 217.9±5.75µs ? ?/sec 1.06 231.6±6.85µs ? ?/sec
arrow_array_reader/StringArray/const prefix delta byte array encoded, mandatory, no NULLs 1.00 782.5±42.78µs ? ?/sec 1.01 787.3±43.69µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.00 278.1±0.34µs ? ?/sec 1.02 284.3±0.38µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.00 389.9±0.69µs ? ?/sec 1.00 390.3±0.73µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.04 290.5±0.87µs ? ?/sec 1.00 279.7±0.27µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.00 447.9±7.41µs ? ?/sec 1.01 452.0±4.70µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.00 495.7±4.60µs ? ?/sec 1.01 501.4±4.99µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.00 459.6±5.23µs ? ?/sec 1.02 470.4±14.43µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.00 194.4±1.35µs ? ?/sec 1.00 194.1±0.72µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.00 243.9±1.18µs ? ?/sec 1.00 244.6±0.99µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.00 197.8±1.28µs ? ?/sec 1.00 198.2±0.87µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 80.4±0.05µs ? ?/sec 1.00 80.4±0.06µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.00 111.3±0.11µs ? ?/sec 1.00 111.5±0.14µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 84.1±0.06µs ? ?/sec 1.00 84.3±0.09µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.01 223.2±1.57µs ? ?/sec 1.00 221.5±1.85µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.00 186.1±1.04µs ? ?/sec 1.00 185.5±0.62µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.00 229.7±1.70µs ? ?/sec 1.00 230.0±1.94µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed increasing value 1.00 54.6±0.69µs ? ?/sec 1.00 54.9±0.60µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed single value 1.02 47.3±0.40µs ? ?/sec 1.00 46.2±0.50µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value 1.00 33.6±0.42µs ? ?/sec 1.01 33.9±0.29µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value 1.02 29.6±0.26µs ? ?/sec 1.00 29.0±0.31µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value 1.01 58.6±0.38µs ? ?/sec 1.00 58.2±0.25µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.00 76.0±0.25µs ? ?/sec 1.00 76.2±0.12µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.00 73.0±0.17µs ? ?/sec 1.00 73.2±0.14µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 78.0±0.26µs ? ?/sec 1.00 78.4±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed stepped increasing value 1.00 75.5±0.15µs ? ?/sec 1.00 75.4±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.00 97.2±0.18µs ? ?/sec 1.01 97.9±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.00 115.0±0.25µs ? ?/sec 1.00 114.5±0.25µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 101.1±0.16µs ? ?/sec 1.00 101.1±0.13µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 21.0±0.05µs ? ?/sec 1.00 21.0±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.00 74.2±0.17µs ? ?/sec 1.01 74.8±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.02µs ? ?/sec 1.00 24.4±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.00 51.6±0.04µs ? ?/sec 1.00 51.4±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.00 92.3±0.21µs ? ?/sec 1.00 92.6±0.21µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.00 55.0±0.05µs ? ?/sec 1.00 54.9±0.06µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.03 14.2±0.08µs ? ?/sec 1.00 13.8±0.10µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.01 72.6±0.28µs ? ?/sec 1.00 71.6±0.22µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.00 17.6±0.09µs ? ?/sec 1.00 17.6±0.05µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed increasing value 1.00 51.7±0.59µs ? ?/sec 1.00 51.6±0.61µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed single value 1.00 42.8±0.55µs ? ?/sec 1.00 42.9±0.55µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value 1.00 32.1±0.39µs ? ?/sec 1.00 32.0±0.31µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value 1.00 27.2±0.31µs ? ?/sec 1.00 27.3±0.35µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value 1.00 56.6±0.38µs ? ?/sec 1.00 56.5±0.24µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.00 71.4±1.01µs ? ?/sec 1.00 71.4±0.94µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.00 69.4±0.53µs ? ?/sec 1.00 69.3±0.46µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.00 73.2±1.01µs ? ?/sec 1.00 73.3±0.93µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed stepped increasing value 1.00 72.1±0.17µs ? ?/sec 1.00 71.9±0.16µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.00 90.2±0.98µs ? ?/sec 1.00 90.3±0.94µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.00 108.6±0.52µs ? ?/sec 1.01 109.3±0.45µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.00 93.7±0.95µs ? ?/sec 1.00 94.0±0.92µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.01 17.6±0.04µs ? ?/sec 1.00 17.5±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.00 71.0±0.19µs ? ?/sec 1.01 71.5±0.18µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.00 21.0±0.04µs ? ?/sec 1.00 21.0±0.04µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.00 47.9±0.03µs ? ?/sec 1.00 47.8±0.04µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.00 89.0±0.22µs ? ?/sec 1.00 89.0±0.12µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.00 51.5±0.03µs ? ?/sec 1.00 51.5±0.05µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.00 10.0±0.02µs ? ?/sec 1.01 10.1±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.00 68.6±0.19µs ? ?/sec 1.00 68.9±0.17µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.00 13.8±0.03µs ? ?/sec 1.00 13.8±0.03µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed increasing value 1.01 42.4±0.13µs ? ?/sec 1.00 41.8±0.13µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed single value 1.00 38.7±0.10µs ? ?/sec 1.00 38.6±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value 1.01 24.9±0.07µs ? ?/sec 1.00 24.6±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value 1.00 23.0±0.07µs ? ?/sec 1.00 23.0±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value 1.00 49.1±0.07µs ? ?/sec 1.00 49.1±0.08µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.00 60.9±0.11µs ? ?/sec 1.00 60.8±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.00 65.0±0.14µs ? ?/sec 1.00 64.8±0.10µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.00 62.7±0.10µs ? ?/sec 1.00 62.6±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed stepped increasing value 1.00 63.9±0.10µs ? ?/sec 1.00 63.6±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 77.9±0.11µs ? ?/sec 1.00 77.8±0.16µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.01 104.6±0.22µs ? ?/sec 1.00 103.5±0.23µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.00 81.7±0.13µs ? ?/sec 1.00 81.7±0.14µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 46.3±0.08µs ? ?/sec 1.00 46.0±0.09µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.00 87.7±0.26µs ? ?/sec 1.00 87.4±0.25µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.9±0.12µs ? ?/sec 1.00 49.7±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.01 50.6±0.05µs ? ?/sec 1.00 50.1±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.00 90.8±0.25µs ? ?/sec 1.00 90.9±0.24µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.01 53.9±0.07µs ? ?/sec 1.00 53.6±0.07µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.03 16.8±0.06µs ? ?/sec 1.00 16.3±0.05µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.00 73.3±0.24µs ? ?/sec 1.00 73.4±0.19µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.00 20.4±0.06µs ? ?/sec 1.00 20.4±0.06µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed increasing value 1.00 54.5±0.67µs ? ?/sec 1.00 54.3±0.74µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed single value 1.00 46.4±0.38µs ? ?/sec 1.02 47.2±0.47µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value 1.00 33.4±0.43µs ? ?/sec 1.00 33.6±0.30µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value 1.00 29.2±0.27µs ? ?/sec 1.01 29.6±0.29µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value 1.00 58.4±0.40µs ? ?/sec 1.00 58.1±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.00 75.7±0.38µs ? ?/sec 1.00 75.7±0.43µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.00 72.4±0.24µs ? ?/sec 1.00 72.4±0.19µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.00 77.8±0.33µs ? ?/sec 1.00 77.5±0.31µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed stepped increasing value 1.00 75.4±0.14µs ? ?/sec 1.00 75.2±0.15µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.00 97.3±0.12µs ? ?/sec 1.00 97.4±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.00 113.8±0.25µs ? ?/sec 1.00 114.1±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.00 100.5±0.12µs ? ?/sec 1.00 100.3±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.05 22.0±0.03µs ? ?/sec 1.00 21.0±0.04µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.01 75.1±0.18µs ? ?/sec 1.00 74.5±0.19µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.7±0.03µs ? ?/sec 1.00 24.7±0.04µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.00 51.2±0.03µs ? ?/sec 1.00 51.2±0.06µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.00 92.2±0.22µs ? ?/sec 1.00 92.4±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.00 54.8±0.05µs ? ?/sec 1.01 55.1±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.01 13.9±0.09µs ? ?/sec 1.00 13.8±0.04µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.00 71.3±0.21µs ? ?/sec 1.02 72.4±0.20µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.03 17.9±0.02µs ? ?/sec 1.00 17.4±0.04µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, half NULLs 1.00 68.9±0.16µs ? ?/sec 1.00 68.8±0.16µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, no NULLs 1.01 13.9±0.03µs ? ?/sec 1.00 13.8±0.05µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, half NULLs 1.01 129.9±0.19µs ? ?/sec 1.00 128.7±0.19µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, no NULLs 1.03 66.9±0.17µs ? ?/sec 1.00 65.1±0.19µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>arrow_reader — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-k44dd5ur-8831919f-9dvr
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6040190549-3122-c5srt 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 3010.7s |
| Peak memory | 189.6 MiB |
| Avg memory | 41.3 MiB |
| CPU user | 2994.5s |
| CPU sys | 13.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 3025.7s |
| Peak memory | 189.3 MiB |
| Avg memory | 43.5 MiB |
| CPU user | 3007.7s |
| CPU sys | 12.1s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6040190549:finish:af342652bf60e31b2927dfb7fb47c56d -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6044079451
I'll plan to merge this tomorrow unless anyone wants more time to review
Thank you @HippoBaro and @viirya for the reviews
I filed a ticket to try and consolidate the planning logic as I am very worried about the long term maintenance / understandability of splitting the ScanPlan and actual reader logic
- https://github.com/apache/arrow-rs/issues/11428
I also can help shake the feeling that the current structure is now more complicated than the problem that it is trying to solve -- and that the PushDecoder has exceeeded my ability to understand the code anymore.
So basically this is a plean to help try and simplify before we embark on the next big refactor (aka incrementally decoding row groups..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6065605336
This one looks like it has a small conflict and then it will be ready to merge
I am sorry I can't resolve the conflict myself, but I don't have push access to your fork
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555#issuecomment-6067308353
@alamb , I rebased can you requeue please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead planning [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10555:
URL: https://github.com/apache/arrow-rs/pull/10555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org