You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/05/22 03:22:39 UTC
[PR] feat(parquet): apply TopK threshold to row-group statistics mid-scan [datafusion]
zhuqi-lucas opened a new pull request, #22450:
URL: https://github.com/apache/datafusion/pull/22450
<!--
Thanks for contributing to DataFusion!
-->
## Which issue does this PR close?
Closes #22407.
## Rationale for this change
DataFusion already prunes parquet at three granularities — **file**
(`EarlyStoppingStream` + `FilePruner`), **row group at scan-startup**
(`PruningPredicate` → `RowGroupAccessPlanFilter`), and **row inside an
open RG** (`RowFilter`).
There's a gap in the middle: once Layer 1 (RG-static) picks the row
groups at file open, that decision is **frozen** because the dynamic
filter is still `lit(true)` then. As `TopK` tightens its threshold at
runtime, subsequent RGs in the already-opened file keep getting decoded
even when their stats already prove they can't beat the threshold. This
is the dominant cost for `ORDER BY ... LIMIT` queries on multi-RG files
where file-level pruning can't help (single large file, or scrambled-RG
multi-file).
See the issue for a full architectural diagram and a concrete trace
showing where the wasted I/O / decompression / decode lives.
## What changes are included in this PR?
Two coordinated pieces that close the gap:
1. **`RowGroupPruner`** (in `datafusion/datasource-parquet/src/push_decoder.rs`)
mirrors `FilePruner`'s pattern at row-group granularity. Tracks
`snapshot_generation(&predicate)` so the cached `PruningPredicate`
is rebuilt only when the dynamic filter has actually moved, then
evaluates against the next pending decoder run's row-group stats
via the existing `RowGroupPruningStatistics` adapter. Errors fall
back to "don't prune" — a flaky pruning path never silently drops
data.
2. **Per-row-group decoder splitting when the predicate is dynamic**.
`ParquetAccessPlan::split_runs` previously coalesced consecutive
same-`fully_matched` RGs into a single run. For ORDER BY + LIMIT
the initial dynamic filter is `lit(true)`, so the static
fully-matched analysis marks nothing and `split_runs` collapsed
every RG into one run — leaving no inter-run hook. A new
`force_per_row_group` flag (set by `is_dynamic_physical_expr`)
disables coalescing **for dynamic predicates only**, so static
WHERE queries pay nothing.
`PendingDecoderRun` wraps each queued decoder with its row group
indices. `PushDecoderStreamState::transition` consults the pruner at
every run boundary and skips runs whose row groups are proved
unwinnable.
### Observability
- New `Count` metric `row_groups_pruned_dynamic_filter` on
`ParquetFileMetrics` surfaces the runtime saving.
- New `dynamic_rg_pruning=eligible` marker on `ParquetSource`'s
`EXPLAIN` (`fmt_extra` Default + Verbose) signals plan-time
eligibility. **Eligible** rather than **true** because the static
plan can't predict the runtime outcome.
### Benchmarks (`benchmarks/sort_pushdown_inexact`, 5 iterations)
| Query | main | this PR | Δ |
|---|---|---|---|
| Q1 `ORDER BY l_orderkey DESC LIMIT 100` | 6.99 ms | 3.80 ms | **−46%** |
| Q2 `ORDER BY l_orderkey DESC LIMIT 1000` | 3.29 ms | 1.33 ms | **−60%** |
| Q3 `SELECT * ... DESC LIMIT 100` | 11.17 ms | 9.91 ms | −11% |
| Q4 `SELECT * ... DESC LIMIT 1000` | 9.28 ms | 7.95 ms | −14% |
Narrow-projection queries gain the most — their per-RG cost is
dominated by metadata + sort-column read, which this PR eliminates
for unwinnable RGs. Wide-projection queries gain less because the
*kept* RG's all-column decode dominates total time, but still see
meaningful savings.
## Are these changes tested?
Yes. Three layers:
- **6 unit tests**:
- 3 in `push_decoder.rs::tests`: `RowGroupPruner` basic pruning,
generation-tracked dynamic-filter updates, fallback when the
predicate has no analyzable bounds.
- 3 in `source.rs::tests`: `dynamic_rg_pruning=eligible` marker
present on dynamic predicate, absent on static predicate, absent
when there is no predicate at all.
- **2 integration tests** in
`datafusion/core/tests/parquet/dynamic_row_group_pruning.rs`:
asserts `row_groups_pruned_dynamic_filter >= 1` end-to-end on a
5-RG `ORDER BY DESC LIMIT 5` scan, and asserts the metric stays at
0 when there is no TopK (no spurious firing).
- **New SLT**
`datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt`:
asserts both `EXPLAIN` surfaces — plain `EXPLAIN` shows
`dynamic_rg_pruning=eligible`, and `EXPLAIN ANALYZE` pins
`row_groups_pruned_dynamic_filter=4` (five RGs, four pruned at
runtime).
129 parquet unit + 204 parquet integration + SLT all pass.
`cargo clippy --all-targets --all-features -- -D warnings` clean.
## Are there any user-facing changes?
Two visible additions, both opt-in via existing dynamic-filter
infrastructure:
- New `row_groups_pruned_dynamic_filter` counter visible in
`EXPLAIN ANALYZE` for queries whose plan carries a
`DynamicFilterPhysicalExpr` (today: only TopK with
`enable_topk_dynamic_filter_pushdown=true`, which is the default).
- New `dynamic_rg_pruning=eligible` marker visible in `EXPLAIN`
output for the same queries.
No config changes, no API breakage, no behavior change for queries
without a dynamic predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515502111
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.13 / 4.63 ±6.95 / 18.54 ms │ 1.11 / 4.59 ±6.90 / 18.39 ms │ no change │
│ QQuery 1 │ 12.84 / 12.94 ±0.08 / 13.04 ms │ 12.42 / 12.73 ±0.18 / 12.95 ms │ no change │
│ QQuery 2 │ 35.38 / 35.81 ±0.30 / 36.23 ms │ 35.63 / 35.87 ±0.28 / 36.35 ms │ no change │
│ QQuery 3 │ 30.33 / 30.94 ±0.92 / 32.76 ms │ 30.38 / 30.89 ±0.60 / 32.06 ms │ no change │
│ QQuery 4 │ 221.08 / 222.52 ±1.32 / 224.77 ms │ 217.25 / 223.77 ±4.47 / 230.65 ms │ no change │
│ QQuery 5 │ 271.34 / 273.02 ±1.66 / 276.10 ms │ 267.27 / 271.69 ±2.28 / 273.41 ms │ no change │
│ QQuery 6 │ 1.16 / 1.31 ±0.22 / 1.75 ms │ 1.18 / 1.32 ±0.21 / 1.73 ms │ no change │
│ QQuery 7 │ 13.82 / 14.01 ±0.16 / 14.24 ms │ 14.29 / 14.44 ±0.14 / 14.67 ms │ no change │
│ QQuery 8 │ 319.01 / 323.19 ±3.92 / 329.01 ms │ 342.38 / 356.25 ±7.86 / 364.96 ms │ 1.10x slower │
│ QQuery 9 │ 446.98 / 451.24 ±3.38 / 455.61 ms │ 463.18 / 475.35 ±10.89 / 489.79 ms │ 1.05x slower │
│ QQuery 10 │ 68.91 / 69.79 ±0.81 / 71.12 ms │ 70.01 / 71.13 ±0.70 / 71.92 ms │ no change │
│ QQuery 11 │ 80.17 / 81.70 ±1.04 / 83.41 ms │ 80.46 / 81.98 ±1.59 / 84.88 ms │ no change │
│ QQuery 12 │ 262.57 / 271.05 ±6.00 / 278.82 ms │ 263.13 / 272.47 ±6.80 / 283.05 ms │ no change │
│ QQuery 13 │ 360.50 / 369.16 ±10.13 / 388.45 ms │ 357.78 / 364.59 ±3.90 / 369.19 ms │ no change │
│ QQuery 14 │ 276.41 / 281.73 ±4.61 / 289.96 ms │ 280.05 / 286.88 ±6.92 / 298.91 ms │ no change │
│ QQuery 15 │ 264.25 / 276.30 ±11.55 / 292.24 ms │ 294.81 / 307.60 ±14.15 / 333.58 ms │ 1.11x slower │
│ QQuery 16 │ 627.27 / 659.89 ±17.60 / 680.12 ms │ 616.34 / 643.05 ±19.90 / 673.10 ms │ no change │
│ QQuery 17 │ 624.54 / 639.67 ±10.36 / 653.36 ms │ 602.95 / 613.57 ±9.74 / 631.66 ms │ no change │
│ QQuery 18 │ 1231.71 / 1264.30 ±32.09 / 1324.58 ms │ 1230.87 / 1249.03 ±13.77 / 1269.77 ms │ no change │
│ QQuery 19 │ 29.95 / 33.49 ±3.93 / 38.81 ms │ 27.88 / 34.23 ±7.63 / 44.83 ms │ no change │
│ QQuery 20 │ 531.85 / 542.39 ±8.77 / 552.84 ms │ 518.59 / 522.88 ±3.22 / 527.36 ms │ no change │
│ QQuery 21 │ 590.04 / 600.69 ±8.53 / 615.62 ms │ 589.86 / 595.16 ±4.54 / 603.56 ms │ no change │
│ QQuery 22 │ 1047.46 / 1051.15 ±3.07 / 1054.60 ms │ 1085.28 / 1112.76 ±21.95 / 1146.31 ms │ 1.06x slower │
│ QQuery 23 │ 3143.81 / 3215.59 ±61.28 / 3327.12 ms │ 3009.38 / 3115.16 ±74.14 / 3210.79 ms │ no change │
│ QQuery 24 │ 43.61 / 45.10 ±1.49 / 47.69 ms │ 41.73 / 49.54 ±6.86 / 58.93 ms │ 1.10x slower │
│ QQuery 25 │ 116.41 / 117.72 ±0.81 / 118.72 ms │ 112.35 / 113.26 ±0.66 / 114.42 ms │ no change │
│ QQuery 26 │ 44.00 / 45.32 ±1.39 / 47.91 ms │ 41.64 / 42.41 ±0.68 / 43.37 ms │ +1.07x faster │
│ QQuery 27 │ 668.27 / 677.75 ±9.71 / 696.11 ms │ 666.49 / 674.74 ±7.19 / 686.27 ms │ no change │
│ QQuery 28 │ 3004.40 / 3024.76 ±13.42 / 3039.76 ms │ 3014.79 / 3056.61 ±36.42 / 3111.94 ms │ no change │
│ QQuery 29 │ 40.04 / 51.77 ±14.67 / 76.55 ms │ 41.49 / 49.56 ±15.74 / 81.03 ms │ no change │
│ QQuery 30 │ 325.79 / 329.77 ±3.21 / 335.46 ms │ 311.00 / 329.53 ±13.27 / 351.40 ms │ no change │
│ QQuery 31 │ 296.89 / 313.26 ±9.84 / 323.06 ms │ 278.41 / 291.30 ±11.42 / 310.81 ms │ +1.08x faster │
│ QQuery 32 │ 934.13 / 949.51 ±9.91 / 963.93 ms │ 914.90 / 929.16 ±15.07 / 958.29 ms │ no change │
│ QQuery 33 │ 1410.05 / 1512.63 ±79.05 / 1623.38 ms │ 1406.57 / 1430.30 ±12.88 / 1443.94 ms │ +1.06x faster │
│ QQuery 34 │ 1422.56 / 1450.90 ±17.81 / 1474.31 ms │ 1424.50 / 1486.88 ±73.50 / 1624.17 ms │ no change │
│ QQuery 35 │ 272.88 / 291.31 ±26.72 / 344.13 ms │ 276.79 / 319.68 ±35.69 / 379.77 ms │ 1.10x slower │
│ QQuery 36 │ 61.63 / 71.05 ±7.09 / 80.23 ms │ 65.89 / 82.70 ±12.28 / 102.66 ms │ 1.16x slower │
│ QQuery 37 │ 34.65 / 35.40 ±0.76 / 36.80 ms │ 36.38 / 40.19 ±5.70 / 51.48 ms │ 1.14x slower │
│ QQuery 38 │ 42.14 / 46.54 ±3.98 / 52.83 ms │ 41.07 / 43.14 ±1.72 / 45.06 ms │ +1.08x faster │
│ QQuery 39 │ 144.74 / 151.58 ±5.96 / 160.59 ms │ 151.07 / 153.66 ±2.72 / 158.21 ms │ no change │
│ QQuery 40 │ 13.68 / 16.08 ±3.85 / 23.74 ms │ 15.04 / 15.47 ±0.36 / 15.96 ms │ no change │
│ QQuery 41 │ 13.32 / 13.46 ±0.13 / 13.69 ms │ 14.42 / 16.08 ±3.04 / 22.15 ms │ 1.19x slower │
│ QQuery 42 │ 12.88 / 14.68 ±3.42 / 21.51 ms │ 14.07 / 14.26 ±0.12 / 14.40 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19885.11ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 19835.88ms │
│ Average Time (HEAD) │ 462.44ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 461.30ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 9 │
│ Queries with No Change │ 30 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 29.8 GiB |
| Avg memory | 22.9 GiB |
| CPU user | 1032.4s |
| CPU sys | 67.0s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 30.6 GiB |
| Avg memory | 23.2 GiB |
| CPU user | 1033.3s |
| CPU sys | 67.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4705753361
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4705734492)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4705734492-567-njzv5 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (ef23530f9c32f39d349e4a8396ec9b963c106c68) to d428760 (merge-base) [diff](https://github.com/apache/datafusion/compare/d428760d709a375f3d997c84e9c4748a22584149..ef23530f9c32f39d349e4a8396ec9b963c106c68) using: topk_tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421029198
##########
datafusion/core/tests/parquet/dynamic_row_group_pruning.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! End-to-end test for **runtime row-group pruning** driven by a TopK
+//! `SortExec`'s `DynamicFilterPhysicalExpr`.
+//!
+//! A 5-row-group parquet file is constructed with disjoint statistics on
+//! the sort column (`v`): row group `i` contains values
+//! `[i*100, (i+1)*100)`. The query `ORDER BY v DESC LIMIT 5` fills the
+//! TopK heap from the row group with the largest values; the threshold
+//! then proves the remaining row groups cannot contribute. The runtime
+//! `RowGroupPruner` in the parquet scan must observe the tightened
+//! threshold and increment `row_groups_pruned_dynamic_filter`.
+//!
+//! We assert a property (`pruned >= 1`) rather than an exact count
+//! because batch-arrival timing affects how soon the TopK heap fills,
+//! and we don't want this test to become flaky.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+
+use crate::parquet::Unit::RowGroup;
+use crate::parquet::{ContextWithParquet, Scenario};
+
+/// Build five `RecordBatch`es whose `v` column ranges are disjoint:
+/// batch `i` carries `v` values `[i*100, (i+1)*100)`. When written with
+/// `max_row_group_row_count = 100` each batch lands in its own row group.
+fn build_five_disjoint_batches(schema: &Arc<Schema>) -> Vec<RecordBatch> {
+ (0..5i64)
+ .map(|rg| {
+ let base = rg * 100;
+ let values: Vec<i64> = (base..base + 100).collect();
+ let col: ArrayRef = Arc::new(Int64Array::from(values));
+ RecordBatch::try_new(Arc::clone(schema), vec![col]).unwrap()
+ })
+ .collect()
+}
+
+/// Build five `RecordBatch`es in *descending* value order: batch 0 holds
+/// `v ∈ [400, 500)`, batch 4 holds `v ∈ [0, 100)`. The physical row-group
+/// order on disk therefore does **not** match the order a `ORDER BY v ASC`
+/// query wants — sort-pushdown's `reorder_by_statistics` must rearrange
+/// the access plan so the scan reads RG 4 first, then RG 3, etc.
+fn build_five_disjoint_batches_desc(schema: &Arc<Schema>) -> Vec<RecordBatch> {
+ (0..5i64)
+ .map(|rg| {
+ let base = (4 - rg) * 100;
+ let values: Vec<i64> = (base..base + 100).collect();
+ let col: ArrayRef = Arc::new(Int64Array::from(values));
+ RecordBatch::try_new(Arc::clone(schema), vec![col]).unwrap()
+ })
+ .collect()
+}
+
+/// `ORDER BY v DESC LIMIT 5` against a 5-RG file with disjoint per-RG
+/// stats must trigger runtime RG pruning: the first RG read fills the
+/// heap, and the tightened threshold proves every other RG unreachable.
+#[tokio::test]
+async fn dynamic_rg_pruning_metric_fires_for_topk_descending_limit() {
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batches = build_five_disjoint_batches(&schema);
+
+ // `with_custom_data` honors the custom schema + batches and ignores
+ // `Scenario`. `Unit::RowGroup(100)` enables `pushdown_filters`, which
+ // is required for the TopK dynamic filter to reach the parquet scan.
+ let mut ctx = ContextWithParquet::with_custom_data(
+ Scenario::Int,
+ RowGroup(100),
+ Arc::clone(&schema),
+ batches,
+ )
+ .await;
+
+ let output = ctx.query("SELECT v FROM t ORDER BY v DESC LIMIT 5").await;
+
+ assert_eq!(output.result_rows, 5, "query must return LIMIT rows",);
+
+ let pruned = output
+ .row_groups_pruned_dynamic_filter()
+ .expect("`row_groups_pruned_dynamic_filter` metric must be registered");
+ assert!(
+ pruned >= 1,
+ "dynamic RG pruner must skip at least one row group; \
+ pruned={pruned}\n{}",
+ output.description(),
+ );
+}
+
+/// Regression for the rg_plan / `reorder_by_statistics` ordering bug.
+///
+/// When `sort_order_for_reorder` is set on the parquet scan,
+/// `prepare_access_plan` calls
+/// [`PreparedAccessPlan::reorder_by_statistics`], which rearranges
+/// `row_group_indexes` so the decoder reads row groups in stats-optimal
+/// order (smallest-min first for ASC, etc.). The stream's per-RG plan
+/// (`rg_plan`) — which the runtime pruner walks one entry at a time —
+/// **must use this reordered list**, not the access plan's natural
+/// (index-ascending) order. Otherwise the pruner would consult the
+/// metadata of RG K while the decoder is actually about to yield RG K',
+/// silently producing wrong results.
+///
+/// This test makes the failure visible:
+///
+/// - File is written with RGs in *descending* `v` order (RG 0 has the
+/// largest values, RG 4 has the smallest).
+/// - Query is `ORDER BY v ASC LIMIT 5`, so sort-pushdown reorders the
+/// access plan to read RG 4 first, then RG 3, etc.
+/// - The smallest five values (which form the entire correct LIMIT
+/// answer) live in RG 4 alone. After they are emitted, the TopK
+/// threshold tightens enough that the per-RG pruner skips every other
+/// RG.
+///
+/// Without the fix, `rg_plan` would be `[0, 1, 2, 3, 4]` while the
+/// decoder reads `[4, 3, 2, 1, 0]`. The first yielded reader (for RG 4
+/// in the decoder) would be tracked as if it were RG 0, the pruner
+/// would check RG 1's stats (id range 300..400) against a threshold
+/// already tightened to `v < 5`, prune RG 1 (because nothing in
+/// 300..400 can satisfy `v < 5`), and then the rebuild via
+/// `into_builder` would scan a row group whose data does not match its
+/// expected metadata. The query would return fewer than five rows or
+/// the wrong rows.
+#[tokio::test]
+async fn dynamic_rg_pruning_handles_sort_pushdown_reorder() {
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batches = build_five_disjoint_batches_desc(&schema);
+
+ let mut ctx = ContextWithParquet::with_custom_data(
+ Scenario::Int,
+ RowGroup(100),
+ Arc::clone(&schema),
+ batches,
+ )
+ .await;
+
+ let output = ctx.query("SELECT v FROM t ORDER BY v ASC LIMIT 5").await;
+
+ // Correctness — the five smallest values in the file are 0..=4.
+ // If `rg_plan` is misaligned with the decoder's read order, the
+ // pruner consults the wrong RG's stats and the result row count or
+ // values would drift.
+ assert_eq!(output.result_rows, 5, "query must return LIMIT rows");
+ let formatted = output.pretty_results();
+ for v in 0..=4i64 {
+ assert!(
+ formatted.contains(&format!("| {v} ")),
+ "output must contain the smallest value {v}; got:\n{formatted}",
+ );
+ }
+
+ // Behavior — the per-RG pruner must engage. We don't pin the exact
+ // count (batch-arrival timing affects how soon the heap fills); we
+ // only require that at least one row group is skipped at runtime.
+ let pruned = output
+ .row_groups_pruned_dynamic_filter()
+ .expect("`row_groups_pruned_dynamic_filter` metric must be registered");
+ assert!(
+ pruned >= 1,
+ "with `sort_order_for_reorder` active and a tight TopK, the \
+ runtime pruner must skip at least one row group; pruned={pruned}\n{}",
+ output.description(),
+ );
+}
+
+/// A query without ORDER BY does not produce a TopK and therefore no
+/// `DynamicFilterPhysicalExpr` reaches the scan. The runtime pruner must
+/// stay quiet — the metric should be 0.
+#[tokio::test]
+async fn dynamic_rg_pruning_metric_quiet_without_topk() {
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batches = build_five_disjoint_batches(&schema);
+
+ let mut ctx = ContextWithParquet::with_custom_data(
+ Scenario::Int,
+ RowGroup(100),
+ Arc::clone(&schema),
+ batches,
+ )
+ .await;
+
+ // Plain `SELECT *` — no sort, no limit, no dynamic filter.
+ let output = ctx.query("SELECT v FROM t").await;
+ assert_eq!(output.result_rows, 500);
+
+ let pruned = output.row_groups_pruned_dynamic_filter().unwrap_or(0);
+ assert_eq!(
+ pruned,
+ 0,
+ "without TopK there is no dynamic filter, so the runtime pruner \
+ must not fire; pruned={pruned}\n{}",
+ output.description(),
+ );
+}
Review Comment:
Added the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421988206
##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -151,7 +305,72 @@ impl PushDecoderStreamState {
if self.remaining_limit == Some(0) {
return None;
}
- match self.decoder.try_decode() {
+
+ // Step 1: drain a batch from the active reader if any.
+ if let Some(reader) = self.active_reader.as_mut() {
+ match reader.next() {
+ Some(Ok(batch)) => {
+ let mut timer = self.baseline_metrics.elapsed_compute().timer();
+ self.copy_arrow_reader_metrics();
+ let result = self.project_batch(&batch);
+ timer.stop();
+ drop(timer);
+ return Some((result, self));
+ }
+ Some(Err(e)) => {
+ return Some((Err(DataFusionError::from(e)), self));
+ }
+ None => {
+ // Reader exhausted: drop and fall through to per-RG
+ // boundary handling, then try_next_reader.
+ self.active_reader = None;
+ }
+ }
+ }
+
+ // Step 2: at RG boundary, drop pruned RGs from the head of the
+ // plan. Rebuild the decoder via `into_builder` so it skips the
+ // pruned RGs entirely. Buffered bytes for already-fetched RGs
+ // carry across the rebuild.
+ if !self.rg_plan.is_empty() {
+ let mut pruned_head = 0usize;
+ while let Some(next) = self.rg_plan.front() {
+ let pruned = self
+ .row_group_pruner
+ .as_mut()
+ .map(|p| p.should_prune(&[next.rg_index]))
+ .unwrap_or(false);
+ if pruned {
+ pruned_head += 1;
+ self.rg_plan.pop_front();
+ self.row_groups_pruned_dynamic.add(1);
+ } else {
+ break;
+ }
+ }
+ if pruned_head > 0 {
+ if self.rg_plan.is_empty() {
+ return None;
+ }
+ let decoder = self.decoder.take().expect("decoder present");
+ let new_indices: Vec<usize> =
+ self.rg_plan.iter().map(|e| e.rg_index).collect();
+ let rebuilt = match decoder.into_builder() {
+ Ok(b) => b.with_row_groups(new_indices).build(),
+ Err(e) => Err(e),
+ };
Review Comment:
Hmm I'm not sure we've hit it. I would add a test with a filter like `where v > 250` and make the page index size target 1 byte and the write chunk size 1 row so that we get 1 row per page -> the filter should generate a `RowSelection` based on page index pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765211777
Thank you @adriangb @Dandandan @alamb for review.
Merged to main branch, and the final result is good, the topk benchmark shows almost 50% reduce for the total time.
https://github.com/apache/datafusion/pull/22450#issuecomment-4765161078
```rust
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark run_topk_tpch.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 2.10 / 2.57 ±0.68 / 3.92 ms │ 2.13 / 2.65 ±0.75 / 4.15 ms │ no change │
│ Q2 │ 10.45 / 10.87 ±0.53 / 11.91 ms │ 2.94 / 2.99 ±0.04 / 3.04 ms │ +3.64x faster │
│ Q3 │ 31.39 / 31.67 ±0.29 / 32.15 ms │ 31.74 / 32.30 ±0.40 / 32.91 ms │ no change │
│ Q4 │ 11.62 / 12.45 ±0.79 / 13.48 ms │ 3.24 / 3.31 ±0.08 / 3.46 ms │ +3.76x faster │
│ Q5 │ 9.80 / 10.31 ±0.32 / 10.72 ms │ 10.12 / 10.24 ±0.07 / 10.34 ms │ no change │
│ Q6 │ 16.82 / 16.85 ±0.03 / 16.90 ms │ 17.28 / 17.45 ±0.12 / 17.62 ms │ no change │
│ Q7 │ 36.31 / 37.01 ±0.66 / 38.06 ms │ 37.17 / 37.35 ±0.18 / 37.65 ms │ no change │
│ Q8 │ 27.38 / 28.15 ±0.92 / 29.80 ms │ 6.85 / 6.97 ±0.09 / 7.10 ms │ +4.04x faster │
│ Q9 │ 34.83 / 35.50 ±1.16 / 37.82 ms │ 8.28 / 8.53 ±0.29 / 9.10 ms │ +4.16x faster │
│ Q10 │ 53.00 / 53.78 ±0.96 / 55.65 ms │ 12.45 / 12.54 ±0.06 / 12.62 ms │ +4.29x faster │
│ Q11 │ 3.77 / 3.81 ±0.04 / 3.88 ms │ 3.78 / 4.16 ±0.63 / 5.42 ms │ 1.09x slower │
└───────┴────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 242.96ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 138.48ms │
│ Average Time (HEAD) │ 22.09ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 12.59ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 5 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765161078
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765059254)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark run_topk_tpch.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 2.10 / 2.57 ±0.68 / 3.92 ms │ 2.13 / 2.65 ±0.75 / 4.15 ms │ no change │
│ Q2 │ 10.45 / 10.87 ±0.53 / 11.91 ms │ 2.94 / 2.99 ±0.04 / 3.04 ms │ +3.64x faster │
│ Q3 │ 31.39 / 31.67 ±0.29 / 32.15 ms │ 31.74 / 32.30 ±0.40 / 32.91 ms │ no change │
│ Q4 │ 11.62 / 12.45 ±0.79 / 13.48 ms │ 3.24 / 3.31 ±0.08 / 3.46 ms │ +3.76x faster │
│ Q5 │ 9.80 / 10.31 ±0.32 / 10.72 ms │ 10.12 / 10.24 ±0.07 / 10.34 ms │ no change │
│ Q6 │ 16.82 / 16.85 ±0.03 / 16.90 ms │ 17.28 / 17.45 ±0.12 / 17.62 ms │ no change │
│ Q7 │ 36.31 / 37.01 ±0.66 / 38.06 ms │ 37.17 / 37.35 ±0.18 / 37.65 ms │ no change │
│ Q8 │ 27.38 / 28.15 ±0.92 / 29.80 ms │ 6.85 / 6.97 ±0.09 / 7.10 ms │ +4.04x faster │
│ Q9 │ 34.83 / 35.50 ±1.16 / 37.82 ms │ 8.28 / 8.53 ±0.29 / 9.10 ms │ +4.16x faster │
│ Q10 │ 53.00 / 53.78 ±0.96 / 55.65 ms │ 12.45 / 12.54 ±0.06 / 12.62 ms │ +4.29x faster │
│ Q11 │ 3.77 / 3.81 ±0.04 / 3.88 ms │ 3.78 / 4.16 ±0.63 / 5.42 ms │ 1.09x slower │
└───────┴────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 242.96ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 138.48ms │
│ Average Time (HEAD) │ 22.09ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 12.59ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 5 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**topk_tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 576.2 MiB |
| Avg memory | 96.1 MiB |
| CPU user | 0.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**topk_tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 236.0 KiB |
| Avg memory | 39.3 KiB |
| CPU user | 0.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765073499
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4765060057-604-m75rc 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (23339c539055c69a5157f8b00de7581c862810b7) to ba67bb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/ba67bb467a30c3dbe7f78089451ea4c4d1a191d5..23339c539055c69a5157f8b00de7581c862810b7) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4705854354
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4705734492)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark run_topk_tpch.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 2.22 / 3.84 ±1.73 / 7.19 ms │ 2.13 / 2.56 ±0.79 / 4.15 ms │ +1.50x faster │
│ Q2 │ 10.44 / 10.83 ±0.51 / 11.81 ms │ 2.79 / 2.92 ±0.09 / 3.04 ms │ +3.71x faster │
│ Q3 │ 31.94 / 33.19 ±1.07 / 34.80 ms │ 31.85 / 32.96 ±1.07 / 34.48 ms │ no change │
│ Q4 │ 11.53 / 11.67 ±0.08 / 11.79 ms │ 3.08 / 3.22 ±0.18 / 3.58 ms │ +3.62x faster │
│ Q5 │ 9.41 / 9.58 ±0.17 / 9.87 ms │ 9.70 / 10.20 ±0.47 / 10.94 ms │ 1.06x slower │
│ Q6 │ 16.65 / 16.81 ±0.14 / 17.00 ms │ 17.00 / 17.06 ±0.05 / 17.15 ms │ no change │
│ Q7 │ 36.72 / 37.36 ±0.84 / 38.96 ms │ 36.64 / 37.62 ±0.94 / 38.82 ms │ no change │
│ Q8 │ 27.80 / 28.82 ±0.88 / 30.35 ms │ 6.94 / 7.03 ±0.10 / 7.22 ms │ +4.10x faster │
│ Q9 │ 35.11 / 35.64 ±0.72 / 37.06 ms │ 8.50 / 8.54 ±0.04 / 8.59 ms │ +4.17x faster │
│ Q10 │ 53.86 / 55.12 ±1.13 / 56.84 ms │ 12.74 / 13.66 ±0.91 / 15.17 ms │ +4.03x faster │
│ Q11 │ 3.86 / 4.38 ±0.87 / 6.12 ms │ 3.75 / 3.88 ±0.10 / 4.02 ms │ +1.13x faster │
└───────┴────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 247.22ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 139.65ms │
│ Average Time (HEAD) │ 22.47ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 12.70ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 3 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**topk_tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 607.1 MiB |
| Avg memory | 101.2 MiB |
| CPU user | 0.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**topk_tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 232.0 KiB |
| Avg memory | 38.7 KiB |
| CPU user | 0.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720420349
run benchmark topk_tpch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720637135
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.33 / 3.70 ±4.54 / 12.78 ms │ 1.28 / 3.56 ±4.40 / 12.36 ms │ no change │
│ QQuery 1 │ 13.75 / 13.91 ±0.10 / 14.02 ms │ 12.78 / 13.40 ±0.38 / 13.72 ms │ no change │
│ QQuery 2 │ 37.51 / 38.04 ±0.42 / 38.56 ms │ 35.68 / 36.55 ±0.57 / 37.46 ms │ no change │
│ QQuery 3 │ 31.72 / 32.78 ±1.76 / 36.27 ms │ 30.00 / 30.94 ±0.80 / 32.13 ms │ +1.06x faster │
│ QQuery 4 │ 241.06 / 244.12 ±1.74 / 246.50 ms │ 225.84 / 236.90 ±8.58 / 246.61 ms │ no change │
│ QQuery 5 │ 273.65 / 279.66 ±3.39 / 283.13 ms │ 267.85 / 286.82 ±18.06 / 318.94 ms │ no change │
│ QQuery 6 │ 1.23 / 1.38 ±0.21 / 1.80 ms │ 1.25 / 1.39 ±0.21 / 1.81 ms │ no change │
│ QQuery 7 │ 14.32 / 14.70 ±0.53 / 15.75 ms │ 13.83 / 13.89 ±0.07 / 14.01 ms │ +1.06x faster │
│ QQuery 8 │ 324.30 / 332.09 ±10.62 / 352.92 ms │ 325.75 / 352.61 ±13.59 / 363.18 ms │ 1.06x slower │
│ QQuery 9 │ 465.92 / 487.53 ±20.05 / 519.15 ms │ 454.61 / 483.60 ±29.96 / 524.30 ms │ no change │
│ QQuery 10 │ 70.44 / 72.47 ±1.28 / 74.25 ms │ 72.90 / 74.84 ±1.49 / 76.68 ms │ no change │
│ QQuery 11 │ 81.58 / 83.53 ±2.22 / 87.66 ms │ 84.45 / 85.37 ±0.98 / 86.98 ms │ no change │
│ QQuery 12 │ 268.23 / 272.30 ±3.69 / 276.60 ms │ 265.51 / 288.01 ±14.04 / 309.67 ms │ 1.06x slower │
│ QQuery 13 │ 365.42 / 376.31 ±11.17 / 397.11 ms │ 401.82 / 418.72 ±13.96 / 439.14 ms │ 1.11x slower │
│ QQuery 14 │ 279.33 / 286.47 ±5.06 / 294.69 ms │ 300.76 / 308.04 ±3.99 / 312.63 ms │ 1.08x slower │
│ QQuery 15 │ 267.01 / 276.92 ±5.14 / 280.54 ms │ 269.98 / 301.21 ±26.06 / 346.03 ms │ 1.09x slower │
│ QQuery 16 │ 603.47 / 624.95 ±18.27 / 647.49 ms │ 612.58 / 626.17 ±10.11 / 638.37 ms │ no change │
│ QQuery 17 │ 606.67 / 621.75 ±11.32 / 636.40 ms │ 613.64 / 655.11 ±23.80 / 682.19 ms │ 1.05x slower │
│ QQuery 18 │ 1247.45 / 1284.49 ±29.95 / 1323.16 ms │ 1256.16 / 1270.53 ±9.45 / 1283.13 ms │ no change │
│ QQuery 19 │ 29.27 / 31.94 ±4.96 / 41.86 ms │ 27.68 / 35.43 ±14.77 / 64.96 ms │ 1.11x slower │
│ QQuery 20 │ 513.36 / 524.36 ±11.38 / 546.24 ms │ 514.05 / 535.86 ±20.31 / 572.13 ms │ no change │
│ QQuery 21 │ 517.66 / 526.40 ±7.28 / 536.32 ms │ 521.77 / 526.62 ±3.85 / 532.36 ms │ no change │
│ QQuery 22 │ 989.14 / 1007.10 ±11.37 / 1018.44 ms │ 984.43 / 988.81 ±3.85 / 994.52 ms │ no change │
│ QQuery 23 │ 3213.26 / 3289.95 ±40.02 / 3331.09 ms │ 3098.84 / 3131.07 ±20.45 / 3152.50 ms │ no change │
│ QQuery 24 │ 41.97 / 47.71 ±5.80 / 56.58 ms │ FAIL │ incomparable │
│ QQuery 25 │ 115.07 / 116.35 ±1.16 / 118.51 ms │ 112.55 / 114.07 ±1.40 / 116.54 ms │ no change │
│ QQuery 26 │ 42.01 / 42.91 ±1.24 / 45.33 ms │ FAIL │ incomparable │
│ QQuery 27 │ 684.49 / 693.97 ±5.03 / 698.53 ms │ 669.41 / 671.56 ±1.91 / 673.95 ms │ no change │
│ QQuery 28 │ 3036.81 / 3068.89 ±27.87 / 3113.73 ms │ 3075.02 / 3108.78 ±27.19 / 3138.43 ms │ no change │
│ QQuery 29 │ 40.66 / 46.38 ±7.02 / 57.30 ms │ 41.87 / 46.60 ±8.34 / 63.25 ms │ no change │
│ QQuery 30 │ 296.32 / 312.81 ±12.86 / 334.11 ms │ 316.66 / 319.79 ±2.74 / 324.33 ms │ no change │
│ QQuery 31 │ 293.57 / 303.07 ±11.67 / 325.90 ms │ 289.03 / 309.80 ±18.32 / 342.78 ms │ no change │
│ QQuery 32 │ 942.96 / 973.13 ±22.34 / 1005.06 ms │ 988.74 / 1012.95 ±30.19 / 1070.29 ms │ no change │
│ QQuery 33 │ 1447.84 / 1517.78 ±53.37 / 1607.93 ms │ 1447.38 / 1482.14 ±23.25 / 1519.67 ms │ no change │
│ QQuery 34 │ 1449.56 / 1505.81 ±44.82 / 1563.49 ms │ 1484.86 / 1515.54 ±25.77 / 1558.20 ms │ no change │
│ QQuery 35 │ 284.40 / 308.70 ±29.39 / 366.58 ms │ 272.74 / 319.50 ±67.78 / 454.36 ms │ no change │
│ QQuery 36 │ 63.67 / 69.67 ±4.44 / 76.37 ms │ 67.37 / 68.81 ±1.36 / 71.10 ms │ no change │
│ QQuery 37 │ 35.60 / 38.95 ±4.00 / 44.38 ms │ 35.31 / 38.11 ±2.61 / 41.39 ms │ no change │
│ QQuery 38 │ 39.04 / 42.76 ±2.17 / 45.04 ms │ 42.20 / 48.14 ±5.80 / 57.95 ms │ 1.13x slower │
│ QQuery 39 │ 135.77 / 147.49 ±8.64 / 160.59 ms │ 140.86 / 154.58 ±8.16 / 161.78 ms │ no change │
│ QQuery 40 │ 14.14 / 14.28 ±0.11 / 14.44 ms │ 14.80 / 16.70 ±2.75 / 22.15 ms │ 1.17x slower │
│ QQuery 41 │ 13.55 / 15.21 ±2.85 / 20.90 ms │ 14.68 / 15.76 ±1.91 / 19.58 ms │ no change │
│ QQuery 42 │ 13.47 / 18.19 ±4.91 / 24.85 ms │ 14.00 / 14.21 ±0.20 / 14.57 ms │ +1.28x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19920.30ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 19962.48ms │
│ Average Time (HEAD) │ 485.86ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 486.89ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 9 │
│ Queries with No Change │ 29 │
│ Queries with Failure │ 2 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 11.8 GiB |
| Avg memory | 4.3 GiB |
| CPU user | 1024.4s |
| CPU sys | 70.4s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 9.9 GiB |
| Avg memory | 4.2 GiB |
| CPU user | 1027.1s |
| CPU sys | 71.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515470817
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515459477)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4515459477-277-x4d6g 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (691926f605ec05196e3235f4f1d036ae4d981b8d) to 077f08a (merge-base) [diff](https://github.com/apache/datafusion/compare/077f08a9a6632324c95275dd15b5dd5b1f14006f..691926f605ec05196e3235f4f1d036ae4d981b8d) using: sort_pushdown_inexact
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4539955751
Thanks! I feel this may be simplified even further by https://github.com/apache/arrow-rs/pull/9968?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515299963
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4515274607-274-nhjxx 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (691926f605ec05196e3235f4f1d036ae4d981b8d) to 077f08a (merge-base) [diff](https://github.com/apache/datafusion/compare/077f08a9a6632324c95275dd15b5dd5b1f14006f..691926f605ec05196e3235f4f1d036ae4d981b8d) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4518898867
I like the idea! I had to do something similar in https://github.com/apache/datafusion/pull/22144 (re-evaluate filter decisions every row group). We should look at the APIs and check how they match up. I am also interested in how this intersects with @Dandandan 's work on row group = morsel (morsels would be a good natural boundary to re-evaluate pruning at).
> A new force_per_row_group flag (set by is_dynamic_physical_expr) disables coalescing for dynamic predicates only, so static WHERE queries pay nothing.
Do we pay this price as well for hash join dynamic filters (where the filter won't be changing)? Maybe we should check `!filter.is_complete()` or something...
I am traveling the next couple days but will try to review more in depth soon 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4518886956
> Is this expected?
>
> [#22450 (comment)](https://github.com/apache/datafusion/pull/22450#issuecomment-4515744044)
>
> ```
> Benchmark sort_pushdown_inexact.json
> --------------------
> ┏━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
> ┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
> ┡━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
> │ Q1 │ FAIL │ FAIL │ incomparable │
> │ Q2 │ FAIL │ FAIL │ incomparable │
> │ Q3 │ FAIL │ FAIL │ incomparable │
> │ Q4 │ FAIL │ FAIL │ incomparable │
> └───────┴──────┴────────────────────────────────────┴──────────────┘
> ```
@adriangb I can run locally, i am not sure if it's related to benchmark machine. I can test again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421003080
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,51 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when all three conditions hold:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) the predicate has at least one not-yet-complete dynamic filter
+ // (`DynamicFilterTracking::Watching`) — static or already-complete
+ // predicates were fully consumed by `prune_by_statistics` at file
+ // open, so re-evaluating them per RG boundary would be wasted work,
+ // 3) there is at least one pending RG that could be skipped.
+ // The pruner subscribes once to every still-incomplete dynamic filter
+ // via the `DynamicFilterTracker` watch channel (#22460), so detecting
+ // a threshold change is a single atomic load — not a tree walk per
+ // RG check.
+ let row_group_pruner = match (&prepared.predicate, rg_plan.len() > 1) {
+ (Some(predicate), true)
+ if DynamicFilterTracking::classify(predicate)
+ .contains_dynamic_filter() =>
+ {
+ Some(RowGroupPruner::new(
+ Arc::clone(predicate),
+ Arc::clone(&prepared.physical_file_schema),
+ Arc::clone(reader_metadata.metadata()),
+ prepared.predicate_creation_errors.clone(),
+ prepared.file_metrics.predicate_evaluation_errors.clone(),
+ ))
+ }
+ _ => None,
+ };
+ let row_groups_pruned_dynamic = prepared
+ .file_metrics
+ .row_groups_pruned_dynamic_filter
+ .clone();
+
let stream = PushDecoderStreamState {
- decoder,
- pending_decoders,
- remaining_limit,
+ decoder: Some(decoder),
+ active_reader: None,
+ rg_plan,
+ remaining_limit: None,
Review Comment:
Good catch, addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4725618235
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4725518286)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.18 / 3.33 ±4.22 / 11.77 ms │ 1.21 / 3.40 ±4.30 / 11.99 ms │ no change │
│ QQuery 1 │ 12.67 / 13.12 ±0.24 / 13.37 ms │ 12.52 / 12.85 ±0.18 / 13.03 ms │ no change │
│ QQuery 2 │ 35.85 / 36.12 ±0.21 / 36.40 ms │ 35.92 / 36.17 ±0.30 / 36.75 ms │ no change │
│ QQuery 3 │ 30.08 / 30.90 ±0.88 / 32.23 ms │ 30.30 / 30.88 ±0.82 / 32.47 ms │ no change │
│ QQuery 4 │ 218.97 / 225.05 ±3.60 / 229.28 ms │ 221.60 / 225.46 ±2.31 / 228.78 ms │ no change │
│ QQuery 5 │ 268.66 / 271.68 ±1.65 / 273.05 ms │ 267.63 / 272.06 ±3.29 / 277.71 ms │ no change │
│ QQuery 6 │ 1.23 / 1.38 ±0.23 / 1.82 ms │ 1.25 / 1.39 ±0.22 / 1.82 ms │ no change │
│ QQuery 7 │ 14.37 / 14.50 ±0.10 / 14.67 ms │ 13.62 / 13.70 ±0.09 / 13.86 ms │ +1.06x faster │
│ QQuery 8 │ 321.81 / 324.56 ±1.98 / 327.43 ms │ 321.72 / 324.38 ±1.47 / 326.02 ms │ no change │
│ QQuery 9 │ 453.20 / 461.77 ±7.89 / 472.07 ms │ 448.10 / 455.97 ±7.21 / 468.60 ms │ no change │
│ QQuery 10 │ 70.01 / 70.90 ±0.93 / 72.61 ms │ 68.93 / 70.08 ±1.11 / 71.54 ms │ no change │
│ QQuery 11 │ 80.57 / 82.01 ±0.92 / 83.35 ms │ 80.14 / 81.12 ±0.80 / 82.00 ms │ no change │
│ QQuery 12 │ 266.64 / 270.40 ±3.37 / 275.31 ms │ 264.59 / 269.35 ±3.59 / 273.81 ms │ no change │
│ QQuery 13 │ 359.72 / 369.04 ±5.60 / 375.09 ms │ 358.32 / 374.82 ±17.38 / 407.83 ms │ no change │
│ QQuery 14 │ 283.76 / 288.39 ±4.08 / 295.51 ms │ 281.21 / 286.81 ±5.03 / 296.20 ms │ no change │
│ QQuery 15 │ 272.89 / 282.02 ±6.45 / 291.68 ms │ 266.06 / 271.06 ±5.72 / 281.87 ms │ no change │
│ QQuery 16 │ 612.87 / 622.91 ±14.15 / 650.98 ms │ 605.35 / 618.32 ±9.43 / 632.67 ms │ no change │
│ QQuery 17 │ 621.61 / 629.35 ±4.35 / 633.73 ms │ 618.49 / 628.15 ±15.98 / 660.04 ms │ no change │
│ QQuery 18 │ 1255.37 / 1275.37 ±15.36 / 1301.12 ms │ 1240.33 / 1276.90 ±29.39 / 1329.33 ms │ no change │
│ QQuery 19 │ 28.11 / 28.33 ±0.23 / 28.73 ms │ 27.62 / 28.95 ±2.09 / 33.13 ms │ no change │
│ QQuery 20 │ 515.88 / 528.41 ±12.31 / 545.66 ms │ 518.99 / 528.90 ±6.37 / 537.95 ms │ no change │
│ QQuery 21 │ 516.73 / 521.59 ±6.33 / 533.89 ms │ 524.11 / 535.73 ±7.26 / 544.04 ms │ no change │
│ QQuery 22 │ 982.07 / 991.12 ±7.19 / 1003.13 ms │ 994.90 / 1008.56 ±8.86 / 1022.40 ms │ no change │
│ QQuery 23 │ 3176.23 / 3202.05 ±19.86 / 3227.01 ms │ 3092.23 / 3112.29 ±18.68 / 3136.49 ms │ no change │
│ QQuery 24 │ 40.89 / 44.26 ±6.13 / 56.50 ms │ 40.77 / 42.13 ±1.73 / 45.49 ms │ no change │
│ QQuery 25 │ 111.18 / 117.20 ±7.00 / 129.61 ms │ 110.28 / 111.22 ±0.96 / 112.80 ms │ +1.05x faster │
│ QQuery 26 │ 41.35 / 42.48 ±1.06 / 44.19 ms │ 41.21 / 45.54 ±4.17 / 52.60 ms │ 1.07x slower │
│ QQuery 27 │ 669.76 / 674.83 ±3.45 / 678.92 ms │ 672.23 / 681.12 ±8.22 / 695.36 ms │ no change │
│ QQuery 28 │ 3024.02 / 3040.31 ±8.74 / 3048.95 ms │ 3002.37 / 3040.75 ±24.78 / 3079.61 ms │ no change │
│ QQuery 29 │ 40.42 / 54.39 ±16.76 / 77.40 ms │ 40.33 / 41.10 ±0.53 / 41.85 ms │ +1.32x faster │
│ QQuery 30 │ 301.31 / 306.47 ±4.27 / 311.72 ms │ 293.22 / 306.55 ±7.58 / 313.20 ms │ no change │
│ QQuery 31 │ 290.61 / 295.37 ±5.06 / 305.04 ms │ 278.15 / 287.97 ±7.57 / 300.62 ms │ no change │
│ QQuery 32 │ 954.35 / 981.75 ±23.39 / 1022.63 ms │ 937.50 / 948.73 ±12.92 / 971.62 ms │ no change │
│ QQuery 33 │ 1440.34 / 1478.84 ±32.21 / 1523.17 ms │ 1447.23 / 1473.63 ±21.57 / 1512.90 ms │ no change │
│ QQuery 34 │ 1473.82 / 1512.34 ±33.42 / 1552.74 ms │ 1452.38 / 1489.44 ±21.21 / 1516.71 ms │ no change │
│ QQuery 35 │ 278.84 / 316.74 ±42.33 / 369.59 ms │ 274.24 / 306.81 ±58.80 / 424.28 ms │ no change │
│ QQuery 36 │ 65.74 / 69.17 ±4.10 / 76.45 ms │ 70.18 / 74.76 ±4.07 / 81.35 ms │ 1.08x slower │
│ QQuery 37 │ 35.34 / 36.07 ±0.71 / 37.42 ms │ 35.61 / 38.28 ±4.18 / 46.61 ms │ 1.06x slower │
│ QQuery 38 │ 45.16 / 49.76 ±2.74 / 52.86 ms │ 39.83 / 43.91 ±4.31 / 52.25 ms │ +1.13x faster │
│ QQuery 39 │ 145.60 / 150.23 ±4.59 / 158.26 ms │ 144.34 / 148.48 ±2.85 / 151.93 ms │ no change │
│ QQuery 40 │ 14.20 / 15.18 ±1.22 / 17.39 ms │ 13.63 / 14.33 ±0.61 / 15.44 ms │ +1.06x faster │
│ QQuery 41 │ 13.59 / 16.36 ±4.27 / 24.85 ms │ 13.42 / 14.84 ±2.51 / 19.85 ms │ +1.10x faster │
│ QQuery 42 │ 12.93 / 16.79 ±6.87 / 30.50 ms │ 12.69 / 12.95 ±0.17 / 13.20 ms │ +1.30x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19762.87ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 19589.85ms │
│ Average Time (HEAD) │ 459.60ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 455.58ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 33 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 9.5 GiB |
| Avg memory | 4.1 GiB |
| CPU user | 1008.9s |
| CPU sys | 71.9s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 11.8 GiB |
| Avg memory | 4.6 GiB |
| CPU user | 1004.5s |
| CPU sys | 71.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3411801814
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,40 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when both halves are useful:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) there is at least one pending run that could be skipped.
+ // The pruner consults the predicate's `snapshot_generation` so the
+ // cost is one rebuild per dynamic-filter update, not per RG check.
Review Comment:
Addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4725515987
> @zhuqi-lucas these failures in clickbench are worth looking at:
>
> │ QQuery 24 │ 41.97 / 47.71 ±5.80 / 56.58 ms │ FAIL │ incomparable │
Thanks @adriangb for good catch, and fixed in latest PR, let me rerun it to see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765160631
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 38.02 / 38.72 ±0.95 / 40.58 ms │ 37.97 / 39.15 ±1.14 / 40.95 ms │ no change │
│ QQuery 2 │ 19.12 / 19.32 ±0.24 / 19.78 ms │ 18.92 / 19.46 ±0.49 / 20.16 ms │ no change │
│ QQuery 3 │ 31.30 / 32.54 ±0.88 / 33.39 ms │ 31.64 / 33.18 ±0.84 / 34.14 ms │ no change │
│ QQuery 4 │ 17.46 / 17.58 ±0.08 / 17.67 ms │ 17.28 / 17.36 ±0.07 / 17.46 ms │ no change │
│ QQuery 5 │ 38.42 / 39.99 ±0.91 / 41.19 ms │ 37.71 / 40.47 ±2.25 / 43.03 ms │ no change │
│ QQuery 6 │ 16.14 / 16.26 ±0.09 / 16.38 ms │ 15.98 / 16.16 ±0.15 / 16.38 ms │ no change │
│ QQuery 7 │ 44.54 / 45.12 ±0.38 / 45.65 ms │ 44.36 / 45.89 ±1.19 / 47.20 ms │ no change │
│ QQuery 8 │ 42.88 / 43.07 ±0.12 / 43.23 ms │ 42.93 / 43.15 ±0.18 / 43.45 ms │ no change │
│ QQuery 9 │ 49.41 / 50.18 ±0.63 / 50.97 ms │ 49.26 / 50.61 ±0.78 / 51.31 ms │ no change │
│ QQuery 10 │ 42.10 / 43.37 ±1.29 / 44.96 ms │ 42.42 / 42.50 ±0.11 / 42.71 ms │ no change │
│ QQuery 11 │ 13.31 / 13.48 ±0.13 / 13.67 ms │ 13.22 / 13.55 ±0.17 / 13.69 ms │ no change │
│ QQuery 12 │ 23.89 / 25.20 ±1.39 / 27.21 ms │ 24.04 / 24.24 ±0.19 / 24.56 ms │ no change │
│ QQuery 13 │ 33.16 / 34.68 ±1.14 / 36.41 ms │ 32.14 / 33.04 ±1.29 / 35.59 ms │ no change │
│ QQuery 14 │ 23.37 / 24.05 ±0.76 / 25.51 ms │ 23.63 / 24.27 ±1.08 / 26.43 ms │ no change │
│ QQuery 15 │ 31.00 / 31.15 ±0.14 / 31.33 ms │ 30.95 / 31.25 ±0.15 / 31.36 ms │ no change │
│ QQuery 16 │ 14.15 / 14.42 ±0.27 / 14.82 ms │ 14.52 / 14.72 ±0.11 / 14.81 ms │ no change │
│ QQuery 17 │ 73.16 / 74.45 ±1.22 / 76.42 ms │ 72.47 / 73.10 ±0.56 / 73.95 ms │ no change │
│ QQuery 18 │ 58.38 / 60.36 ±1.28 / 61.78 ms │ 58.24 / 59.91 ±1.35 / 62.34 ms │ no change │
│ QQuery 19 │ 33.08 / 33.66 ±0.41 / 34.14 ms │ 33.10 / 33.97 ±0.67 / 34.76 ms │ no change │
│ QQuery 20 │ 31.80 / 32.06 ±0.19 / 32.37 ms │ 31.66 / 31.87 ±0.27 / 32.40 ms │ no change │
│ QQuery 21 │ 54.14 / 55.87 ±1.50 / 57.78 ms │ 54.64 / 56.06 ±0.94 / 57.50 ms │ no change │
│ QQuery 22 │ 13.58 / 14.10 ±0.62 / 15.28 ms │ 13.83 / 13.97 ±0.10 / 14.10 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 759.60ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 757.86ms │
│ Average Time (HEAD) │ 34.53ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 34.45ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 540.4 MiB |
| CPU user | 21.9s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 511.0 MiB |
| CPU user | 22.0s |
| CPU sys | 1.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720420349)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark run_topk_tpch.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 2.41 / 3.36 ±0.83 / 4.66 ms │ 2.29 / 2.84 ±0.89 / 4.61 ms │ +1.19x faster │
│ Q2 │ 10.85 / 11.44 ±0.58 / 12.55 ms │ 3.01 / 3.13 ±0.15 / 3.41 ms │ +3.66x faster │
│ Q3 │ 33.31 / 34.39 ±1.23 / 36.59 ms │ 33.42 / 34.18 ±0.86 / 35.80 ms │ no change │
│ Q4 │ 12.30 / 12.73 ±0.64 / 14.01 ms │ 3.44 / 3.52 ±0.07 / 3.65 ms │ +3.62x faster │
│ Q5 │ 9.86 / 10.13 ±0.18 / 10.41 ms │ 10.44 / 10.83 ±0.41 / 11.49 ms │ 1.07x slower │
│ Q6 │ 17.28 / 17.62 ±0.31 / 18.05 ms │ 17.88 / 18.48 ±0.95 / 20.37 ms │ no change │
│ Q7 │ 38.59 / 39.33 ±0.50 / 40.01 ms │ 38.89 / 39.62 ±0.65 / 40.75 ms │ no change │
│ Q8 │ 28.88 / 30.41 ±1.41 / 32.76 ms │ 7.18 / 7.27 ±0.11 / 7.48 ms │ +4.18x faster │
│ Q9 │ 36.03 / 36.53 ±0.46 / 37.18 ms │ 8.88 / 8.95 ±0.06 / 9.06 ms │ +4.08x faster │
│ Q10 │ 56.65 / 57.04 ±0.34 / 57.65 ms │ 13.56 / 14.30 ±1.11 / 16.50 ms │ +3.99x faster │
│ Q11 │ 4.03 / 4.45 ±0.72 / 5.89 ms │ 4.31 / 4.47 ±0.27 / 5.00 ms │ no change │
└───────┴────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 257.43ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 147.59ms │
│ Average Time (HEAD) │ 23.40ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 13.42ms │
│ Queries Faster │ 6 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 4 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**topk_tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 587.4 MiB |
| Avg memory | 97.9 MiB |
| CPU user | 0.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**topk_tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 236.0 KiB |
| Avg memory | 39.3 KiB |
| CPU user | 0.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421007923
##########
datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt:
##########
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# End-to-end SLT for **dynamic row-group pruning** driven by a TopK
+# `SortExec`'s `DynamicFilterPhysicalExpr`.
+#
+# Builds a 5-row-group parquet file with disjoint per-RG ranges of `v`:
+# RG 0: 0..3, RG 1: 3..6, RG 2: 6..9, RG 3: 9..12, RG 4: 12..15
+# `ORDER BY v DESC LIMIT 3` fills the TopK heap from the row group with
+# the largest values; the tightened threshold then proves every other
+# row group unreachable. The opener splits decoder runs per row group
Review Comment:
Removed in latest PR, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421005710
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,51 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when all three conditions hold:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) the predicate has at least one not-yet-complete dynamic filter
+ // (`DynamicFilterTracking::Watching`) — static or already-complete
+ // predicates were fully consumed by `prune_by_statistics` at file
+ // open, so re-evaluating them per RG boundary would be wasted work,
+ // 3) there is at least one pending RG that could be skipped.
+ // The pruner subscribes once to every still-incomplete dynamic filter
+ // via the `DynamicFilterTracker` watch channel (#22460), so detecting
+ // a threshold change is a single atomic load — not a tree walk per
+ // RG check.
+ let row_group_pruner = match (&prepared.predicate, rg_plan.len() > 1) {
+ (Some(predicate), true)
+ if DynamicFilterTracking::classify(predicate)
+ .contains_dynamic_filter() =>
Review Comment:
Good suggestion, addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515573016
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515459477)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark sort_pushdown_inexact.json
--------------------
┏━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ Q1 │ FAIL │ FAIL │ incomparable │
│ Q2 │ FAIL │ FAIL │ incomparable │
│ Q3 │ FAIL │ FAIL │ incomparable │
│ Q4 │ FAIL │ FAIL │ incomparable │
└───────┴──────┴────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD) │ 0.00ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 0.00ms │
│ Average Time (HEAD) │ 0.00ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 0.00ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 0 │
│ Queries with Failure │ 4 │
└───────────────────────────────────────────────────┴────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sort_pushdown_inexact — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 3.7 GiB |
| Avg memory | 3.7 GiB |
| CPU user | 0.2s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**sort_pushdown_inexact — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 3.7 GiB |
| Avg memory | 3.7 GiB |
| CPU user | 0.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515299998
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4515274607-275-nknmm 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (691926f605ec05196e3235f4f1d036ae4d981b8d) to 077f08a (merge-base) [diff](https://github.com/apache/datafusion/compare/077f08a9a6632324c95275dd15b5dd5b1f14006f..691926f605ec05196e3235f4f1d036ae4d981b8d) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515872406
run benchmark topk_tpch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515989448
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515872406)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark run_topk_tpch.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 2.14 / 2.74 ±0.76 / 4.10 ms │ 2.12 / 2.79 ±0.68 / 4.02 ms │ no change │
│ Q2 │ 10.66 / 11.36 ±0.68 / 12.23 ms │ 2.81 / 3.61 ±0.87 / 4.72 ms │ +3.15x faster │
│ Q3 │ 31.77 / 32.15 ±0.43 / 32.83 ms │ 31.71 / 31.92 ±0.16 / 32.18 ms │ no change │
│ Q4 │ 11.83 / 12.29 ±0.77 / 13.82 ms │ 3.13 / 3.25 ±0.13 / 3.48 ms │ +3.78x faster │
│ Q5 │ 9.94 / 10.14 ±0.18 / 10.46 ms │ 9.95 / 10.02 ±0.05 / 10.09 ms │ no change │
│ Q6 │ 17.19 / 17.39 ±0.15 / 17.56 ms │ 17.11 / 17.36 ±0.37 / 18.09 ms │ no change │
│ Q7 │ 37.07 / 38.08 ±1.17 / 40.08 ms │ 37.00 / 37.41 ±0.37 / 38.07 ms │ no change │
│ Q8 │ 28.13 / 28.59 ±0.60 / 29.71 ms │ 6.86 / 7.16 ±0.42 / 7.98 ms │ +3.99x faster │
│ Q9 │ 35.34 / 36.86 ±1.54 / 38.77 ms │ 8.36 / 8.50 ±0.08 / 8.60 ms │ +4.34x faster │
│ Q10 │ 54.13 / 55.29 ±1.83 / 58.93 ms │ 12.77 / 13.00 ±0.45 / 13.89 ms │ +4.25x faster │
│ Q11 │ 3.75 / 3.91 ±0.11 / 4.05 ms │ 3.82 / 4.08 ±0.31 / 4.68 ms │ no change │
└───────┴────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 248.79ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 139.08ms │
│ Average Time (HEAD) │ 22.62ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 12.64ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**topk_tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 4.9 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 11.4s |
| CPU sys | 1.1s |
| Peak spill | 0 B |
**topk_tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 4.4 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 6.5s |
| CPU sys | 0.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4702243897
Makes sense thanks! Could you do a pass of updating comments + PR description and maybe restacking the commits for easier review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765072359
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4765060057-602-459tz 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (23339c539055c69a5157f8b00de7581c862810b7) to ba67bb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/ba67bb467a30c3dbe7f78089451ea4c4d1a191d5..23339c539055c69a5157f8b00de7581c862810b7) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515900480
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515872406)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4515872406-280-dfq5d 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (0828f1b5a97d26e527cb33b0efa840113e266290) to a8f03fd (merge-base) [diff](https://github.com/apache/datafusion/compare/a8f03fdae5b27465a6e4389017c43cf06b4c59ac..0828f1b5a97d26e527cb33b0efa840113e266290) using: topk_tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4719660563
Update on this — I prototyped the per-RG toggle: push `fully_matched` through `PreparedAccessPlan` into the per-RG `rg_plan` entry, then at each row-group boundary call `decoder.into_builder().with_row_filter(empty_or_real).build()` to flip the filter. Limit budget carries across the rebuild correctly. But it exposed a latent invariant in `push_decoder.rs` worth flagging.
The current `rg_plan` assumes a 1:1 correspondence between `try_next_reader()` results and the head of the plan — we `pop_front()` once per reader. That holds when every RG in the plan actually yields a reader. It does **not** hold when page-index pruning eliminates every page of an RG: arrow-rs's `try_next_reader` silently advances past the empty RG and returns the reader for the next one. Concretely, on `limit_pruning.slt`'s `species > 'M' AND s >= 50 ORDER BY species LIMIT 3`:
- Initial plan `[RG1 (not fm), RG2 (fm), RG3 (not fm)]`, filter installed.
- arrow-rs page-index-prunes RG 1 entirely (`page_index_pages_pruned=2`).
- `try_next_reader` returns the reader for **RG 2**; my code pops RG 1, leaving plan `[RG2, RG3]`.
- Next boundary sees `front = RG2 (fm)`, triggers a toggle to empty filter, and rebuilds with `with_row_groups([2, 3])` — the new decoder reads RG 2 **again**.
- Scan output: 6 rows instead of 3 (final query result still correct because TopK ranks them, but the metric is off).
Without the toggle, this off-by-one was harmless — we never used `rg_plan.front()` to redirect the decoder, so the wrong-head condition was invisible. The toggle made it observable.
Fixing it cleanly needs one of:
1. An arrow-rs hook to query the decoder's *actual* next-RG index after `try_next_reader` (doesn't exist today).
2. Pre-computing which RGs page-index will fully eliminate at file open and dropping them from `rg_plan` (replicates arrow-rs internals — fragile).
3. Tracking position by RG metadata identity rather than queue order.
I'm leaving the per-RG toggle out of this PR and will pick it up as a follow-up. The practical perf gap is narrow: arrow-rs's page-level \`page_index_pages_skipped_by_fully_matched\` already skips per-row filter eval on fully-matched RGs whenever the page index is populated — i.e. for any RG large enough to span multiple data pages, which is the typical production layout. The case this PR doesn't (yet) recover is small RGs where no page index helps, plus predicate columns that the page index doesn't cover. Happy to file an issue and tackle it after this lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720418223
> Addressed all comments besides:
>
> > split_runs used to build a different decoder for each row group when there was row groups that fully matched filters, allowing those to skip applying the row filter. I think we're loosing that now?
>
> I tried this before, but it failed for testing, i am trying more for this part.
makes sense, but doesn't that mean this PR is possibly a regression for some queries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720454104
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720420349)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4720420349-576-ttwzc 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (6c554cce66f8bf6126f821bb8822f740d24287cc) to 6176a6d (merge-base) [diff](https://github.com/apache/datafusion/compare/6176a6dae1f36e30c83bc880bb8b6cb1d1a175a8..6c554cce66f8bf6126f821bb8822f740d24287cc) using: topk_tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720452009
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4720419417-574-pb9fs 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (6c554cce66f8bf6126f821bb8822f740d24287cc) to 6176a6d (merge-base) [diff](https://github.com/apache/datafusion/compare/6176a6dae1f36e30c83bc880bb8b6cb1d1a175a8..6c554cce66f8bf6126f821bb8822f740d24287cc) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720651242
@zhuqi-lucas these failures in clickbench are worth looking at:
│ QQuery 24 │ 41.97 / 47.71 ±5.80 / 56.58 ms │ FAIL │ incomparable │
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515459477
run benchmark sort_pushdown_inexact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4519217985
Thanks @adriangb! On the `is_complete` gate — you're right. For hash-join filters that are already complete at scan-startup, the static `PruningPredicate` at file-open has used the final value, and per-RG splitting buys nothing after that. `Inner::is_complete` is exposed; I'll tighten the gate to `is_dynamic_physical_expr(p) && !is_filter_complete(p)`, which keeps TopK and concurrent hash-join (build finishing after probe starts) on the splitting path, and lets already-complete filters fall through. Pushing in a moment.
On the #22144 overlap and morsel intersection — agreed on both; will trace #22144 for a shared abstraction in a follow-up, and morsels would be the natural cross-file extension point. Have a good trip :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515464016
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 5.69 / 6.30 ±0.88 / 8.05 ms │ 5.78 / 6.31 ±0.95 / 8.21 ms │ no change │
│ QQuery 2 │ 80.75 / 81.16 ±0.31 / 81.58 ms │ 80.16 / 80.81 ±0.40 / 81.25 ms │ no change │
│ QQuery 3 │ 29.38 / 29.57 ±0.13 / 29.71 ms │ 28.91 / 29.15 ±0.16 / 29.38 ms │ no change │
│ QQuery 4 │ 512.73 / 515.37 ±2.82 / 520.75 ms │ 509.87 / 514.41 ±2.49 / 516.87 ms │ no change │
│ QQuery 5 │ 50.96 / 51.70 ±0.61 / 52.66 ms │ 50.75 / 51.13 ±0.44 / 51.94 ms │ no change │
│ QQuery 6 │ 35.28 / 35.77 ±0.38 / 36.31 ms │ 35.30 / 35.86 ±0.37 / 36.34 ms │ no change │
│ QQuery 7 │ 109.30 / 110.73 ±1.72 / 112.86 ms │ 108.59 / 109.56 ±0.94 / 110.99 ms │ no change │
│ QQuery 8 │ 36.33 / 36.65 ±0.31 / 37.03 ms │ 36.32 / 37.02 ±0.53 / 37.95 ms │ no change │
│ QQuery 9 │ 53.25 / 55.51 ±2.05 / 58.81 ms │ 53.15 / 54.23 ±0.72 / 54.96 ms │ no change │
│ QQuery 10 │ 80.67 / 82.08 ±1.67 / 85.32 ms │ 81.02 / 81.34 ±0.21 / 81.55 ms │ no change │
│ QQuery 11 │ 314.18 / 317.53 ±4.18 / 325.76 ms │ 310.62 / 316.44 ±4.99 / 322.48 ms │ no change │
│ QQuery 12 │ 28.43 / 28.63 ±0.20 / 28.94 ms │ 28.49 / 28.69 ±0.17 / 28.94 ms │ no change │
│ QQuery 13 │ 125.87 / 126.80 ±1.06 / 128.82 ms │ 126.23 / 126.71 ±0.54 / 127.68 ms │ no change │
│ QQuery 14 │ 502.60 / 505.33 ±1.50 / 506.71 ms │ 502.81 / 505.10 ±1.82 / 507.78 ms │ no change │
│ QQuery 15 │ 60.38 / 61.69 ±1.71 / 65.03 ms │ 60.49 / 61.18 ±0.66 / 62.17 ms │ no change │
│ QQuery 16 │ 6.30 / 6.52 ±0.26 / 7.03 ms │ 6.34 / 6.51 ±0.16 / 6.81 ms │ no change │
│ QQuery 17 │ 80.24 / 81.16 ±0.67 / 82.22 ms │ 80.30 / 80.98 ±0.58 / 81.73 ms │ no change │
│ QQuery 18 │ 152.37 / 153.04 ±0.40 / 153.48 ms │ 151.31 / 152.10 ±0.68 / 153.13 ms │ no change │
│ QQuery 19 │ 40.79 / 41.20 ±0.25 / 41.56 ms │ 41.17 / 41.46 ±0.21 / 41.79 ms │ no change │
│ QQuery 20 │ 34.92 / 35.59 ±0.65 / 36.69 ms │ 35.35 / 35.91 ±0.30 / 36.17 ms │ no change │
│ QQuery 21 │ 16.69 / 16.98 ±0.24 / 17.33 ms │ 16.78 / 17.01 ±0.23 / 17.45 ms │ no change │
│ QQuery 22 │ 61.65 / 62.39 ±0.46 / 63.01 ms │ 61.77 / 62.59 ±1.17 / 64.87 ms │ no change │
│ QQuery 23 │ 480.28 / 482.80 ±1.89 / 485.52 ms │ 479.56 / 482.95 ±3.05 / 487.66 ms │ no change │
│ QQuery 24 │ 236.03 / 239.91 ±6.29 / 252.43 ms │ 233.29 / 235.80 ±2.41 / 239.92 ms │ no change │
│ QQuery 25 │ 114.42 / 114.91 ±0.67 / 116.14 ms │ 112.30 / 114.90 ±1.52 / 116.61 ms │ no change │
│ QQuery 26 │ 70.92 / 71.14 ±0.34 / 71.82 ms │ 69.91 / 70.46 ±0.30 / 70.78 ms │ no change │
│ QQuery 27 │ 6.42 / 6.56 ±0.16 / 6.87 ms │ 6.47 / 6.64 ±0.22 / 7.08 ms │ no change │
│ QQuery 28 │ 57.25 / 60.78 ±1.85 / 62.74 ms │ 57.93 / 61.09 ±1.61 / 62.34 ms │ no change │
│ QQuery 29 │ 98.46 / 100.38 ±2.61 / 105.53 ms │ 98.98 / 101.66 ±3.64 / 108.85 ms │ no change │
│ QQuery 30 │ 30.12 / 30.48 ±0.31 / 31.00 ms │ 29.94 / 30.30 ±0.30 / 30.83 ms │ no change │
│ QQuery 31 │ 111.59 / 113.79 ±2.44 / 118.38 ms │ 111.55 / 112.71 ±1.71 / 116.05 ms │ no change │
│ QQuery 32 │ 20.35 / 20.93 ±0.34 / 21.38 ms │ 20.17 / 20.46 ±0.27 / 20.79 ms │ no change │
│ QQuery 33 │ 38.68 / 39.14 ±0.35 / 39.58 ms │ 38.31 / 38.57 ±0.20 / 38.80 ms │ no change │
│ QQuery 34 │ 9.29 / 9.58 ±0.29 / 9.98 ms │ 9.20 / 9.57 ±0.36 / 10.20 ms │ no change │
│ QQuery 35 │ 80.49 / 81.05 ±0.48 / 81.93 ms │ 80.76 / 81.51 ±0.49 / 82.23 ms │ no change │
│ QQuery 36 │ 5.75 / 5.91 ±0.17 / 6.25 ms │ 5.89 / 6.01 ±0.15 / 6.30 ms │ no change │
│ QQuery 37 │ 6.78 / 6.95 ±0.11 / 7.06 ms │ 6.72 / 6.94 ±0.27 / 7.42 ms │ no change │
│ QQuery 38 │ 68.29 / 69.85 ±1.23 / 71.85 ms │ 68.57 / 69.09 ±0.38 / 69.67 ms │ no change │
│ QQuery 39 │ 98.14 / 98.47 ±0.32 / 99.05 ms │ 97.99 / 98.65 ±0.51 / 99.53 ms │ no change │
│ QQuery 40 │ 22.81 / 23.33 ±0.79 / 24.91 ms │ 23.09 / 23.27 ±0.15 / 23.51 ms │ no change │
│ QQuery 41 │ 11.03 / 11.79 ±1.11 / 13.97 ms │ 11.24 / 11.71 ±0.44 / 12.29 ms │ no change │
│ QQuery 42 │ 23.77 / 24.22 ±0.29 / 24.69 ms │ 24.09 / 24.42 ±0.37 / 25.05 ms │ no change │
│ QQuery 43 │ 4.64 / 4.76 ±0.17 / 5.09 ms │ 4.80 / 4.91 ±0.17 / 5.24 ms │ no change │
│ QQuery 44 │ 10.51 / 10.58 ±0.07 / 10.71 ms │ 10.60 / 10.85 ±0.17 / 11.09 ms │ no change │
│ QQuery 45 │ 39.70 / 40.69 ±0.70 / 41.49 ms │ 40.41 / 40.86 ±0.33 / 41.33 ms │ no change │
│ QQuery 46 │ 12.83 / 13.15 ±0.27 / 13.55 ms │ 12.69 / 12.87 ±0.16 / 13.12 ms │ no change │
│ QQuery 47 │ 229.18 / 232.49 ±2.73 / 236.09 ms │ 228.68 / 231.85 ±1.91 / 233.47 ms │ no change │
│ QQuery 48 │ 102.74 / 103.73 ±0.80 / 104.88 ms │ 103.30 / 103.95 ±0.91 / 105.69 ms │ no change │
│ QQuery 49 │ 78.52 / 79.33 ±0.62 / 80.41 ms │ 78.96 / 79.36 ±0.45 / 80.18 ms │ no change │
│ QQuery 50 │ 59.63 / 60.33 ±0.37 / 60.68 ms │ 59.40 / 59.73 ±0.22 / 60.07 ms │ no change │
│ QQuery 51 │ 92.53 / 95.36 ±1.81 / 97.37 ms │ 92.09 / 95.60 ±4.02 / 102.85 ms │ no change │
│ QQuery 52 │ 23.93 / 24.34 ±0.37 / 25.02 ms │ 23.81 / 24.28 ±0.38 / 24.74 ms │ no change │
│ QQuery 53 │ 29.46 / 29.71 ±0.16 / 29.96 ms │ 29.04 / 29.30 ±0.20 / 29.57 ms │ no change │
│ QQuery 54 │ 53.72 / 54.35 ±0.36 / 54.67 ms │ 54.44 / 55.05 ±0.44 / 55.75 ms │ no change │
│ QQuery 55 │ 23.52 / 24.34 ±1.07 / 26.44 ms │ 23.21 / 23.56 ±0.24 / 23.91 ms │ no change │
│ QQuery 56 │ 39.01 / 39.29 ±0.26 / 39.78 ms │ 38.43 / 38.84 ±0.28 / 39.20 ms │ no change │
│ QQuery 57 │ 178.37 / 180.09 ±1.83 / 183.56 ms │ 175.76 / 176.93 ±1.12 / 178.61 ms │ no change │
│ QQuery 58 │ 118.44 / 118.92 ±0.37 / 119.44 ms │ 117.20 / 117.99 ±0.54 / 118.58 ms │ no change │
│ QQuery 59 │ 117.61 / 119.84 ±2.42 / 123.96 ms │ 117.87 / 118.49 ±0.72 / 119.91 ms │ no change │
│ QQuery 60 │ 39.05 / 39.84 ±0.52 / 40.49 ms │ 38.74 / 39.22 ±0.42 / 39.82 ms │ no change │
│ QQuery 61 │ 12.52 / 12.63 ±0.10 / 12.81 ms │ 12.61 / 12.79 ±0.24 / 13.25 ms │ no change │
│ QQuery 62 │ 46.89 / 47.53 ±0.38 / 47.98 ms │ 46.32 / 46.78 ±0.48 / 47.68 ms │ no change │
│ QQuery 63 │ 29.70 / 31.05 ±2.04 / 35.10 ms │ 29.42 / 29.66 ±0.25 / 30.09 ms │ no change │
│ QQuery 64 │ 462.94 / 468.99 ±7.22 / 482.49 ms │ 458.19 / 462.97 ±5.14 / 471.10 ms │ no change │
│ QQuery 65 │ 146.94 / 150.20 ±2.58 / 152.60 ms │ 149.10 / 151.28 ±1.81 / 153.79 ms │ no change │
│ QQuery 66 │ 78.92 / 81.62 ±3.98 / 89.53 ms │ 78.42 / 80.72 ±2.20 / 84.79 ms │ no change │
│ QQuery 67 │ 245.43 / 251.17 ±5.20 / 259.23 ms │ 248.87 / 251.06 ±2.35 / 255.17 ms │ no change │
│ QQuery 68 │ 12.91 / 13.11 ±0.22 / 13.53 ms │ 13.06 / 13.23 ±0.15 / 13.51 ms │ no change │
│ QQuery 69 │ 76.13 / 79.82 ±4.99 / 89.27 ms │ 76.72 / 77.40 ±0.76 / 78.84 ms │ no change │
│ QQuery 70 │ 106.87 / 110.44 ±3.00 / 115.47 ms │ 105.01 / 109.92 ±6.93 / 123.68 ms │ no change │
│ QQuery 71 │ 35.58 / 35.96 ±0.27 / 36.41 ms │ 35.37 / 36.07 ±0.57 / 36.94 ms │ no change │
│ QQuery 72 │ 2132.94 / 2183.71 ±42.17 / 2236.73 ms │ 2111.52 / 2158.63 ±35.37 / 2214.83 ms │ no change │
│ QQuery 73 │ 9.03 / 9.24 ±0.22 / 9.65 ms │ 9.11 / 9.30 ±0.17 / 9.63 ms │ no change │
│ QQuery 74 │ 177.63 / 180.03 ±2.51 / 183.70 ms │ 175.98 / 181.10 ±5.65 / 191.23 ms │ no change │
│ QQuery 75 │ 145.51 / 146.86 ±1.25 / 149.13 ms │ 146.38 / 147.93 ±1.47 / 150.33 ms │ no change │
│ QQuery 76 │ 35.58 / 35.88 ±0.29 / 36.38 ms │ 35.34 / 35.87 ±0.46 / 36.52 ms │ no change │
│ QQuery 77 │ 59.95 / 60.42 ±0.49 / 61.35 ms │ 60.26 / 60.70 ±0.42 / 61.43 ms │ no change │
│ QQuery 78 │ 188.05 / 191.81 ±4.01 / 199.22 ms │ 187.70 / 191.78 ±3.19 / 195.27 ms │ no change │
│ QQuery 79 │ 67.60 / 68.26 ±0.66 / 69.06 ms │ 66.72 / 67.34 ±0.46 / 68.11 ms │ no change │
│ QQuery 80 │ 100.87 / 101.21 ±0.28 / 101.63 ms │ 100.03 / 101.28 ±1.64 / 104.53 ms │ no change │
│ QQuery 81 │ 24.09 / 24.30 ±0.12 / 24.43 ms │ 24.27 / 24.51 ±0.17 / 24.74 ms │ no change │
│ QQuery 82 │ 16.37 / 16.55 ±0.19 / 16.91 ms │ 16.57 / 16.72 ±0.17 / 17.03 ms │ no change │
│ QQuery 83 │ 36.90 / 38.82 ±2.29 / 43.14 ms │ 37.15 / 37.48 ±0.40 / 38.16 ms │ no change │
│ QQuery 84 │ 43.61 / 43.96 ±0.34 / 44.57 ms │ 44.04 / 45.58 ±1.86 / 49.12 ms │ no change │
│ QQuery 85 │ 135.59 / 136.80 ±1.39 / 139.37 ms │ 136.81 / 137.88 ±0.89 / 139.18 ms │ no change │
│ QQuery 86 │ 24.79 / 25.64 ±0.98 / 27.53 ms │ 25.15 / 25.53 ±0.22 / 25.76 ms │ no change │
│ QQuery 87 │ 68.82 / 70.07 ±0.82 / 71.02 ms │ 69.29 / 70.06 ±0.61 / 71.17 ms │ no change │
│ QQuery 88 │ 61.33 / 61.95 ±0.45 / 62.63 ms │ 62.63 / 63.33 ±0.57 / 64.00 ms │ no change │
│ QQuery 89 │ 35.25 / 35.73 ±0.26 / 36.03 ms │ 35.55 / 36.03 ±0.26 / 36.27 ms │ no change │
│ QQuery 90 │ 16.65 / 16.84 ±0.17 / 17.15 ms │ 16.98 / 17.23 ±0.20 / 17.48 ms │ no change │
│ QQuery 91 │ 52.24 / 54.16 ±2.38 / 58.84 ms │ 52.30 / 53.65 ±1.75 / 57.02 ms │ no change │
│ QQuery 92 │ 29.88 / 30.31 ±0.50 / 31.18 ms │ 29.84 / 30.43 ±0.40 / 30.97 ms │ no change │
│ QQuery 93 │ 50.10 / 50.89 ±0.46 / 51.43 ms │ 50.33 / 51.35 ±0.74 / 52.58 ms │ no change │
│ QQuery 94 │ 37.79 / 38.44 ±0.65 / 39.69 ms │ 37.56 / 38.59 ±0.57 / 39.14 ms │ no change │
│ QQuery 95 │ 85.10 / 85.88 ±0.49 / 86.60 ms │ 84.23 / 85.24 ±0.88 / 86.74 ms │ no change │
│ QQuery 96 │ 24.11 / 24.27 ±0.22 / 24.69 ms │ 24.29 / 24.50 ±0.19 / 24.83 ms │ no change │
│ QQuery 97 │ 45.50 / 46.04 ±0.35 / 46.55 ms │ 46.05 / 46.31 ±0.25 / 46.74 ms │ no change │
│ QQuery 98 │ 41.76 / 42.66 ±0.54 / 43.28 ms │ 42.44 / 43.15 ±0.62 / 43.97 ms │ no change │
│ QQuery 99 │ 70.01 / 70.78 ±0.61 / 71.75 ms │ 69.97 / 70.68 ±0.58 / 71.49 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 10498.86ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 10448.87ms │
│ Average Time (HEAD) │ 106.05ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 105.54ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 99 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 6.9 GiB |
| Avg memory | 6.1 GiB |
| CPU user | 234.8s |
| CPU sys | 6.0s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 7.0 GiB |
| Avg memory | 6.3 GiB |
| CPU user | 232.0s |
| CPU sys | 5.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4516196518
Nice, impressive 🚀🚀🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4705734492
run benchmark topk_tpch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4725640462
@adriangb It works well now for the latest run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720600971
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 5.20 / 5.75 ±0.90 / 7.54 ms │ 5.29 / 5.79 ±0.92 / 7.63 ms │ no change │
│ QQuery 2 │ 80.50 / 80.64 ±0.12 / 80.78 ms │ 81.17 / 81.49 ±0.22 / 81.76 ms │ no change │
│ QQuery 3 │ 28.65 / 29.04 ±0.23 / 29.33 ms │ 29.41 / 29.52 ±0.10 / 29.66 ms │ no change │
│ QQuery 4 │ 510.46 / 518.33 ±6.04 / 527.26 ms │ 491.25 / 499.11 ±5.52 / 508.09 ms │ no change │
│ QQuery 5 │ 51.08 / 51.49 ±0.55 / 52.54 ms │ 51.02 / 51.40 ±0.47 / 52.26 ms │ no change │
│ QQuery 6 │ 36.28 / 36.70 ±0.31 / 37.21 ms │ 36.11 / 36.77 ±0.47 / 37.38 ms │ no change │
│ QQuery 7 │ 94.54 / 96.00 ±1.73 / 99.30 ms │ 94.69 / 95.27 ±0.55 / 96.16 ms │ no change │
│ QQuery 8 │ 37.07 / 37.58 ±0.41 / 38.20 ms │ 36.66 / 38.80 ±3.00 / 44.71 ms │ no change │
│ QQuery 9 │ 53.25 / 54.35 ±0.86 / 55.47 ms │ 51.43 / 54.57 ±1.71 / 56.42 ms │ no change │
│ QQuery 10 │ 62.30 / 62.62 ±0.30 / 63.10 ms │ 62.82 / 63.23 ±0.25 / 63.56 ms │ no change │
│ QQuery 11 │ 321.25 / 322.41 ±1.01 / 324.14 ms │ 303.67 / 308.99 ±6.68 / 321.49 ms │ no change │
│ QQuery 12 │ 28.47 / 28.88 ±0.28 / 29.20 ms │ 28.14 / 28.65 ±0.41 / 29.32 ms │ no change │
│ QQuery 13 │ 119.32 / 120.38 ±1.66 / 123.67 ms │ 118.50 / 120.00 ±1.32 / 122.43 ms │ no change │
│ QQuery 14 │ 414.43 / 416.17 ±1.87 / 419.68 ms │ 408.04 / 412.86 ±4.28 / 419.65 ms │ no change │
│ QQuery 15 │ 60.13 / 60.67 ±0.57 / 61.70 ms │ 59.34 / 59.96 ±0.36 / 60.42 ms │ no change │
│ QQuery 16 │ 6.26 / 6.44 ±0.20 / 6.83 ms │ 6.37 / 6.53 ±0.24 / 7.00 ms │ no change │
│ QQuery 17 │ 80.66 / 82.60 ±1.85 / 85.10 ms │ 79.50 / 80.19 ±0.70 / 81.53 ms │ no change │
│ QQuery 18 │ 123.01 / 125.07 ±1.51 / 127.53 ms │ 123.58 / 125.14 ±1.46 / 127.15 ms │ no change │
│ QQuery 19 │ 41.84 / 42.17 ±0.31 / 42.75 ms │ 41.37 / 41.54 ±0.13 / 41.68 ms │ no change │
│ QQuery 20 │ 35.64 / 37.34 ±1.36 / 39.41 ms │ 35.49 / 36.58 ±0.81 / 38.01 ms │ no change │
│ QQuery 21 │ 17.64 / 17.83 ±0.24 / 18.30 ms │ 17.32 / 17.52 ±0.17 / 17.76 ms │ no change │
│ QQuery 22 │ 63.63 / 64.23 ±0.46 / 65.05 ms │ 61.57 / 63.18 ±1.02 / 64.72 ms │ no change │
│ QQuery 23 │ 349.68 / 355.04 ±3.52 / 360.70 ms │ 342.40 / 344.53 ±1.52 / 347.07 ms │ no change │
│ QQuery 24 │ 224.22 / 228.17 ±5.94 / 239.96 ms │ 222.95 / 225.64 ±1.84 / 227.53 ms │ no change │
│ QQuery 25 │ 110.46 / 111.53 ±0.75 / 112.48 ms │ 109.32 / 110.54 ±0.66 / 111.21 ms │ no change │
│ QQuery 26 │ 57.89 / 59.31 ±1.42 / 61.95 ms │ 58.20 / 58.97 ±0.45 / 59.52 ms │ no change │
│ QQuery 27 │ 6.17 / 6.34 ±0.14 / 6.59 ms │ 5.95 / 6.11 ±0.19 / 6.48 ms │ no change │
│ QQuery 28 │ 56.65 / 60.54 ±1.98 / 62.09 ms │ 60.74 / 61.00 ±0.30 / 61.52 ms │ no change │
│ QQuery 29 │ 97.88 / 99.57 ±1.45 / 101.59 ms │ 96.88 / 98.33 ±1.53 / 101.24 ms │ no change │
│ QQuery 30 │ 32.29 / 32.62 ±0.27 / 33.10 ms │ 31.85 / 33.05 ±1.06 / 34.99 ms │ no change │
│ QQuery 31 │ 110.58 / 111.39 ±0.49 / 112.03 ms │ 111.86 / 112.19 ±0.39 / 112.93 ms │ no change │
│ QQuery 32 │ 20.34 / 20.58 ±0.23 / 20.98 ms │ 20.66 / 20.93 ±0.19 / 21.20 ms │ no change │
│ QQuery 33 │ 37.53 / 39.51 ±2.24 / 43.75 ms │ 38.25 / 38.34 ±0.10 / 38.51 ms │ no change │
│ QQuery 34 │ 9.67 / 9.99 ±0.33 / 10.57 ms │ 9.70 / 9.91 ±0.18 / 10.21 ms │ no change │
│ QQuery 35 │ 71.94 / 72.76 ±0.52 / 73.42 ms │ 72.23 / 74.45 ±2.44 / 79.12 ms │ no change │
│ QQuery 36 │ 5.55 / 5.69 ±0.19 / 6.07 ms │ 5.54 / 5.66 ±0.20 / 6.06 ms │ no change │
│ QQuery 37 │ 6.86 / 6.89 ±0.04 / 6.96 ms │ 6.83 / 6.93 ±0.09 / 7.10 ms │ no change │
│ QQuery 38 │ 64.04 / 64.29 ±0.19 / 64.56 ms │ 63.05 / 63.68 ±0.51 / 64.57 ms │ no change │
│ QQuery 39 │ 464.70 / 471.86 ±6.59 / 483.19 ms │ 459.21 / 466.78 ±5.03 / 472.18 ms │ no change │
│ QQuery 40 │ 23.50 / 23.81 ±0.28 / 24.28 ms │ 23.19 / 23.35 ±0.12 / 23.56 ms │ no change │
│ QQuery 41 │ 11.64 / 11.86 ±0.22 / 12.23 ms │ 11.29 / 12.57 ±2.26 / 17.09 ms │ 1.06x slower │
│ QQuery 42 │ 23.98 / 24.15 ±0.10 / 24.30 ms │ 24.14 / 24.24 ±0.07 / 24.36 ms │ no change │
│ QQuery 43 │ 4.70 / 4.84 ±0.22 / 5.29 ms │ 4.55 / 4.67 ±0.14 / 4.93 ms │ no change │
│ QQuery 44 │ 9.11 / 9.55 ±0.59 / 10.72 ms │ 8.87 / 9.00 ±0.09 / 9.12 ms │ +1.06x faster │
│ QQuery 45 │ 39.68 / 40.40 ±0.42 / 40.90 ms │ 38.72 / 39.45 ±0.57 / 40.24 ms │ no change │
│ QQuery 46 │ 11.53 / 11.64 ±0.08 / 11.78 ms │ 11.26 / 11.40 ±0.15 / 11.65 ms │ no change │
│ QQuery 47 │ 235.71 / 238.58 ±1.93 / 240.61 ms │ 226.55 / 236.79 ±10.58 / 256.19 ms │ no change │
│ QQuery 48 │ 96.37 / 97.21 ±0.57 / 97.84 ms │ 96.65 / 99.54 ±2.87 / 103.15 ms │ no change │
│ QQuery 49 │ 75.82 / 76.33 ±0.42 / 76.89 ms │ 75.44 / 76.15 ±0.61 / 77.27 ms │ no change │
│ QQuery 50 │ 59.53 / 62.59 ±4.00 / 70.12 ms │ 58.91 / 60.56 ±2.22 / 64.92 ms │ no change │
│ QQuery 51 │ 91.42 / 92.98 ±2.03 / 96.93 ms │ 92.51 / 93.59 ±0.82 / 94.92 ms │ no change │
│ QQuery 52 │ 24.32 / 24.46 ±0.15 / 24.71 ms │ 23.80 / 24.14 ±0.23 / 24.43 ms │ no change │
│ QQuery 53 │ 30.01 / 31.82 ±3.20 / 38.21 ms │ 29.15 / 29.54 ±0.22 / 29.78 ms │ +1.08x faster │
│ QQuery 54 │ 55.34 / 56.04 ±0.38 / 56.52 ms │ 54.43 / 55.79 ±1.74 / 59.17 ms │ no change │
│ QQuery 55 │ 23.95 / 24.24 ±0.23 / 24.63 ms │ 23.27 / 23.95 ±0.45 / 24.34 ms │ no change │
│ QQuery 56 │ 38.97 / 39.37 ±0.43 / 40.12 ms │ 38.92 / 40.07 ±1.63 / 43.20 ms │ no change │
│ QQuery 57 │ 178.34 / 181.33 ±4.09 / 189.42 ms │ 176.09 / 177.94 ±1.96 / 181.45 ms │ no change │
│ QQuery 58 │ 115.33 / 117.99 ±1.95 / 120.37 ms │ 115.39 / 116.79 ±0.86 / 117.88 ms │ no change │
│ QQuery 59 │ 116.64 / 118.12 ±0.85 / 119.10 ms │ 118.62 / 119.89 ±1.39 / 122.12 ms │ no change │
│ QQuery 60 │ 39.41 / 39.97 ±0.40 / 40.64 ms │ 38.85 / 39.41 ±0.36 / 39.78 ms │ no change │
│ QQuery 61 │ 11.93 / 12.01 ±0.10 / 12.20 ms │ 11.60 / 11.68 ±0.10 / 11.88 ms │ no change │
│ QQuery 62 │ 47.45 / 48.96 ±1.63 / 51.53 ms │ 46.89 / 47.91 ±1.73 / 51.36 ms │ no change │
│ QQuery 63 │ 29.93 / 30.45 ±0.32 / 30.92 ms │ 29.39 / 29.94 ±0.43 / 30.65 ms │ no change │
│ QQuery 64 │ 413.11 / 417.57 ±3.30 / 423.14 ms │ 408.86 / 413.58 ±3.72 / 419.15 ms │ no change │
│ QQuery 65 │ 141.49 / 147.62 ±4.72 / 153.76 ms │ 139.52 / 142.60 ±1.93 / 145.01 ms │ no change │
│ QQuery 66 │ 79.48 / 80.56 ±0.73 / 81.36 ms │ 79.52 / 82.86 ±3.40 / 88.62 ms │ no change │
│ QQuery 67 │ 252.54 / 257.91 ±3.55 / 262.95 ms │ 244.19 / 250.83 ±4.20 / 255.50 ms │ no change │
│ QQuery 68 │ 11.77 / 11.82 ±0.04 / 11.88 ms │ 11.40 / 11.61 ±0.15 / 11.85 ms │ no change │
│ QQuery 69 │ 56.93 / 58.94 ±1.86 / 61.54 ms │ 57.55 / 57.70 ±0.14 / 57.92 ms │ no change │
│ QQuery 70 │ 105.08 / 109.89 ±4.49 / 118.09 ms │ 105.83 / 107.53 ±1.31 / 109.88 ms │ no change │
│ QQuery 71 │ 35.07 / 35.54 ±0.30 / 35.93 ms │ 34.97 / 35.24 ±0.22 / 35.49 ms │ no change │
│ QQuery 72 │ 2053.44 / 2199.73 ±76.94 / 2264.84 ms │ 2112.29 / 2188.38 ±54.66 / 2269.41 ms │ no change │
│ QQuery 73 │ 9.45 / 9.70 ±0.22 / 10.10 ms │ 9.37 / 13.55 ±7.94 / 29.42 ms │ 1.40x slower │
│ QQuery 74 │ 179.14 / 181.60 ±1.71 / 183.97 ms │ 172.77 / 175.37 ±1.46 / 176.75 ms │ no change │
│ QQuery 75 │ 148.57 / 152.25 ±4.35 / 159.84 ms │ 149.12 / 152.68 ±2.77 / 157.29 ms │ no change │
│ QQuery 76 │ 34.95 / 35.71 ±0.55 / 36.52 ms │ 35.94 / 36.54 ±0.49 / 37.14 ms │ no change │
│ QQuery 77 │ 60.14 / 63.69 ±5.45 / 74.44 ms │ 61.91 / 63.22 ±1.59 / 66.26 ms │ no change │
│ QQuery 78 │ 186.97 / 189.36 ±2.65 / 194.33 ms │ 185.83 / 195.81 ±10.71 / 216.59 ms │ no change │
│ QQuery 79 │ 66.66 / 67.53 ±1.09 / 69.67 ms │ 66.40 / 66.85 ±0.30 / 67.33 ms │ no change │
│ QQuery 80 │ 99.30 / 99.99 ±0.70 / 100.95 ms │ 98.23 / 100.00 ±2.03 / 103.68 ms │ no change │
│ QQuery 81 │ 25.59 / 27.32 ±2.41 / 32.04 ms │ 25.36 / 25.52 ±0.12 / 25.64 ms │ +1.07x faster │
│ QQuery 82 │ 16.90 / 18.02 ±1.69 / 21.34 ms │ 16.14 / 18.34 ±3.75 / 25.83 ms │ no change │
│ QQuery 83 │ 40.11 / 41.85 ±2.71 / 47.21 ms │ 39.91 / 40.59 ±0.49 / 41.40 ms │ no change │
│ QQuery 84 │ 30.71 / 30.97 ±0.22 / 31.28 ms │ 31.12 / 31.93 ±1.46 / 34.85 ms │ no change │
│ QQuery 85 │ 106.86 / 108.55 ±1.98 / 112.29 ms │ 107.55 / 109.43 ±2.23 / 113.79 ms │ no change │
│ QQuery 86 │ 25.65 / 27.01 ±1.33 / 29.53 ms │ 25.60 / 27.19 ±2.93 / 33.06 ms │ no change │
│ QQuery 87 │ 64.38 / 65.29 ±0.81 / 66.59 ms │ 64.42 / 65.12 ±0.52 / 65.86 ms │ no change │
│ QQuery 88 │ 61.01 / 61.95 ±0.54 / 62.59 ms │ 62.13 / 63.32 ±1.58 / 66.30 ms │ no change │
│ QQuery 89 │ 35.37 / 35.78 ±0.32 / 36.29 ms │ 35.71 / 36.32 ±0.48 / 37.10 ms │ no change │
│ QQuery 90 │ 16.68 / 18.78 ±2.88 / 24.23 ms │ 16.57 / 16.76 ±0.14 / 16.95 ms │ +1.12x faster │
│ QQuery 91 │ 46.48 / 47.19 ±0.72 / 48.11 ms │ 46.23 / 46.38 ±0.16 / 46.63 ms │ no change │
│ QQuery 92 │ 29.81 / 30.63 ±0.59 / 31.58 ms │ 29.73 / 31.21 ±1.67 / 34.10 ms │ no change │
│ QQuery 93 │ 50.40 / 51.14 ±0.74 / 52.46 ms │ 50.72 / 51.55 ±0.81 / 53.09 ms │ no change │
│ QQuery 94 │ 38.08 / 38.61 ±0.44 / 39.33 ms │ 38.54 / 38.90 ±0.24 / 39.26 ms │ no change │
│ QQuery 95 │ 81.75 / 84.69 ±2.81 / 90.06 ms │ 81.05 / 81.92 ±0.57 / 82.78 ms │ no change │
│ QQuery 96 │ 24.08 / 24.23 ±0.12 / 24.41 ms │ 24.36 / 25.22 ±1.27 / 27.73 ms │ no change │
│ QQuery 97 │ 46.30 / 46.69 ±0.29 / 47.05 ms │ 46.75 / 47.56 ±0.57 / 48.42 ms │ no change │
│ QQuery 98 │ 42.55 / 43.10 ±0.47 / 43.98 ms │ 42.84 / 43.52 ±0.45 / 44.15 ms │ no change │
│ QQuery 99 │ 71.84 / 73.27 ±2.43 / 78.11 ms │ 71.63 / 71.89 ±0.29 / 72.41 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 10465.97ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 10369.53ms │
│ Average Time (HEAD) │ 105.72ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 104.74ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 93 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.6 GiB |
| CPU user | 231.8s |
| CPU sys | 5.9s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.4 GiB |
| Avg memory | 1.7 GiB |
| CPU user | 237.4s |
| CPU sys | 5.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas merged PR #22450:
URL: https://github.com/apache/datafusion/pull/22450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765059254
run benchmark topk_tpch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4702240019
> Did some quick review but need to find more time to look in details.
>
> I want to double check:
>
> 1. Logic for deciding when we need to go down a different path. I see this is using `DynamicFilterTracker::classify` but I'm not sure if we should always or optionally create a RowGroupPruner`, etc.
> 2. Logic for splitting runs. I thought the APIs we added in arrow-rs are flexible enough to not require splitting runs, they allow pausing a run at row group boundaries and changing it (e.g. skipping subsequent row groups) but maybe it doesn't have the right apis...
Thanks @adriangb for the review!
(2) splitting runs: Already gone — 7e1aae26a collapses to a single decoder paused at RG boundaries, rebuilt via decoder.into_builder().with_row_groups(remaining).build() to skip pruned RGs. RowGroupRun /
split_runs are deleted. The arrow-rs API is exactly enough.
(1) when to create RowGroupPruner: Agreed, current gate is too loose. Static predicates were already fully consumed by prune_by_statistics at open time, so a runtime pruner over them is wasted work. Will tighten
to classify(&predicate) == Watching, matching the fmt_extra marker.
Inline on snapshot_generation: Stale — the pruner already uses the DynamicFilterTracker watcher from #22460. Will fix the comment in the same follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720452696
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4720419417-575-nclfk 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (6c554cce66f8bf6126f821bb8822f740d24287cc) to 6176a6d (merge-base) [diff](https://github.com/apache/datafusion/compare/6176a6dae1f36e30c83bc880bb8b6cb1d1a175a8..6c554cce66f8bf6126f821bb8822f740d24287cc) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3419902761
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,51 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when all three conditions hold:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) the predicate has at least one not-yet-complete dynamic filter
+ // (`DynamicFilterTracking::Watching`) — static or already-complete
+ // predicates were fully consumed by `prune_by_statistics` at file
+ // open, so re-evaluating them per RG boundary would be wasted work,
+ // 3) there is at least one pending RG that could be skipped.
+ // The pruner subscribes once to every still-incomplete dynamic filter
+ // via the `DynamicFilterTracker` watch channel (#22460), so detecting
+ // a threshold change is a single atomic load — not a tree walk per
+ // RG check.
+ let row_group_pruner = match (&prepared.predicate, rg_plan.len() > 1) {
+ (Some(predicate), true)
+ if DynamicFilterTracking::classify(predicate)
+ .contains_dynamic_filter() =>
Review Comment:
I think `contains_dynamic_filter()` also returns `true` for `AllComplete`.
I suggest we `if matches!(DynamicFilterTracking::classify(predicate), DynamicFilterTracking::Watching(_))` or something like that.
##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -151,7 +305,72 @@ impl PushDecoderStreamState {
if self.remaining_limit == Some(0) {
return None;
}
- match self.decoder.try_decode() {
+
+ // Step 1: drain a batch from the active reader if any.
+ if let Some(reader) = self.active_reader.as_mut() {
+ match reader.next() {
+ Some(Ok(batch)) => {
+ let mut timer = self.baseline_metrics.elapsed_compute().timer();
+ self.copy_arrow_reader_metrics();
+ let result = self.project_batch(&batch);
+ timer.stop();
+ drop(timer);
+ return Some((result, self));
+ }
+ Some(Err(e)) => {
+ return Some((Err(DataFusionError::from(e)), self));
+ }
+ None => {
+ // Reader exhausted: drop and fall through to per-RG
+ // boundary handling, then try_next_reader.
+ self.active_reader = None;
+ }
+ }
+ }
+
+ // Step 2: at RG boundary, drop pruned RGs from the head of the
+ // plan. Rebuild the decoder via `into_builder` so it skips the
+ // pruned RGs entirely. Buffered bytes for already-fetched RGs
+ // carry across the rebuild.
+ if !self.rg_plan.is_empty() {
+ let mut pruned_head = 0usize;
+ while let Some(next) = self.rg_plan.front() {
+ let pruned = self
+ .row_group_pruner
+ .as_mut()
+ .map(|p| p.should_prune(&[next.rg_index]))
+ .unwrap_or(false);
+ if pruned {
+ pruned_head += 1;
+ self.rg_plan.pop_front();
+ self.row_groups_pruned_dynamic.add(1);
+ } else {
+ break;
+ }
+ }
+ if pruned_head > 0 {
+ if self.rg_plan.is_empty() {
+ return None;
+ }
+ let decoder = self.decoder.take().expect("decoder present");
+ let new_indices: Vec<usize> =
+ self.rg_plan.iter().map(|e| e.rg_index).collect();
+ let rebuilt = match decoder.into_builder() {
+ Ok(b) => b.with_row_groups(new_indices).build(),
+ Err(e) => Err(e),
+ };
Review Comment:
Do we need to adjust the `RowSelection` here? Or does that just work fine? Might be worth adding a test where there will be a row selection (e.g. every other page skipped because of a filter).
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1366,50 +1368,41 @@ impl RowGroupsPrunedParquetOpen {
&prepared.file_metrics,
);
- // Split into consecutive runs of row groups that share the same filter
- // requirement. Fully matched row groups skip the RowFilter; others need it.
- // Reverse the run order for reverse scans so the combined decoder stream
- // preserves the requested global row group order.
- let mut runs = access_plan.split_runs(row_filter_generator.has_row_filter());
- if prepared.reverse_row_groups {
- runs.reverse();
- }
- let run_count = runs.len();
- let decoder_limit = prepared.limit.filter(|_| run_count == 1);
- let remaining_limit = prepared.limit.filter(|_| run_count > 1);
-
+ // Build the prepared access plan first — `prepare_access_plan` may
+ // call `reorder_by_statistics` (for `sort_order_for_reorder`) and
+ // `reverse` (for `reverse_row_groups`), both of which mutate
+ // `row_group_indexes` to the physical scan order the decoder will
+ // actually read. We MUST build our `rg_plan` from this reordered
+ // list, otherwise our per-RG pruner check would consult the
+ // metadata of a different RG than the decoder is about to yield.
let decoder_config = DecoderBuilderConfig {
projection_mask: decoder_projection.projection_mask(),
batch_size: prepared.batch_size,
arrow_reader_metrics: &arrow_reader_metrics,
force_filter_selections: prepared.force_filter_selections,
- decoder_limit,
+ decoder_limit: prepared.limit,
};
- // Build a decoder per run.
- let mut decoders = VecDeque::with_capacity(runs.len());
- for run in runs {
- let prepared_access_plan = prepare_access_plan(run.access_plan)?;
- let mut builder =
- decoder_config.build(prepared_access_plan, reader_metadata.clone());
- if run.needs_filter {
- if let Some(row_filter) = row_filter_generator.next_filter() {
- builder = builder.with_row_filter(row_filter);
- }
- if let Some(max_predicate_cache_size) =
- prepared.max_predicate_cache_size
- {
- builder = builder
- .with_max_predicate_cache_size(max_predicate_cache_size);
- }
+ let prepared_access_plan = prepare_access_plan(access_plan)?;
+ let rg_plan: VecDeque<RgPlanEntry> = prepared_access_plan
+ .row_group_indexes
+ .iter()
+ .copied()
+ .map(|rg_index| RgPlanEntry { rg_index })
+ .collect();
+
+ let mut builder =
+ decoder_config.build(prepared_access_plan, reader_metadata.clone());
+ if let Some(row_filter) = row_filter_generator.next_filter() {
+ builder = builder.with_row_filter(row_filter);
Review Comment:
`split_runs` used to build a different decoder for each row group when there was row groups that fully matched filters, allowing those to skip applying the row filter. I think we're loosing that now?
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,51 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when all three conditions hold:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) the predicate has at least one not-yet-complete dynamic filter
+ // (`DynamicFilterTracking::Watching`) — static or already-complete
+ // predicates were fully consumed by `prune_by_statistics` at file
+ // open, so re-evaluating them per RG boundary would be wasted work,
+ // 3) there is at least one pending RG that could be skipped.
+ // The pruner subscribes once to every still-incomplete dynamic filter
+ // via the `DynamicFilterTracker` watch channel (#22460), so detecting
+ // a threshold change is a single atomic load — not a tree walk per
+ // RG check.
+ let row_group_pruner = match (&prepared.predicate, rg_plan.len() > 1) {
+ (Some(predicate), true)
+ if DynamicFilterTracking::classify(predicate)
+ .contains_dynamic_filter() =>
+ {
+ Some(RowGroupPruner::new(
+ Arc::clone(predicate),
+ Arc::clone(&prepared.physical_file_schema),
+ Arc::clone(reader_metadata.metadata()),
+ prepared.predicate_creation_errors.clone(),
+ prepared.file_metrics.predicate_evaluation_errors.clone(),
+ ))
+ }
+ _ => None,
+ };
+ let row_groups_pruned_dynamic = prepared
+ .file_metrics
+ .row_groups_pruned_dynamic_filter
+ .clone();
+
let stream = PushDecoderStreamState {
- decoder,
- pending_decoders,
- remaining_limit,
+ decoder: Some(decoder),
+ active_reader: None,
+ rg_plan,
+ remaining_limit: None,
Review Comment:
`remaining_limit` seems like dead code to me, I don't see that it's actually used anywhere
##########
datafusion/core/tests/parquet/dynamic_row_group_pruning.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! End-to-end test for **runtime row-group pruning** driven by a TopK
+//! `SortExec`'s `DynamicFilterPhysicalExpr`.
+//!
+//! A 5-row-group parquet file is constructed with disjoint statistics on
+//! the sort column (`v`): row group `i` contains values
+//! `[i*100, (i+1)*100)`. The query `ORDER BY v DESC LIMIT 5` fills the
+//! TopK heap from the row group with the largest values; the threshold
+//! then proves the remaining row groups cannot contribute. The runtime
+//! `RowGroupPruner` in the parquet scan must observe the tightened
+//! threshold and increment `row_groups_pruned_dynamic_filter`.
+//!
+//! We assert a property (`pruned >= 1`) rather than an exact count
+//! because batch-arrival timing affects how soon the TopK heap fills,
+//! and we don't want this test to become flaky.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+
+use crate::parquet::Unit::RowGroup;
+use crate::parquet::{ContextWithParquet, Scenario};
+
+/// Build five `RecordBatch`es whose `v` column ranges are disjoint:
+/// batch `i` carries `v` values `[i*100, (i+1)*100)`. When written with
+/// `max_row_group_row_count = 100` each batch lands in its own row group.
+fn build_five_disjoint_batches(schema: &Arc<Schema>) -> Vec<RecordBatch> {
+ (0..5i64)
+ .map(|rg| {
+ let base = rg * 100;
+ let values: Vec<i64> = (base..base + 100).collect();
+ let col: ArrayRef = Arc::new(Int64Array::from(values));
+ RecordBatch::try_new(Arc::clone(schema), vec![col]).unwrap()
+ })
+ .collect()
+}
+
+/// Build five `RecordBatch`es in *descending* value order: batch 0 holds
+/// `v ∈ [400, 500)`, batch 4 holds `v ∈ [0, 100)`. The physical row-group
+/// order on disk therefore does **not** match the order a `ORDER BY v ASC`
+/// query wants — sort-pushdown's `reorder_by_statistics` must rearrange
+/// the access plan so the scan reads RG 4 first, then RG 3, etc.
+fn build_five_disjoint_batches_desc(schema: &Arc<Schema>) -> Vec<RecordBatch> {
+ (0..5i64)
+ .map(|rg| {
+ let base = (4 - rg) * 100;
+ let values: Vec<i64> = (base..base + 100).collect();
+ let col: ArrayRef = Arc::new(Int64Array::from(values));
+ RecordBatch::try_new(Arc::clone(schema), vec![col]).unwrap()
+ })
+ .collect()
+}
+
+/// `ORDER BY v DESC LIMIT 5` against a 5-RG file with disjoint per-RG
+/// stats must trigger runtime RG pruning: the first RG read fills the
+/// heap, and the tightened threshold proves every other RG unreachable.
+#[tokio::test]
+async fn dynamic_rg_pruning_metric_fires_for_topk_descending_limit() {
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batches = build_five_disjoint_batches(&schema);
+
+ // `with_custom_data` honors the custom schema + batches and ignores
+ // `Scenario`. `Unit::RowGroup(100)` enables `pushdown_filters`, which
+ // is required for the TopK dynamic filter to reach the parquet scan.
+ let mut ctx = ContextWithParquet::with_custom_data(
+ Scenario::Int,
+ RowGroup(100),
+ Arc::clone(&schema),
+ batches,
+ )
+ .await;
+
+ let output = ctx.query("SELECT v FROM t ORDER BY v DESC LIMIT 5").await;
+
+ assert_eq!(output.result_rows, 5, "query must return LIMIT rows",);
+
+ let pruned = output
+ .row_groups_pruned_dynamic_filter()
+ .expect("`row_groups_pruned_dynamic_filter` metric must be registered");
+ assert!(
+ pruned >= 1,
+ "dynamic RG pruner must skip at least one row group; \
+ pruned={pruned}\n{}",
+ output.description(),
+ );
+}
+
+/// Regression for the rg_plan / `reorder_by_statistics` ordering bug.
+///
+/// When `sort_order_for_reorder` is set on the parquet scan,
+/// `prepare_access_plan` calls
+/// [`PreparedAccessPlan::reorder_by_statistics`], which rearranges
+/// `row_group_indexes` so the decoder reads row groups in stats-optimal
+/// order (smallest-min first for ASC, etc.). The stream's per-RG plan
+/// (`rg_plan`) — which the runtime pruner walks one entry at a time —
+/// **must use this reordered list**, not the access plan's natural
+/// (index-ascending) order. Otherwise the pruner would consult the
+/// metadata of RG K while the decoder is actually about to yield RG K',
+/// silently producing wrong results.
+///
+/// This test makes the failure visible:
+///
+/// - File is written with RGs in *descending* `v` order (RG 0 has the
+/// largest values, RG 4 has the smallest).
+/// - Query is `ORDER BY v ASC LIMIT 5`, so sort-pushdown reorders the
+/// access plan to read RG 4 first, then RG 3, etc.
+/// - The smallest five values (which form the entire correct LIMIT
+/// answer) live in RG 4 alone. After they are emitted, the TopK
+/// threshold tightens enough that the per-RG pruner skips every other
+/// RG.
+///
+/// Without the fix, `rg_plan` would be `[0, 1, 2, 3, 4]` while the
+/// decoder reads `[4, 3, 2, 1, 0]`. The first yielded reader (for RG 4
+/// in the decoder) would be tracked as if it were RG 0, the pruner
+/// would check RG 1's stats (id range 300..400) against a threshold
+/// already tightened to `v < 5`, prune RG 1 (because nothing in
+/// 300..400 can satisfy `v < 5`), and then the rebuild via
+/// `into_builder` would scan a row group whose data does not match its
+/// expected metadata. The query would return fewer than five rows or
+/// the wrong rows.
+#[tokio::test]
+async fn dynamic_rg_pruning_handles_sort_pushdown_reorder() {
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batches = build_five_disjoint_batches_desc(&schema);
+
+ let mut ctx = ContextWithParquet::with_custom_data(
+ Scenario::Int,
+ RowGroup(100),
+ Arc::clone(&schema),
+ batches,
+ )
+ .await;
+
+ let output = ctx.query("SELECT v FROM t ORDER BY v ASC LIMIT 5").await;
+
+ // Correctness — the five smallest values in the file are 0..=4.
+ // If `rg_plan` is misaligned with the decoder's read order, the
+ // pruner consults the wrong RG's stats and the result row count or
+ // values would drift.
+ assert_eq!(output.result_rows, 5, "query must return LIMIT rows");
+ let formatted = output.pretty_results();
+ for v in 0..=4i64 {
+ assert!(
+ formatted.contains(&format!("| {v} ")),
+ "output must contain the smallest value {v}; got:\n{formatted}",
+ );
+ }
+
+ // Behavior — the per-RG pruner must engage. We don't pin the exact
+ // count (batch-arrival timing affects how soon the heap fills); we
+ // only require that at least one row group is skipped at runtime.
+ let pruned = output
+ .row_groups_pruned_dynamic_filter()
+ .expect("`row_groups_pruned_dynamic_filter` metric must be registered");
+ assert!(
+ pruned >= 1,
+ "with `sort_order_for_reorder` active and a tight TopK, the \
+ runtime pruner must skip at least one row group; pruned={pruned}\n{}",
+ output.description(),
+ );
+}
+
+/// A query without ORDER BY does not produce a TopK and therefore no
+/// `DynamicFilterPhysicalExpr` reaches the scan. The runtime pruner must
+/// stay quiet — the metric should be 0.
+#[tokio::test]
+async fn dynamic_rg_pruning_metric_quiet_without_topk() {
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batches = build_five_disjoint_batches(&schema);
+
+ let mut ctx = ContextWithParquet::with_custom_data(
+ Scenario::Int,
+ RowGroup(100),
+ Arc::clone(&schema),
+ batches,
+ )
+ .await;
+
+ // Plain `SELECT *` — no sort, no limit, no dynamic filter.
+ let output = ctx.query("SELECT v FROM t").await;
+ assert_eq!(output.result_rows, 500);
+
+ let pruned = output.row_groups_pruned_dynamic_filter().unwrap_or(0);
+ assert_eq!(
+ pruned,
+ 0,
+ "without TopK there is no dynamic filter, so the runtime pruner \
+ must not fire; pruned={pruned}\n{}",
+ output.description(),
+ );
+}
Review Comment:
I think it's worth adding a test that would trigger a `RowFilter`, e.g. selects 1/2 of the pages or every other page.
##########
datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt:
##########
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# End-to-end SLT for **dynamic row-group pruning** driven by a TopK
+# `SortExec`'s `DynamicFilterPhysicalExpr`.
+#
+# Builds a 5-row-group parquet file with disjoint per-RG ranges of `v`:
+# RG 0: 0..3, RG 1: 3..6, RG 2: 6..9, RG 3: 9..12, RG 4: 12..15
+# `ORDER BY v DESC LIMIT 3` fills the TopK heap from the row group with
+# the largest values; the tightened threshold then proves every other
+# row group unreachable. The opener splits decoder runs per row group
Review Comment:
I think this is outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720587483
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 38.42 / 39.13 ±0.90 / 40.89 ms │ 38.44 / 39.37 ±1.08 / 41.46 ms │ no change │
│ QQuery 2 │ 18.84 / 19.28 ±0.27 / 19.67 ms │ 19.07 / 19.45 ±0.26 / 19.83 ms │ no change │
│ QQuery 3 │ 30.82 / 32.90 ±1.49 / 34.37 ms │ 30.68 / 33.39 ±1.42 / 34.76 ms │ no change │
│ QQuery 4 │ 17.50 / 17.95 ±0.57 / 19.01 ms │ 17.49 / 17.70 ±0.14 / 17.86 ms │ no change │
│ QQuery 5 │ 39.15 / 41.18 ±1.16 / 42.75 ms │ 37.54 / 39.56 ±1.50 / 41.02 ms │ no change │
│ QQuery 6 │ 16.12 / 16.30 ±0.13 / 16.45 ms │ 16.38 / 16.46 ±0.11 / 16.66 ms │ no change │
│ QQuery 7 │ 45.53 / 47.60 ±1.41 / 49.70 ms │ 44.30 / 46.14 ±1.17 / 47.63 ms │ no change │
│ QQuery 8 │ 42.84 / 43.13 ±0.25 / 43.47 ms │ 42.96 / 43.20 ±0.17 / 43.39 ms │ no change │
│ QQuery 9 │ 49.38 / 50.71 ±0.95 / 52.27 ms │ 50.71 / 50.92 ±0.14 / 51.13 ms │ no change │
│ QQuery 10 │ 42.45 / 43.35 ±1.02 / 45.23 ms │ 42.39 / 42.62 ±0.18 / 42.94 ms │ no change │
│ QQuery 11 │ 13.17 / 13.39 ±0.20 / 13.73 ms │ 13.76 / 13.96 ±0.19 / 14.26 ms │ no change │
│ QQuery 12 │ 23.76 / 24.07 ±0.23 / 24.42 ms │ 24.47 / 24.76 ±0.22 / 25.09 ms │ no change │
│ QQuery 13 │ 33.41 / 35.29 ±1.64 / 37.97 ms │ 32.79 / 34.69 ±1.93 / 37.80 ms │ no change │
│ QQuery 14 │ 23.79 / 24.03 ±0.14 / 24.16 ms │ 23.87 / 24.31 ±0.56 / 25.41 ms │ no change │
│ QQuery 15 │ 31.07 / 32.68 ±1.79 / 35.51 ms │ 31.50 / 32.40 ±0.58 / 33.16 ms │ no change │
│ QQuery 16 │ 14.19 / 14.38 ±0.15 / 14.64 ms │ 14.12 / 14.36 ±0.22 / 14.63 ms │ no change │
│ QQuery 17 │ 74.50 / 75.89 ±2.03 / 79.85 ms │ 73.90 / 75.00 ±0.70 / 75.90 ms │ no change │
│ QQuery 18 │ 59.18 / 60.45 ±0.93 / 61.68 ms │ 59.64 / 60.31 ±0.47 / 60.96 ms │ no change │
│ QQuery 19 │ 33.18 / 34.49 ±1.37 / 36.49 ms │ 33.33 / 34.35 ±0.83 / 35.69 ms │ no change │
│ QQuery 20 │ 31.98 / 32.21 ±0.24 / 32.51 ms │ 31.99 / 32.64 ±0.64 / 33.58 ms │ no change │
│ QQuery 21 │ 56.72 / 57.56 ±0.87 / 58.82 ms │ 57.08 / 57.89 ±0.71 / 58.95 ms │ no change │
│ QQuery 22 │ 14.12 / 14.33 ±0.17 / 14.51 ms │ 14.22 / 15.16 ±1.09 / 17.23 ms │ 1.06x slower │
└───────────┴────────────────────────────────┴────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 770.31ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 768.66ms │
│ Average Time (HEAD) │ 35.01ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 34.94ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 519.2 MiB |
| CPU user | 22.4s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 534.6 MiB |
| CPU user | 22.3s |
| CPU sys | 1.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4718824695
> > If the pruner proves the head RG unwinnable, that index is dropped from the plan
>
> Why only the head row group? I think it will cost the same to evaluate all of them, might as well prune as many as we can? It will lead to less rebuilds. Otherwise if we have rgs 1, 2, 3, 4 and after 1 we could skip 2 and 4 but not 3 we will do 2 rebuilds instead of 1.
Good catch @adriangb !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4769466926
nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4539412889
> I did a little experiment: #22460
Thanks @adriangb , i have updated the PR based on #22460.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4540404565
> fwiw that should go out in next minor release, so 2-4 weeks until it's available in DataFusion
Good, i will wait it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3286139005
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1234,16 +1255,38 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when both halves are useful:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) there is at least one pending run that could be skipped.
+ // The pruner consults the predicate's `snapshot_generation` so the
+ // cost is one rebuild per dynamic-filter update, not per RG check.
+ let row_group_pruner = match (&prepared.predicate, pending_runs.is_empty()) {
+ (Some(predicate), false) => Some(RowGroupPruner::new(
+ Arc::clone(predicate),
+ Arc::clone(&prepared.physical_file_schema),
+ Arc::clone(reader_metadata.metadata()),
+ prepared.file_metrics.predicate_evaluation_errors.clone(),
Review Comment:
`RowGroupPruner::new` takes a `predicate_creation_errors` counter (used by `build_pruning_predicate`), but this call site passes `prepared.file_metrics.predicate_evaluation_errors`. This will misattribute pruning-predicate construction failures as evaluation failures, and also makes the metric semantics inconsistent with the existing static row-group pruning path (which uses `prepared.predicate_creation_errors` for construction and `file_metrics.predicate_evaluation_errors` for evaluation). Consider passing `prepared.predicate_creation_errors.clone()` here, and (if you also want to track `PruningPredicate::prune` errors) add a separate counter for evaluation errors instead of reusing the creation counter.
##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -94,6 +101,118 @@ impl DecoderBuilderConfig<'_> {
}
}
+/// A decoder queued to run after the current one finishes, together with
+/// the metadata needed to decide whether it should be skipped.
+///
+/// We keep the row group indices alongside the decoder so that, between
+/// runs, the stream state can apply runtime decisions — e.g. dynamic
+/// `TopK`-driven RG-level pruning — by inspecting which row groups a
+/// pending run covers without having to crack the decoder open.
+#[derive(Debug)]
+pub(crate) struct PendingDecoderRun {
+ pub(crate) decoder: ParquetPushDecoder,
+ /// Row group indices this decoder will scan.
+ pub(crate) row_group_indices: Vec<usize>,
+}
+
+/// Runtime row-group pruner driven by a dynamic predicate (e.g. the
+/// threshold expression a `TopK` operator pushes down).
+///
+/// Mirrors the [`FilePruner`](datafusion_pruning::FilePruner) pattern at
+/// the row-group level: tracks the predicate's `snapshot_generation` so it
+/// only rebuilds the [`PruningPredicate`] when the dynamic filter has
+/// actually moved, then evaluates the cached predicate against the
+/// statistics of the requested row groups.
+pub(crate) struct RowGroupPruner {
+ predicate: Arc<dyn PhysicalExpr>,
+ arrow_schema: SchemaRef,
+ parquet_metadata: Arc<ParquetMetaData>,
+ /// Last predicate `snapshot_generation` we built `pruning_predicate` for.
+ last_generation: Option<u64>,
+ /// Cached pruning predicate. `None` means we couldn't build one for the
+ /// current generation (e.g. the predicate has no analyzable bounds);
+ /// in that case we conservatively don't prune.
+ pruning_predicate: Option<Arc<PruningPredicate>>,
+ /// Metric for pruning-predicate construction failures.
+ predicate_creation_errors: Count,
+}
+
+impl RowGroupPruner {
+ pub(crate) fn new(
+ predicate: Arc<dyn PhysicalExpr>,
+ arrow_schema: SchemaRef,
+ parquet_metadata: Arc<ParquetMetaData>,
+ predicate_creation_errors: Count,
+ ) -> Self {
+ Self {
+ predicate,
+ arrow_schema,
+ parquet_metadata,
+ last_generation: None,
+ pruning_predicate: None,
+ predicate_creation_errors,
+ }
+ }
+
+ /// Returns `true` when the statistics for `row_group_indices` prove that
+ /// every requested row group can be skipped under the current value of
+ /// the dynamic predicate.
+ ///
+ /// On any error (predicate construction, statistics evaluation) the
+ /// pruner conservatively returns `false` and logs the failure, so a
+ /// flaky pruning path never silently drops data.
+ pub(crate) fn should_prune(&mut self, row_group_indices: &[usize]) -> bool {
+ if row_group_indices.is_empty() {
+ return false;
+ }
+
+ // Refresh the cached `PruningPredicate` if the dynamic filter has
+ // moved since we last evaluated it.
+ let new_generation = snapshot_generation(&self.predicate);
+ if self.last_generation != Some(new_generation) {
+ self.pruning_predicate = build_pruning_predicate(
+ Arc::clone(&self.predicate),
+ &self.arrow_schema,
+ &self.predicate_creation_errors,
+ );
+ self.last_generation = Some(new_generation);
+ }
+
+ let Some(pp) = self.pruning_predicate.as_ref() else {
+ return false;
+ };
+
+ let row_group_metadatas = row_group_indices
+ .iter()
+ .map(|&i| self.parquet_metadata.row_group(i))
+ .collect::<Vec<_>>();
+ let stats = RowGroupPruningStatistics {
+ parquet_schema: self.parquet_metadata.file_metadata().schema_descr(),
+ row_group_metadatas,
+ arrow_schema: self.arrow_schema.as_ref(),
+ // Match the existing static row-group pruning behavior: when a
+ // statistic's null count is missing, treat it as zero. This is
+ // sound for runtime pruning because the predicate only needs to
+ // prove a row group *cannot* contain matching rows.
+ missing_null_counts_as_zero: true,
+ };
+
+ match pp.prune(&stats) {
+ // `prune` returns `false` per container that the predicate proves
+ // cannot contain matching rows. We can skip the run only when
+ // every requested row group is in that state.
+ Ok(values) => values.iter().all(|&keep| !keep),
+ Err(e) => {
+ debug!(
+ "Ignoring error building runtime row-group pruning predicate: {e}"
+ );
+ self.predicate_creation_errors.add(1);
+ false
+ }
Review Comment:
`RowGroupPruner::should_prune` logs "Ignoring error building runtime row-group pruning predicate" and increments `predicate_creation_errors` when `pp.prune(&stats)` returns `Err`. At this point the pruning predicate is already built; the error is from evaluating it against row-group statistics. This both makes the log message misleading and conflates predicate *creation* vs predicate *evaluation* errors. Consider logging this as an evaluation error and incrementing the existing `predicate_evaluation_errors` metric (keeping `predicate_creation_errors` only for `build_pruning_predicate` failures).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515444212
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 37.91 / 39.47 ±2.07 / 43.42 ms │ 37.82 / 38.74 ±0.94 / 40.41 ms │ no change │
│ QQuery 2 │ 19.53 / 19.69 ±0.21 / 20.10 ms │ 19.91 / 20.59 ±0.57 / 21.21 ms │ no change │
│ QQuery 3 │ 34.33 / 35.15 ±0.46 / 35.69 ms │ 32.22 / 34.22 ±1.50 / 35.76 ms │ no change │
│ QQuery 4 │ 16.91 / 17.09 ±0.17 / 17.40 ms │ 17.03 / 17.66 ±0.67 / 18.79 ms │ no change │
│ QQuery 5 │ 41.39 / 41.63 ±0.22 / 42.01 ms │ 39.52 / 40.94 ±0.75 / 41.54 ms │ no change │
│ QQuery 6 │ 15.93 / 15.99 ±0.06 / 16.09 ms │ 15.90 / 16.11 ±0.16 / 16.33 ms │ no change │
│ QQuery 7 │ 46.71 / 49.37 ±3.17 / 55.06 ms │ 45.70 / 47.23 ±1.55 / 49.93 ms │ no change │
│ QQuery 8 │ 43.93 / 44.87 ±0.82 / 46.09 ms │ 44.03 / 44.42 ±0.37 / 45.10 ms │ no change │
│ QQuery 9 │ 48.69 / 50.08 ±1.04 / 51.92 ms │ 48.60 / 49.70 ±0.91 / 51.25 ms │ no change │
│ QQuery 10 │ 63.20 / 63.42 ±0.21 / 63.74 ms │ 62.93 / 63.36 ±0.39 / 64.02 ms │ no change │
│ QQuery 11 │ 13.16 / 13.34 ±0.16 / 13.64 ms │ 13.04 / 13.27 ±0.26 / 13.77 ms │ no change │
│ QQuery 12 │ 23.70 / 24.54 ±0.93 / 26.30 ms │ 23.40 / 24.02 ±0.42 / 24.44 ms │ no change │
│ QQuery 13 │ 33.54 / 35.52 ±1.26 / 37.06 ms │ 33.30 / 35.22 ±1.09 / 36.63 ms │ no change │
│ QQuery 14 │ 24.96 / 25.10 ±0.09 / 25.20 ms │ 24.90 / 25.37 ±0.64 / 26.62 ms │ no change │
│ QQuery 15 │ 30.72 / 30.88 ±0.08 / 30.95 ms │ 30.34 / 30.93 ±0.49 / 31.80 ms │ no change │
│ QQuery 16 │ 14.44 / 14.65 ±0.16 / 14.84 ms │ 14.67 / 14.84 ±0.24 / 15.30 ms │ no change │
│ QQuery 17 │ 72.04 / 73.15 ±1.03 / 74.89 ms │ 74.86 / 75.90 ±0.62 / 76.76 ms │ no change │
│ QQuery 18 │ 61.21 / 62.59 ±1.05 / 63.75 ms │ 62.12 / 63.07 ±0.64 / 64.05 ms │ no change │
│ QQuery 19 │ 33.14 / 33.64 ±0.83 / 35.29 ms │ 33.47 / 33.73 ±0.34 / 34.40 ms │ no change │
│ QQuery 20 │ 36.90 / 37.57 ±0.77 / 38.86 ms │ 37.22 / 37.47 ±0.24 / 37.89 ms │ no change │
│ QQuery 21 │ 56.14 / 57.72 ±1.20 / 59.48 ms │ 53.82 / 55.70 ±1.59 / 58.18 ms │ no change │
│ QQuery 22 │ 23.10 / 23.81 ±0.50 / 24.61 ms │ 23.31 / 23.99 ±0.93 / 25.83 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 809.27ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 806.50ms │
│ Average Time (HEAD) │ 36.78ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 36.66ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.5 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 29.4s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.5 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 29.6s |
| CPU sys | 1.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4540100662
fwiw that should go out in next minor release, so 2-4 weeks until it's available in DataFusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515300900
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515274607)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4515274607-276-gkpr8 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (691926f605ec05196e3235f4f1d036ae4d981b8d) to 077f08a (merge-base) [diff](https://github.com/apache/datafusion/compare/077f08a9a6632324c95275dd15b5dd5b1f14006f..691926f605ec05196e3235f4f1d036ae4d981b8d) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515641255
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515622680)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4515622680-278-bcnbp 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (0828f1b5a97d26e527cb33b0efa840113e266290) to a8f03fd (merge-base) [diff](https://github.com/apache/datafusion/compare/a8f03fdae5b27465a6e4389017c43cf06b4c59ac..0828f1b5a97d26e527cb33b0efa840113e266290) using: sort_pushdown_inexact
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515744044
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4515622680)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark sort_pushdown_inexact.json
--------------------
┏━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ Q1 │ FAIL │ FAIL │ incomparable │
│ Q2 │ FAIL │ FAIL │ incomparable │
│ Q3 │ FAIL │ FAIL │ incomparable │
│ Q4 │ FAIL │ FAIL │ incomparable │
└───────┴──────┴────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD) │ 0.00ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 0.00ms │
│ Average Time (HEAD) │ 0.00ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 0.00ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 0 │
│ Queries with Failure │ 4 │
└───────────────────────────────────────────────────┴────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sort_pushdown_inexact — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 4.1 GiB |
| Avg memory | 4.1 GiB |
| CPU user | 0.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**sort_pushdown_inexact — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 4.1 GiB |
| Avg memory | 4.1 GiB |
| CPU user | 0.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4516194708
> Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark run_topk_tpch.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 2.14 / 2.74 ±0.76 / 4.10 ms │ 2.12 / 2.79 ±0.68 / 4.02 ms │ no change │
│ Q2 │ 10.66 / 11.36 ±0.68 / 12.23 ms │ 2.81 / 3.61 ±0.87 / 4.72 ms │ +3.15x faster │
│ Q3 │ 31.77 / 32.15 ±0.43 / 32.83 ms │ 31.71 / 31.92 ±0.16 / 32.18 ms │ no change │
│ Q4 │ 11.83 / 12.29 ±0.77 / 13.82 ms │ 3.13 / 3.25 ±0.13 / 3.48 ms │ +3.78x faster │
│ Q5 │ 9.94 / 10.14 ±0.18 / 10.46 ms │ 9.95 / 10.02 ±0.05 / 10.09 ms │ no change │
│ Q6 │ 17.19 / 17.39 ±0.15 / 17.56 ms │ 17.11 / 17.36 ±0.37 / 18.09 ms │ no change │
│ Q7 │ 37.07 / 38.08 ±1.17 / 40.08 ms │ 37.00 / 37.41 ±0.37 / 38.07 ms │ no change │
│ Q8 │ 28.13 / 28.59 ±0.60 / 29.71 ms │ 6.86 / 7.16 ±0.42 / 7.98 ms │ +3.99x faster │
│ Q9 │ 35.34 / 36.86 ±1.54 / 38.77 ms │ 8.36 / 8.50 ±0.08 / 8.60 ms │ +4.34x faster │
│ Q10 │ 54.13 / 55.29 ±1.83 / 58.93 ms │ 12.77 / 13.00 ±0.45 / 13.89 ms │ +4.25x faster │
│ Q11 │ 3.75 / 3.91 ±0.11 / 4.05 ms │ 3.82 / 4.08 ±0.31 / 4.68 ms │ no change │
└───────┴────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 248.79ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 139.08ms │
│ Average Time (HEAD) │ 22.62ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 12.64ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴──────────┘
cc @alamb @adriangb @Dandandan
This is matching my local test, also sort_pushdown_inexact will improve a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4518793959
Is this expected?
https://github.com/apache/datafusion/pull/22450#issuecomment-4515744044
```
Benchmark sort_pushdown_inexact.json
--------------------
┏━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ Q1 │ FAIL │ FAIL │ incomparable │
│ Q2 │ FAIL │ FAIL │ incomparable │
│ Q3 │ FAIL │ FAIL │ incomparable │
│ Q4 │ FAIL │ FAIL │ incomparable │
└───────┴──────┴────────────────────────────────────┴──────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4519831784
I did a little experiment: https://github.com/apache/datafusion/pull/22460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4720450185
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4720419417)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4720419417-573-n6wh9 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (6c554cce66f8bf6126f821bb8822f740d24287cc) to 6176a6d (merge-base) [diff](https://github.com/apache/datafusion/compare/6176a6dae1f36e30c83bc880bb8b6cb1d1a175a8..6c554cce66f8bf6126f821bb8822f740d24287cc) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4725531237
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4725518286)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4725518286-578-2sdzh 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (608591f8a8aa670954feb7061c4c641abc64e7b8) to 96a6096 (merge-base) [diff](https://github.com/apache/datafusion/compare/96a6096c6f4b924e8cab4bc1629759a948e12939..608591f8a8aa670954feb7061c4c641abc64e7b8) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4725518286
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4705710247
All three done:
- Tightened the `RowGroupPruner` gate to `DynamicFilterTracking::classify == Watching` so static predicates don't pay for runtime re-pruning.
- Comment pass — `snapshot_generation` reference dropped, multi-decoder / split-runs language replaced with the single-decoder + `into_builder` flow throughout `push_decoder.rs`, `source.rs`, `opener/mod.rs`, `decoder_projection.rs`.
- PR description rewritten to match the final design.
- Restacked the branch into a single atomic commit (`ef23530f9`).
Ready for another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4667821416
Happy to see https://github.com/apache/datafusion/pull/22744 is ready, i will rework this after it merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4678169051
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v54.0.0 (current)
Built [ 57.685s] (current)
Parsing datafusion v54.0.0 (current)
Parsed [ 0.024s] (current)
Building datafusion v54.0.0 (baseline)
Built [ 57.506s] (baseline)
Parsing datafusion v54.0.0 (baseline)
Parsed [ 0.024s] (baseline)
Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.399s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 116.996s] datafusion
Building datafusion-datasource-parquet v54.0.0 (current)
Built [ 26.479s] (current)
Parsing datafusion-datasource-parquet v54.0.0 (current)
Parsed [ 0.020s] (current)
Building datafusion-datasource-parquet v54.0.0 (baseline)
Built [ 26.540s] (baseline)
Parsing datafusion-datasource-parquet v54.0.0 (baseline)
Parsed [ 0.020s] (baseline)
Checking datafusion-datasource-parquet v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.094s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ParquetFileMetrics.row_groups_pruned_dynamic_filter in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/metrics.rs:63
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 54.033s] datafusion-datasource-parquet
Building datafusion-sqllogictest v54.0.0 (current)
Built [ 97.377s] (current)
Parsing datafusion-sqllogictest v54.0.0 (current)
Parsed [ 0.016s] (current)
Building datafusion-sqllogictest v54.0.0 (baseline)
Built [ 97.843s] (baseline)
Parsing datafusion-sqllogictest v54.0.0 (baseline)
Parsed [ 0.017s] (baseline)
Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.062s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 197.364s] datafusion-sqllogictest
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3421025319
##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -151,7 +305,72 @@ impl PushDecoderStreamState {
if self.remaining_limit == Some(0) {
return None;
}
- match self.decoder.try_decode() {
+
+ // Step 1: drain a batch from the active reader if any.
+ if let Some(reader) = self.active_reader.as_mut() {
+ match reader.next() {
+ Some(Ok(batch)) => {
+ let mut timer = self.baseline_metrics.elapsed_compute().timer();
+ self.copy_arrow_reader_metrics();
+ let result = self.project_batch(&batch);
+ timer.stop();
+ drop(timer);
+ return Some((result, self));
+ }
+ Some(Err(e)) => {
+ return Some((Err(DataFusionError::from(e)), self));
+ }
+ None => {
+ // Reader exhausted: drop and fall through to per-RG
+ // boundary handling, then try_next_reader.
+ self.active_reader = None;
+ }
+ }
+ }
+
+ // Step 2: at RG boundary, drop pruned RGs from the head of the
+ // plan. Rebuild the decoder via `into_builder` so it skips the
+ // pruned RGs entirely. Buffered bytes for already-fetched RGs
+ // carry across the rebuild.
+ if !self.rg_plan.is_empty() {
+ let mut pruned_head = 0usize;
+ while let Some(next) = self.rg_plan.front() {
+ let pruned = self
+ .row_group_pruner
+ .as_mut()
+ .map(|p| p.should_prune(&[next.rg_index]))
+ .unwrap_or(false);
+ if pruned {
+ pruned_head += 1;
+ self.rg_plan.pop_front();
+ self.row_groups_pruned_dynamic.add(1);
+ } else {
+ break;
+ }
+ }
+ if pruned_head > 0 {
+ if self.rg_plan.is_empty() {
+ return None;
+ }
+ let decoder = self.decoder.take().expect("decoder present");
+ let new_indices: Vec<usize> =
+ self.rg_plan.iter().map(|e| e.rg_index).collect();
+ let rebuilt = match decoder.into_builder() {
+ Ok(b) => b.with_row_groups(new_indices).build(),
+ Err(e) => Err(e),
+ };
Review Comment:
Added test in latest PR, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4719265440
Addressed all comments besides:
> split_runs used to build a different decoder for each row group when there was row groups that fully matched filters, allowing those to skip applying the row filter. I think we're loosing that now?
I tried this before, but it failed for testing, i am trying more for this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4702282986
> Makes sense thanks! Could you do a pass of updating comments + PR description and maybe restacking the commits for easier review?
I agree, PR description updated. I'll do the comment pass + commit restack tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#discussion_r3409746016
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1419,16 +1412,40 @@ impl RowGroupsPrunedParquetOpen {
let files_ranges_pruned_statistics =
prepared.file_metrics.files_ranges_pruned_statistics.clone();
+
+ // Build a dynamic row-group pruner only when both halves are useful:
+ // 1) the scan has a predicate (so there is something to evaluate),
+ // 2) there is at least one pending run that could be skipped.
+ // The pruner consults the predicate's `snapshot_generation` so the
+ // cost is one rebuild per dynamic-filter update, not per RG check.
Review Comment:
Is this comment about `snapshot_generation` still current? I thought https://github.com/apache/datafusion/pull/22460 superseeded that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4762006072
> Thanks @zhuqi-lucas, I think this is ready!
Thank you @adriangb for review, added follow-up issue for full-matched improvement
https://github.com/apache/datafusion/issues/23067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765072688
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765059254)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4765059254-601-wd6jz 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (23339c539055c69a5157f8b00de7581c862810b7) to ba67bb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/ba67bb467a30c3dbe7f78089451ea4c4d1a191d5..23339c539055c69a5157f8b00de7581c862810b7) using: topk_tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765073023
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4765060057-603-bk2pz 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing feat/topk-rg-level-dynamic-pruning (23339c539055c69a5157f8b00de7581c862810b7) to ba67bb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/ba67bb467a30c3dbe7f78089451ea4c4d1a191d5..23339c539055c69a5157f8b00de7581c862810b7) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765169179
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 5.62 / 6.09 ±0.79 / 7.65 ms │ 5.66 / 6.15 ±0.85 / 7.86 ms │ no change │
│ QQuery 2 │ 81.19 / 81.59 ±0.22 / 81.83 ms │ 81.71 / 81.94 ±0.21 / 82.31 ms │ no change │
│ QQuery 3 │ 29.00 / 29.32 ±0.18 / 29.52 ms │ 28.98 / 29.29 ±0.38 / 30.01 ms │ no change │
│ QQuery 4 │ 473.77 / 475.90 ±1.52 / 478.03 ms │ 475.09 / 477.87 ±2.18 / 480.48 ms │ no change │
│ QQuery 5 │ 51.37 / 53.73 ±2.79 / 58.50 ms │ 51.48 / 52.24 ±0.59 / 53.17 ms │ no change │
│ QQuery 6 │ 36.22 / 36.92 ±0.42 / 37.28 ms │ 36.34 / 36.67 ±0.22 / 36.98 ms │ no change │
│ QQuery 7 │ 93.95 / 95.02 ±1.24 / 97.09 ms │ 95.21 / 95.85 ±0.44 / 96.33 ms │ no change │
│ QQuery 8 │ 36.45 / 38.05 ±2.78 / 43.59 ms │ 36.39 / 38.37 ±1.94 / 41.93 ms │ no change │
│ QQuery 9 │ 54.17 / 55.59 ±1.27 / 57.78 ms │ 52.58 / 55.68 ±2.20 / 57.92 ms │ no change │
│ QQuery 10 │ 62.60 / 63.40 ±0.52 / 64.18 ms │ 63.55 / 64.31 ±0.44 / 64.72 ms │ no change │
│ QQuery 11 │ 289.50 / 290.88 ±0.98 / 292.49 ms │ 292.63 / 296.88 ±3.37 / 301.80 ms │ no change │
│ QQuery 12 │ 27.94 / 28.43 ±0.44 / 29.24 ms │ 28.37 / 28.81 ±0.26 / 29.11 ms │ no change │
│ QQuery 13 │ 117.48 / 118.05 ±0.50 / 118.93 ms │ 118.04 / 118.40 ±0.19 / 118.57 ms │ no change │
│ QQuery 14 │ 418.80 / 421.21 ±1.77 / 424.29 ms │ 415.13 / 418.26 ±3.19 / 423.66 ms │ no change │
│ QQuery 15 │ 59.28 / 60.60 ±1.54 / 63.51 ms │ 58.63 / 60.80 ±2.36 / 65.36 ms │ no change │
│ QQuery 16 │ 6.86 / 7.03 ±0.18 / 7.38 ms │ 6.85 / 7.03 ±0.22 / 7.46 ms │ no change │
│ QQuery 17 │ 80.35 / 81.17 ±0.69 / 82.21 ms │ 80.46 / 81.48 ±0.80 / 82.72 ms │ no change │
│ QQuery 18 │ 124.75 / 125.30 ±0.34 / 125.76 ms │ 124.29 / 125.89 ±1.39 / 128.32 ms │ no change │
│ QQuery 19 │ 40.96 / 41.33 ±0.45 / 42.22 ms │ 41.26 / 41.88 ±0.58 / 42.94 ms │ no change │
│ QQuery 20 │ 34.74 / 35.23 ±0.46 / 35.97 ms │ 35.26 / 35.69 ±0.26 / 36.00 ms │ no change │
│ QQuery 21 │ 17.69 / 17.87 ±0.27 / 18.39 ms │ 17.47 / 18.11 ±0.57 / 19.17 ms │ no change │
│ QQuery 22 │ 61.96 / 62.36 ±0.41 / 63.07 ms │ 61.96 / 62.84 ±0.70 / 64.04 ms │ no change │
│ QQuery 23 │ 345.18 / 348.59 ±2.44 / 352.55 ms │ 343.29 / 347.31 ±2.69 / 351.60 ms │ no change │
│ QQuery 24 │ 224.11 / 227.81 ±4.27 / 236.18 ms │ 224.30 / 228.28 ±4.47 / 236.47 ms │ no change │
│ QQuery 25 │ 109.73 / 111.46 ±1.37 / 113.82 ms │ 111.04 / 111.70 ±0.80 / 113.27 ms │ no change │
│ QQuery 26 │ 57.44 / 59.19 ±2.62 / 64.36 ms │ 58.11 / 59.47 ±1.80 / 63.01 ms │ no change │
│ QQuery 27 │ 6.34 / 6.48 ±0.19 / 6.84 ms │ 6.39 / 6.54 ±0.14 / 6.80 ms │ no change │
│ QQuery 28 │ 57.78 / 61.31 ±2.02 / 63.76 ms │ 60.94 / 61.28 ±0.21 / 61.55 ms │ no change │
│ QQuery 29 │ 97.05 / 98.16 ±1.40 / 100.92 ms │ 97.13 / 98.57 ±1.58 / 101.56 ms │ no change │
│ QQuery 30 │ 32.09 / 32.78 ±0.74 / 34.03 ms │ 32.15 / 34.15 ±2.40 / 38.78 ms │ no change │
│ QQuery 31 │ 110.58 / 110.99 ±0.32 / 111.49 ms │ 111.13 / 112.20 ±0.75 / 112.99 ms │ no change │
│ QQuery 32 │ 20.30 / 20.63 ±0.39 / 21.35 ms │ 20.49 / 20.81 ±0.33 / 21.42 ms │ no change │
│ QQuery 33 │ 37.40 / 37.78 ±0.26 / 38.19 ms │ 37.88 / 38.17 ±0.21 / 38.50 ms │ no change │
│ QQuery 34 │ 9.85 / 10.08 ±0.22 / 10.47 ms │ 9.83 / 10.03 ±0.18 / 10.33 ms │ no change │
│ QQuery 35 │ 72.52 / 73.15 ±0.46 / 73.73 ms │ 72.79 / 73.69 ±0.65 / 74.81 ms │ no change │
│ QQuery 36 │ 5.95 / 6.07 ±0.15 / 6.36 ms │ 5.99 / 6.12 ±0.15 / 6.40 ms │ no change │
│ QQuery 37 │ 7.08 / 7.14 ±0.06 / 7.26 ms │ 7.08 / 7.14 ±0.05 / 7.21 ms │ no change │
│ QQuery 38 │ 62.33 / 62.74 ±0.42 / 63.47 ms │ 62.14 / 62.63 ±0.41 / 63.31 ms │ no change │
│ QQuery 39 │ 445.26 / 447.83 ±2.33 / 451.31 ms │ 442.45 / 447.22 ±2.82 / 451.34 ms │ no change │
│ QQuery 40 │ 23.25 / 24.70 ±2.19 / 29.06 ms │ 23.38 / 24.86 ±2.17 / 29.17 ms │ no change │
│ QQuery 41 │ 11.48 / 11.63 ±0.13 / 11.87 ms │ 11.76 / 11.98 ±0.24 / 12.36 ms │ no change │
│ QQuery 42 │ 23.31 / 23.85 ±0.33 / 24.15 ms │ 23.36 / 24.33 ±0.86 / 25.43 ms │ no change │
│ QQuery 43 │ 4.89 / 5.29 ±0.65 / 6.59 ms │ 5.06 / 5.16 ±0.10 / 5.35 ms │ no change │
│ QQuery 44 │ 9.25 / 9.45 ±0.11 / 9.58 ms │ 9.63 / 9.74 ±0.07 / 9.83 ms │ no change │
│ QQuery 45 │ 39.32 / 39.69 ±0.29 / 39.99 ms │ 39.32 / 39.43 ±0.13 / 39.66 ms │ no change │
│ QQuery 46 │ 11.63 / 11.85 ±0.15 / 12.02 ms │ 12.07 / 12.24 ±0.11 / 12.34 ms │ no change │
│ QQuery 47 │ 224.34 / 226.77 ±1.72 / 229.36 ms │ 222.76 / 225.38 ±1.43 / 227.10 ms │ no change │
│ QQuery 48 │ 96.04 / 96.98 ±0.97 / 98.75 ms │ 96.62 / 98.60 ±2.78 / 103.97 ms │ no change │
│ QQuery 49 │ 76.17 / 76.43 ±0.23 / 76.75 ms │ 76.19 / 76.88 ±0.57 / 77.77 ms │ no change │
│ QQuery 50 │ 58.38 / 58.87 ±0.40 / 59.43 ms │ 58.75 / 59.16 ±0.32 / 59.54 ms │ no change │
│ QQuery 51 │ 90.80 / 94.41 ±2.04 / 96.20 ms │ 92.05 / 95.48 ±3.46 / 100.80 ms │ no change │
│ QQuery 52 │ 23.27 / 23.67 ±0.33 / 24.15 ms │ 23.82 / 24.12 ±0.23 / 24.43 ms │ no change │
│ QQuery 53 │ 29.33 / 29.61 ±0.38 / 30.30 ms │ 29.54 / 29.88 ±0.22 / 30.15 ms │ no change │
│ QQuery 54 │ 55.19 / 55.40 ±0.24 / 55.84 ms │ 54.83 / 55.77 ±0.49 / 56.20 ms │ no change │
│ QQuery 55 │ 23.06 / 23.23 ±0.18 / 23.51 ms │ 23.28 / 23.67 ±0.47 / 24.60 ms │ no change │
│ QQuery 56 │ 38.27 / 38.86 ±0.31 / 39.13 ms │ 39.32 / 40.28 ±0.65 / 41.24 ms │ no change │
│ QQuery 57 │ 176.27 / 178.74 ±2.43 / 183.28 ms │ 175.37 / 176.56 ±0.95 / 178.19 ms │ no change │
│ QQuery 58 │ 118.82 / 119.60 ±0.59 / 120.28 ms │ 114.93 / 116.15 ±1.74 / 119.48 ms │ no change │
│ QQuery 59 │ 117.85 / 118.81 ±1.38 / 121.54 ms │ 118.30 / 119.60 ±1.38 / 122.11 ms │ no change │
│ QQuery 60 │ 40.02 / 40.88 ±0.79 / 42.36 ms │ 39.76 / 40.60 ±0.52 / 41.41 ms │ no change │
│ QQuery 61 │ 12.71 / 12.85 ±0.16 / 13.14 ms │ 12.33 / 12.69 ±0.25 / 13.05 ms │ no change │
│ QQuery 62 │ 46.78 / 47.14 ±0.33 / 47.76 ms │ 46.66 / 46.92 ±0.16 / 47.11 ms │ no change │
│ QQuery 63 │ 29.61 / 29.93 ±0.24 / 30.35 ms │ 29.27 / 29.49 ±0.20 / 29.82 ms │ no change │
│ QQuery 64 │ 409.54 / 410.87 ±1.85 / 414.50 ms │ 409.72 / 412.40 ±2.41 / 416.83 ms │ no change │
│ QQuery 65 │ 145.34 / 148.60 ±2.33 / 152.57 ms │ 148.40 / 150.49 ±2.31 / 154.89 ms │ no change │
│ QQuery 66 │ 79.00 / 81.68 ±2.54 / 86.48 ms │ 79.44 / 79.95 ±0.53 / 80.80 ms │ no change │
│ QQuery 67 │ 234.21 / 239.10 ±3.57 / 245.34 ms │ 234.65 / 240.22 ±5.68 / 248.94 ms │ no change │
│ QQuery 68 │ 11.89 / 12.11 ±0.20 / 12.49 ms │ 11.99 / 12.13 ±0.21 / 12.55 ms │ no change │
│ QQuery 69 │ 56.90 / 57.24 ±0.25 / 57.53 ms │ 58.03 / 58.42 ±0.33 / 58.87 ms │ no change │
│ QQuery 70 │ 104.46 / 106.33 ±2.35 / 110.92 ms │ 105.04 / 109.07 ±4.71 / 115.25 ms │ no change │
│ QQuery 71 │ 35.04 / 36.98 ±1.80 / 40.01 ms │ 35.34 / 35.86 ±0.29 / 36.17 ms │ no change │
│ QQuery 72 │ 1979.58 / 2099.03 ±98.05 / 2195.83 ms │ 2069.65 / 2121.95 ±61.87 / 2240.07 ms │ no change │
│ QQuery 73 │ 9.53 / 9.87 ±0.25 / 10.24 ms │ 9.75 / 9.92 ±0.24 / 10.39 ms │ no change │
│ QQuery 74 │ 168.13 / 170.07 ±2.12 / 173.88 ms │ 165.64 / 168.38 ±2.34 / 172.30 ms │ no change │
│ QQuery 75 │ 150.06 / 151.81 ±1.94 / 154.88 ms │ 148.84 / 151.45 ±2.68 / 156.47 ms │ no change │
│ QQuery 76 │ 34.92 / 35.30 ±0.26 / 35.73 ms │ 35.29 / 35.53 ±0.14 / 35.71 ms │ no change │
│ QQuery 77 │ 60.43 / 60.84 ±0.35 / 61.27 ms │ 60.54 / 62.10 ±2.58 / 67.24 ms │ no change │
│ QQuery 78 │ 184.00 / 187.89 ±3.63 / 194.74 ms │ 184.89 / 188.14 ±4.09 / 196.01 ms │ no change │
│ QQuery 79 │ 67.14 / 68.74 ±1.62 / 71.18 ms │ 68.36 / 70.09 ±2.00 / 73.69 ms │ no change │
│ QQuery 80 │ 98.47 / 99.61 ±1.12 / 101.32 ms │ 98.80 / 100.23 ±1.35 / 102.36 ms │ no change │
│ QQuery 81 │ 25.69 / 25.98 ±0.25 / 26.42 ms │ 26.00 / 26.15 ±0.17 / 26.46 ms │ no change │
│ QQuery 82 │ 16.26 / 16.49 ±0.26 / 16.99 ms │ 16.53 / 18.15 ±2.91 / 23.96 ms │ 1.10x slower │
│ QQuery 83 │ 40.28 / 43.58 ±3.39 / 48.63 ms │ 40.72 / 41.63 ±1.35 / 44.32 ms │ no change │
│ QQuery 84 │ 30.33 / 30.64 ±0.56 / 31.76 ms │ 30.53 / 30.92 ±0.34 / 31.37 ms │ no change │
│ QQuery 85 │ 107.43 / 108.03 ±0.49 / 108.91 ms │ 108.39 / 108.62 ±0.15 / 108.86 ms │ no change │
│ QQuery 86 │ 24.68 / 24.91 ±0.23 / 25.26 ms │ 24.98 / 26.55 ±2.52 / 31.55 ms │ 1.07x slower │
│ QQuery 87 │ 62.23 / 63.59 ±0.85 / 64.75 ms │ 62.57 / 63.58 ±0.80 / 64.56 ms │ no change │
│ QQuery 88 │ 62.87 / 63.73 ±0.48 / 64.27 ms │ 62.79 / 63.29 ±0.55 / 64.31 ms │ no change │
│ QQuery 89 │ 35.98 / 36.23 ±0.20 / 36.48 ms │ 35.57 / 36.27 ±0.80 / 37.78 ms │ no change │
│ QQuery 90 │ 17.17 / 17.39 ±0.28 / 17.88 ms │ 17.05 / 17.26 ±0.13 / 17.41 ms │ no change │
│ QQuery 91 │ 45.35 / 46.98 ±2.03 / 50.93 ms │ 46.12 / 48.57 ±2.32 / 52.77 ms │ no change │
│ QQuery 92 │ 29.42 / 30.23 ±0.92 / 31.87 ms │ 28.96 / 29.69 ±0.66 / 30.83 ms │ no change │
│ QQuery 93 │ 49.32 / 50.19 ±0.74 / 51.37 ms │ 49.05 / 50.06 ±0.57 / 50.65 ms │ no change │
│ QQuery 94 │ 37.56 / 37.83 ±0.20 / 38.17 ms │ 37.52 / 38.12 ±0.50 / 38.91 ms │ no change │
│ QQuery 95 │ 80.73 / 82.40 ±2.21 / 86.53 ms │ 82.33 / 83.89 ±1.31 / 85.91 ms │ no change │
│ QQuery 96 │ 24.14 / 24.34 ±0.25 / 24.77 ms │ 24.10 / 24.30 ±0.16 / 24.54 ms │ no change │
│ QQuery 97 │ 46.54 / 47.07 ±0.36 / 47.45 ms │ 46.93 / 47.35 ±0.43 / 48.13 ms │ no change │
│ QQuery 98 │ 41.75 / 42.61 ±0.46 / 43.12 ms │ 42.55 / 42.92 ±0.46 / 43.83 ms │ no change │
│ QQuery 99 │ 70.42 / 72.29 ±2.68 / 77.61 ms │ 70.79 / 73.28 ±3.07 / 79.17 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 10186.54ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 10235.71ms │
│ Average Time (HEAD) │ 102.89ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 103.39ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 97 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.5 GiB |
| CPU user | 227.9s |
| CPU sys | 5.3s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.4 GiB |
| CPU user | 231.1s |
| CPU sys | 5.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): intra-file early stopping via statistics + dynamic filters [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4765187527
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/22450#issuecomment-4765060057)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_topk-rg-level-dynamic-pruning
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_topk-rg-level-dynamic-pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.24 / 4.03 ±5.51 / 15.05 ms │ 1.21 / 3.92 ±5.36 / 14.64 ms │ no change │
│ QQuery 1 │ 12.69 / 12.97 ±0.16 / 13.15 ms │ 12.88 / 13.20 ±0.20 / 13.43 ms │ no change │
│ QQuery 2 │ 36.21 / 36.49 ±0.22 / 36.75 ms │ 36.18 / 36.54 ±0.37 / 37.23 ms │ no change │
│ QQuery 3 │ 30.31 / 31.00 ±0.56 / 31.98 ms │ 30.04 / 30.30 ±0.21 / 30.65 ms │ no change │
│ QQuery 4 │ 222.89 / 229.69 ±4.95 / 234.98 ms │ 219.89 / 222.08 ±1.82 / 224.74 ms │ no change │
│ QQuery 5 │ 270.38 / 272.50 ±2.32 / 276.79 ms │ 269.09 / 271.81 ±2.17 / 275.26 ms │ no change │
│ QQuery 6 │ 1.27 / 1.41 ±0.22 / 1.84 ms │ 1.27 / 1.41 ±0.22 / 1.85 ms │ no change │
│ QQuery 7 │ 14.05 / 14.22 ±0.11 / 14.36 ms │ 14.27 / 14.43 ±0.10 / 14.59 ms │ no change │
│ QQuery 8 │ 316.25 / 320.40 ±2.98 / 325.26 ms │ 317.45 / 320.26 ±2.05 / 322.22 ms │ no change │
│ QQuery 9 │ 449.15 / 455.54 ±3.73 / 460.54 ms │ 447.09 / 459.76 ±9.54 / 470.07 ms │ no change │
│ QQuery 10 │ 69.25 / 70.09 ±0.80 / 71.45 ms │ 71.44 / 72.30 ±0.85 / 73.85 ms │ no change │
│ QQuery 11 │ 80.75 / 81.68 ±1.08 / 83.62 ms │ 82.93 / 83.76 ±1.17 / 86.07 ms │ no change │
│ QQuery 12 │ 263.55 / 270.11 ±4.50 / 276.75 ms │ 269.47 / 274.60 ±4.89 / 283.29 ms │ no change │
│ QQuery 13 │ 361.52 / 370.59 ±12.33 / 394.71 ms │ 371.32 / 377.77 ±5.93 / 386.93 ms │ no change │
│ QQuery 14 │ 280.35 / 286.21 ±4.62 / 291.89 ms │ 283.51 / 290.26 ±7.91 / 305.67 ms │ no change │
│ QQuery 15 │ 261.79 / 271.19 ±7.05 / 279.73 ms │ 273.69 / 279.25 ±5.23 / 288.35 ms │ no change │
│ QQuery 16 │ 602.49 / 609.32 ±8.53 / 625.56 ms │ 619.81 / 623.18 ±3.96 / 629.53 ms │ no change │
│ QQuery 17 │ 611.06 / 624.86 ±12.33 / 647.95 ms │ 616.48 / 632.77 ±13.77 / 651.36 ms │ no change │
│ QQuery 18 │ 1234.70 / 1254.81 ±15.16 / 1280.39 ms │ 1252.78 / 1295.32 ±23.82 / 1315.10 ms │ no change │
│ QQuery 19 │ 27.62 / 28.37 ±0.98 / 30.27 ms │ 27.70 / 35.46 ±15.36 / 66.19 ms │ 1.25x slower │
│ QQuery 20 │ 513.58 / 525.46 ±11.01 / 542.39 ms │ 513.52 / 526.51 ±18.81 / 563.83 ms │ no change │
│ QQuery 21 │ 517.42 / 520.65 ±2.24 / 524.22 ms │ 515.44 / 523.79 ±5.12 / 529.40 ms │ no change │
│ QQuery 22 │ 986.28 / 995.95 ±7.14 / 1002.04 ms │ 986.83 / 1002.47 ±11.29 / 1016.82 ms │ no change │
│ QQuery 23 │ 3158.33 / 3198.83 ±24.03 / 3221.89 ms │ 3055.14 / 3080.77 ±19.45 / 3109.25 ms │ no change │
│ QQuery 24 │ 41.37 / 42.32 ±0.84 / 43.84 ms │ 41.82 / 50.60 ±10.35 / 69.61 ms │ 1.20x slower │
│ QQuery 25 │ 111.12 / 113.65 ±3.42 / 120.31 ms │ 112.51 / 114.33 ±2.97 / 120.21 ms │ no change │
│ QQuery 26 │ 43.16 / 46.62 ±4.02 / 52.73 ms │ 42.33 / 42.75 ±0.44 / 43.29 ms │ +1.09x faster │
│ QQuery 27 │ 673.00 / 677.92 ±4.92 / 685.85 ms │ 671.07 / 676.26 ±3.93 / 681.10 ms │ no change │
│ QQuery 28 │ 3037.33 / 3046.63 ±5.07 / 3051.52 ms │ 3008.06 / 3019.10 ±9.49 / 3036.76 ms │ no change │
│ QQuery 29 │ 40.88 / 53.64 ±13.05 / 76.72 ms │ 41.22 / 42.09 ±0.69 / 43.03 ms │ +1.27x faster │
│ QQuery 30 │ 300.20 / 305.48 ±4.45 / 312.74 ms │ 300.23 / 305.60 ±5.17 / 313.93 ms │ no change │
│ QQuery 31 │ 293.65 / 296.98 ±2.58 / 300.32 ms │ 283.46 / 300.55 ±11.93 / 317.77 ms │ no change │
│ QQuery 32 │ 924.29 / 936.96 ±9.31 / 952.65 ms │ 940.16 / 958.79 ±11.94 / 975.48 ms │ no change │
│ QQuery 33 │ 1455.86 / 1467.52 ±8.81 / 1479.03 ms │ 1453.42 / 1473.52 ±23.80 / 1515.97 ms │ no change │
│ QQuery 34 │ 1463.37 / 1515.02 ±37.14 / 1567.48 ms │ 1457.45 / 1516.44 ±50.51 / 1588.57 ms │ no change │
│ QQuery 35 │ 284.53 / 322.31 ±30.54 / 362.04 ms │ 276.05 / 327.93 ±80.13 / 485.35 ms │ no change │
│ QQuery 36 │ 65.07 / 69.95 ±5.76 / 80.54 ms │ 66.12 / 77.35 ±16.43 / 109.72 ms │ 1.11x slower │
│ QQuery 37 │ 35.27 / 39.67 ±6.71 / 53.03 ms │ 36.18 / 38.24 ±1.68 / 40.72 ms │ no change │
│ QQuery 38 │ 39.42 / 45.10 ±6.10 / 55.60 ms │ 40.24 / 41.96 ±1.57 / 44.28 ms │ +1.07x faster │
│ QQuery 39 │ 133.85 / 148.16 ±8.08 / 158.48 ms │ 143.88 / 155.02 ±8.76 / 170.57 ms │ no change │
│ QQuery 40 │ 13.92 / 14.53 ±0.83 / 16.09 ms │ 14.31 / 15.73 ±1.88 / 19.27 ms │ 1.08x slower │
│ QQuery 41 │ 13.56 / 13.87 ±0.28 / 14.39 ms │ 13.70 / 14.07 ±0.23 / 14.38 ms │ no change │
│ QQuery 42 │ 12.95 / 14.43 ±2.47 / 19.34 ms │ 13.16 / 13.39 ±0.22 / 13.78 ms │ +1.08x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19657.19ms │
│ Total Time (feat_topk-rg-level-dynamic-pruning) │ 19655.61ms │
│ Average Time (HEAD) │ 457.14ms │
│ Average Time (feat_topk-rg-level-dynamic-pruning) │ 457.11ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 35 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 10.3 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 1003.8s |
| CPU sys | 69.4s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 12.2 GiB |
| Avg memory | 4.7 GiB |
| CPU user | 1005.4s |
| CPU sys | 71.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4540053436
> Thanks! I feel this may be simplified even further by [apache/arrow-rs#9968](https://github.com/apache/arrow-rs/pull/9968)?
Nice @adriangb , that does look much cleaner. `into_builder()` only landed in arrow-rs main after 58.3.0 though, so it isn't available to DataFusion yet. I'll keep the current shape for this PR and follow up with a simplification PR once the next arrow-rs release (carrying #9968) lands here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): runtime row-group early stop via TopK dynamic filter [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #22450:
URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4515622680
run benchmark sort_pushdown_inexact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org