You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/08/18 05:59:05 UTC
[PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
zhuqi-lucas opened a new pull request, #24445:
URL: https://github.com/apache/datafusion/pull/24445
## Which issue does this close?
None yet — happy to file one if that is preferred.
## Rationale for this change
`replace_children_if_necessary` already short-circuits two cases: identical child pointers, and identical child `PlanProperties` pointers. A rule that inserts a sort below a projection satisfies neither, because the child is a new object and so its properties pointer differs. Its *equivalence group*, however, is unchanged: sorting changes which orderings hold, not which expressions are equal to one another.
`Recompute` therefore re-projects a group identical to the one the projection already holds. `EquivalenceGroup::project` is a pure function of the group and the mapping, so when both are unchanged the previous result can be handed back instead of recomputed.
This matters on plans with many wide projections, because the recompute is repeated for every projection above the inserted sort, on every pass of the rule.
## What changes are included in this PR?
1. `EquivalenceGroup` gains `PartialEq`, comparing `classes`. `map` is an index into `classes` and carries no additional information.
2. `EquivalenceProperties::project` splits, so `project_with_eq_group` can take an already-projected group. Orderings are still derived, since they are precisely what changes when a sort appears below.
3. `ProjectionExec::replace_children` compares the old and new child equivalence groups on the `Recompute` path and, when they match, reuses the cached group.
The check is local to `ProjectionExec`. No new `ChildrenPropertiesMode` variant and no change to the shared path, so other operators are unaffected.
## Measurements
A query over a 1191-line view with 38 SELECTs, 117 CASE expressions and 7 joins across 11 tables. Ten warm samples per configuration, identical build flags, the only variable being this patch:
| | before (median) | after (median) | |
| --- | --- | --- | --- |
| EnforceSorting | 197.8ms | 90.1ms | −54% |
| optimizer rules | 338.1ms | 214.6ms | −37% |
| planning wall | 420.6ms | 299.0ms | −29% |
The ranges do not overlap: 196.9–199.2 against 88.8–91.0. Logical rules are unchanged across the two configurations, which acts as the control: the saving lands in the physical phase and nowhere else. `EnforceDistribution` improves as well, since it rebuilds the same projections.
The saving scales with projection count times expression size times rule passes, so plans with narrow projections should see little. I expect `sql_planner` to show a small delta for that reason.
## Are there any user-facing changes?
No. Plans are unchanged.
## Testing
`datafusion-physical-expr` 1594, `datafusion-physical-plan` 1716, `datafusion-physical-optimizer` 33, `datafusion-optimizer` 760, `datafusion-common` 550, `datafusion` core 442, and all 502 sqllogictest files. No test or expected plan was modified.
Worth flagging for review: an earlier revision passed the *projected* equivalence properties to `Partitioning::project` where `compute_properties` passes the input's. Every unit test stayed green; only `range_partitioning.slt` caught it, via the case asserting that a join preserving Range partitioning lets the aggregate above it skip a Hash repartition. It produced a worse plan rather than a wrong answer, which is why nothing else noticed.
## Follow-up
Holding the equivalence group behind an `Arc` would reduce the comparison to `Arc::ptr_eq` and make `EquivalenceProperties::clone` cheaper, which happens on every `SortExec` construction. It touches every site that mutates the group, so it seemed better kept separate. The structural comparison's cost is already inside the numbers above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5324675178
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5324655706)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5324655706-1637-dml25 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (82edbc3b2c6a77922a40d5f467565abc48456c07) to 6eaca8b (merge-base) [diff](https://github.com/apache/datafusion/compare/6eaca8bfe8421faca1413f29f71fcaf0ccaf631a..82edbc3b2c6a77922a40d5f467565abc48456c07)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5324655706
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5339955041
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5339090159)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing qizhu/reuse-eq-group-on-child-swap (95cc5360b98bf7d506c9c004ccfe69917fda6c9e) to c429919 (merge-base) [diff](https://github.com/apache/datafusion/compare/c429919c7c8bd81e4e687531a609e2380ed60649..95cc5360b98bf7d506c9c004ccfe69917fda6c9e)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD qizhu_reuse-eq-group-on-child-swap
----- ---- ----------------------------------
logical_aggregate_with_join 1.00 454.2±2.27µs ? ?/sec 1.00 454.3±2.16µs ? ?/sec
logical_correlated_subquery_exists 1.00 282.6±2.45µs ? ?/sec 1.00 283.2±0.54µs ? ?/sec
logical_correlated_subquery_in 1.00 284.3±1.83µs ? ?/sec 1.00 284.8±0.61µs ? ?/sec
logical_distinct_many_columns 1.00 567.4±1.50µs ? ?/sec 1.01 571.1±4.18µs ? ?/sec
logical_join_4_with_agg_and_filter 1.00 250.9±1.21µs ? ?/sec 1.02 257.1±1.27µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.00 428.4±3.91µs ? ?/sec 1.02 436.4±4.58µs ? ?/sec
logical_join_chain_16 1.00 672.3±4.68µs ? ?/sec 1.00 675.6±3.67µs ? ?/sec
logical_join_chain_4 1.00 120.3±0.55µs ? ?/sec 1.01 121.1±0.44µs ? ?/sec
logical_join_chain_8 1.00 246.9±1.14µs ? ?/sec 1.01 248.9±1.59µs ? ?/sec
logical_multiple_subqueries 1.00 518.6±1.74µs ? ?/sec 1.02 529.5±6.86µs ? ?/sec
logical_nested_cte_4_levels 1.00 262.3±1.86µs ? ?/sec 1.00 263.4±1.65µs ? ?/sec
logical_plan_struct_join_agg_sort 1.03 184.9±0.97µs ? ?/sec 1.00 180.3±2.24µs ? ?/sec
logical_plan_tpcds_all 1.00 94.7±0.92ms ? ?/sec 1.01 95.2±0.50ms ? ?/sec
logical_plan_tpch_all 1.00 6.7±0.03ms ? ?/sec 1.01 6.7±0.02ms ? ?/sec
logical_scalar_subquery 1.00 306.5±2.54µs ? ?/sec 1.02 311.7±2.05µs ? ?/sec
logical_select_all_from_1000 1.00 104.7±0.43ms ? ?/sec 1.00 104.4±0.23ms ? ?/sec
logical_select_one_from_700 1.00 327.3±7.79µs ? ?/sec 1.00 325.8±1.80µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 286.4±1.27µs ? ?/sec 1.00 284.4±1.17µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 271.7±0.58µs ? ?/sec 1.00 271.9±0.79µs ? ?/sec
logical_union_4_branches 1.00 418.9±1.97µs ? ?/sec 1.02 428.4±2.30µs ? ?/sec
logical_union_8_branches 1.00 813.6±9.37µs ? ?/sec 1.01 822.8±3.30µs ? ?/sec
logical_wide_aggregate_100_exprs 1.00 4.5±0.01ms ? ?/sec 1.02 4.6±0.04ms ? ?/sec
logical_wide_case_50_exprs 1.00 2.4±0.02ms ? ?/sec 1.01 2.4±0.00ms ? ?/sec
logical_wide_filter_200_predicates 1.00 1312.2±11.49µs ? ?/sec 1.00 1309.4±6.97µs ? ?/sec
logical_wide_filter_50_predicates 1.00 392.3±2.97µs ? ?/sec 1.00 392.8±2.99µs ? ?/sec
optimizer_correlated_exists 1.00 246.1±1.52µs ? ?/sec 1.00 246.8±1.66µs ? ?/sec
optimizer_join_4_with_agg_filter 1.01 458.6±2.78µs ? ?/sec 1.00 456.3±1.84µs ? ?/sec
optimizer_join_chain_4 1.01 181.8±0.38µs ? ?/sec 1.00 179.9±0.34µs ? ?/sec
optimizer_join_chain_8 1.00 565.1±1.47µs ? ?/sec 1.00 566.0±2.91µs ? ?/sec
optimizer_select_all_from_1000 1.01 6.9±0.04ms ? ?/sec 1.00 6.8±0.02ms ? ?/sec
optimizer_select_one_from_700 1.00 258.5±0.96µs ? ?/sec 1.00 259.0±1.03µs ? ?/sec
optimizer_tpcds_all 1.01 315.4±5.31ms ? ?/sec 1.00 312.7±4.40ms ? ?/sec
optimizer_tpch_all 1.02 17.6±0.26ms ? ?/sec 1.00 17.2±0.11ms ? ?/sec
optimizer_wide_aggregate_100 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
optimizer_wide_filter_200 1.01 3.7±0.05ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_intersection 1.02 600.6±3.11µs ? ?/sec 1.00 588.6±2.28µs ? ?/sec
physical_join_consider_sort 1.02 1052.1±2.87µs ? ?/sec 1.00 1035.7±4.09µs ? ?/sec
physical_join_distinct 1.00 265.9±0.55µs ? ?/sec 1.00 265.0±1.59µs ? ?/sec
physical_many_self_joins 1.00 7.7±0.02ms ? ?/sec 1.00 7.7±0.06ms ? ?/sec
physical_plan_clickbench_all 1.00 134.0±8.24ms ? ?/sec 1.03 137.7±11.67ms ? ?/sec
physical_plan_clickbench_q1 1.00 1398.3±10.12µs ? ?/sec 1.00 1400.3±9.30µs ? ?/sec
physical_plan_clickbench_q10 1.01 2.1±0.01ms ? ?/sec 1.00 2.1±0.05ms ? ?/sec
physical_plan_clickbench_q11 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q12 1.00 2.8±0.27ms ? ?/sec 1.05 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.00 2.1±0.01ms ? ?/sec 1.04 2.1±0.19ms ? ?/sec
physical_plan_clickbench_q14 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q15 1.01 2.2±0.23ms ? ?/sec 1.00 2.2±0.20ms ? ?/sec
physical_plan_clickbench_q16 1.00 1843.2±133.56µs ? ?/sec 1.00 1844.7±122.32µs ? ?/sec
physical_plan_clickbench_q17 1.00 1868.4±9.33µs ? ?/sec 1.03 1932.3±184.01µs ? ?/sec
physical_plan_clickbench_q18 1.21 2.0±0.20ms ? ?/sec 1.00 1681.4±7.59µs ? ?/sec
physical_plan_clickbench_q19 1.08 2.3±0.24ms ? ?/sec 1.00 2.1±0.05ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.0±0.25ms ? ?/sec 1.00 2.0±0.24ms ? ?/sec
physical_plan_clickbench_q20 1.00 1560.3±74.99µs ? ?/sec 1.02 1590.6±105.04µs ? ?/sec
physical_plan_clickbench_q21 1.00 1794.7±8.78µs ? ?/sec 1.00 1787.8±48.55µs ? ?/sec
physical_plan_clickbench_q22 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q23 1.00 2.9±0.26ms ? ?/sec 1.02 2.9±0.12ms ? ?/sec
physical_plan_clickbench_q24 1.01 6.8±0.06ms ? ?/sec 1.00 6.8±0.27ms ? ?/sec
physical_plan_clickbench_q25 1.00 1925.5±19.51µs ? ?/sec 1.00 1920.6±46.23µs ? ?/sec
physical_plan_clickbench_q26 1.02 1782.9±123.40µs ? ?/sec 1.00 1747.7±7.94µs ? ?/sec
physical_plan_clickbench_q27 1.02 1987.3±141.76µs ? ?/sec 1.00 1940.7±8.02µs ? ?/sec
physical_plan_clickbench_q28 1.00 2.5±0.28ms ? ?/sec 1.02 2.6±0.29ms ? ?/sec
physical_plan_clickbench_q29 1.02 2.9±0.34ms ? ?/sec 1.00 2.8±0.31ms ? ?/sec
physical_plan_clickbench_q3 1.00 1741.2±143.93µs ? ?/sec 1.00 1736.8±132.44µs ? ?/sec
physical_plan_clickbench_q30 1.03 16.1±0.24ms ? ?/sec 1.00 15.6±0.27ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.6±0.15ms ? ?/sec 1.02 2.6±0.23ms ? ?/sec
physical_plan_clickbench_q32 1.01 2.5±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q33 1.01 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q34 1.02 2.0±0.26ms ? ?/sec 1.00 1979.0±222.37µs ? ?/sec
physical_plan_clickbench_q35 1.02 1932.0±110.32µs ? ?/sec 1.00 1887.0±9.78µs ? ?/sec
physical_plan_clickbench_q36 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q37 1.04 2.7±0.23ms ? ?/sec 1.00 2.6±0.07ms ? ?/sec
physical_plan_clickbench_q38 1.00 2.7±0.26ms ? ?/sec 1.00 2.7±0.25ms ? ?/sec
physical_plan_clickbench_q39 1.01 2.7±0.02ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q4 1.00 1533.9±104.71µs ? ?/sec 1.00 1537.4±100.89µs ? ?/sec
physical_plan_clickbench_q40 1.02 4.2±0.12ms ? ?/sec 1.00 4.1±0.10ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.0±0.25ms ? ?/sec 1.02 3.1±0.28ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.1±0.02ms ? ?/sec 1.01 3.1±0.16ms ? ?/sec
physical_plan_clickbench_q43 1.05 3.4±0.27ms ? ?/sec 1.00 3.2±0.12ms ? ?/sec
physical_plan_clickbench_q44 1.00 1607.1±8.49µs ? ?/sec 1.02 1645.6±123.80µs ? ?/sec
physical_plan_clickbench_q45 1.00 1580.5±8.57µs ? ?/sec 1.02 1606.9±7.19µs ? ?/sec
physical_plan_clickbench_q46 1.04 2.2±0.25ms ? ?/sec 1.00 2.2±0.24ms ? ?/sec
physical_plan_clickbench_q47 1.00 2.6±0.04ms ? ?/sec 1.06 2.8±0.24ms ? ?/sec
physical_plan_clickbench_q48 1.00 2.9±0.27ms ? ?/sec 1.01 3.0±0.28ms ? ?/sec
physical_plan_clickbench_q49 1.05 3.0±0.26ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_clickbench_q5 1.02 1666.1±37.55µs ? ?/sec 1.00 1639.6±9.00µs ? ?/sec
physical_plan_clickbench_q50 1.00 2.7±0.02ms ? ?/sec 1.02 2.8±0.17ms ? ?/sec
physical_plan_clickbench_q51 1.24 2.6±0.15ms ? ?/sec 1.00 2.1±0.13ms ? ?/sec
physical_plan_clickbench_q6 1.02 1665.2±10.42µs ? ?/sec 1.00 1630.6±6.99µs ? ?/sec
physical_plan_clickbench_q7 1.00 1681.2±180.35µs ? ?/sec 1.00 1673.2±154.63µs ? ?/sec
physical_plan_clickbench_q8 1.00 2.0±0.18ms ? ?/sec 1.01 2.1±0.20ms ? ?/sec
physical_plan_clickbench_q9 1.00 2.1±0.20ms ? ?/sec 1.00 2.1±0.16ms ? ?/sec
physical_plan_struct_join_agg_sort 1.05 1362.3±54.26µs ? ?/sec 1.00 1293.8±2.82µs ? ?/sec
physical_plan_tpcds_all 1.02 738.8±19.59ms ? ?/sec 1.00 725.4±17.51ms ? ?/sec
physical_plan_tpch_all 1.01 46.0±1.07ms ? ?/sec 1.00 45.6±1.94ms ? ?/sec
physical_plan_tpch_q1 1.00 1595.5±34.11µs ? ?/sec 1.00 1595.0±69.56µs ? ?/sec
physical_plan_tpch_q10 1.04 3.1±0.15ms ? ?/sec 1.00 3.0±0.16ms ? ?/sec
physical_plan_tpch_q11 1.02 2.3±0.00ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_tpch_q12 1.00 1285.9±7.09µs ? ?/sec 1.00 1282.8±8.01µs ? ?/sec
physical_plan_tpch_q13 1.00 1064.0±3.79µs ? ?/sec 1.01 1077.0±31.55µs ? ?/sec
physical_plan_tpch_q14 1.05 1474.5±44.59µs ? ?/sec 1.00 1401.6±45.59µs ? ?/sec
physical_plan_tpch_q16 1.04 1673.7±63.51µs ? ?/sec 1.00 1607.8±10.91µs ? ?/sec
physical_plan_tpch_q17 1.00 1690.7±25.52µs ? ?/sec 1.01 1708.0±80.85µs ? ?/sec
physical_plan_tpch_q18 1.01 1987.6±5.34µs ? ?/sec 1.00 1959.0±19.12µs ? ?/sec
physical_plan_tpch_q19 1.02 1941.0±58.96µs ? ?/sec 1.00 1901.1±58.56µs ? ?/sec
physical_plan_tpch_q2 1.01 3.7±0.17ms ? ?/sec 1.00 3.7±0.16ms ? ?/sec
physical_plan_tpch_q20 1.03 2.3±0.14ms ? ?/sec 1.00 2.2±0.11ms ? ?/sec
physical_plan_tpch_q21 1.01 2.9±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_tpch_q22 1.02 1671.8±95.34µs ? ?/sec 1.00 1642.1±91.43µs ? ?/sec
physical_plan_tpch_q3 1.01 1902.4±31.74µs ? ?/sec 1.00 1880.0±46.40µs ? ?/sec
physical_plan_tpch_q4 1.01 1223.1±2.59µs ? ?/sec 1.00 1214.3±2.46µs ? ?/sec
physical_plan_tpch_q5 1.00 3.0±0.15ms ? ?/sec 1.00 3.0±0.13ms ? ?/sec
physical_plan_tpch_q6 1.02 639.2±5.09µs ? ?/sec 1.00 626.7±4.59µs ? ?/sec
physical_plan_tpch_q7 1.01 2.9±0.15ms ? ?/sec 1.00 2.9±0.15ms ? ?/sec
physical_plan_tpch_q8 1.01 3.9±0.01ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
physical_plan_tpch_q9 1.02 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.00 15.7±0.13ms ? ?/sec 1.00 15.6±0.12ms ? ?/sec
physical_select_all_from_1000 1.00 115.9±0.27ms ? ?/sec 1.00 115.7±0.26ms ? ?/sec
physical_select_one_from_700 1.00 763.8±3.27µs ? ?/sec 1.01 769.9±4.12µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 4.3±0.11ms ? ?/sec 1.04 4.4±0.02ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.05 9.1±0.05ms ? ?/sec 1.00 8.7±0.10ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.02 98.5±2.26ms ? ?/sec 1.00 96.2±1.88ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.10 410.3±7.29ms ? ?/sec 1.00 373.9±8.28ms ? ?/sec
physical_theta_join_consider_sort 1.01 1082.0±11.86µs ? ?/sec 1.00 1073.4±14.80µs ? ?/sec
physical_unnest_to_join 1.02 641.8±3.51µs ? ?/sec 1.00 632.0±8.57µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.02 724.8±5.20µs ? ?/sec 1.00 712.6±6.99µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.02 1428.8±7.52µs ? ?/sec 1.00 1398.9±5.17µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.03 439.8±1.58µs ? ?/sec 1.00 426.2±2.53µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.03 541.5±2.08µs ? ?/sec 1.00 525.1±2.50µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.03 585.0±3.14µs ? ?/sec 1.00 569.2±4.88µs ? ?/sec
with_param_values_many_columns 1.01 437.7±3.70µs ? ?/sec 1.00 433.7±2.93µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2145.5s |
| Peak memory | 130.8 MiB |
| Avg memory | 73.9 MiB |
| CPU user | 1883.2s |
| CPU sys | 1.3s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2335.5s |
| Peak memory | 133.7 MiB |
| Avg memory | 69.9 MiB |
| CPU user | 1874.6s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5365324761
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5364827227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing qizhu/reuse-eq-group-on-child-swap (724812422d6d2501886c269f3a1795bd8ad3b409) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..724812422d6d2501886c269f3a1795bd8ad3b409)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD qizhu_reuse-eq-group-on-child-swap
----- ---- ----------------------------------
logical_aggregate_with_join 1.00 447.5±1.09µs ? ?/sec 1.02 454.6±1.18µs ? ?/sec
logical_correlated_subquery_exists 1.00 282.0±0.61µs ? ?/sec 1.01 283.6±0.80µs ? ?/sec
logical_correlated_subquery_in 1.00 283.7±0.84µs ? ?/sec 1.00 284.9±0.59µs ? ?/sec
logical_distinct_many_columns 1.00 564.9±0.87µs ? ?/sec 1.00 565.8±1.97µs ? ?/sec
logical_join_4_with_agg_and_filter 1.00 246.6±1.38µs ? ?/sec 1.00 245.7±1.19µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.00 417.0±1.27µs ? ?/sec 1.00 415.6±1.98µs ? ?/sec
logical_join_chain_16 1.00 667.1±3.78µs ? ?/sec 1.01 675.0±3.23µs ? ?/sec
logical_join_chain_4 1.00 119.8±0.50µs ? ?/sec 1.00 119.6±0.44µs ? ?/sec
logical_join_chain_8 1.00 246.0±1.63µs ? ?/sec 1.00 246.4±1.07µs ? ?/sec
logical_multiple_subqueries 1.00 514.1±1.92µs ? ?/sec 1.00 513.7±1.87µs ? ?/sec
logical_nested_cte_4_levels 1.00 255.7±1.45µs ? ?/sec 1.01 257.4±1.33µs ? ?/sec
logical_plan_struct_join_agg_sort 1.03 177.4±1.15µs ? ?/sec 1.00 171.6±1.13µs ? ?/sec
logical_plan_tpcds_all 1.01 93.5±0.22ms ? ?/sec 1.00 92.2±0.14ms ? ?/sec
logical_plan_tpch_all 1.00 6.4±0.02ms ? ?/sec 1.00 6.4±0.02ms ? ?/sec
logical_scalar_subquery 1.01 305.4±1.38µs ? ?/sec 1.00 303.2±1.40µs ? ?/sec
logical_select_all_from_1000 1.00 103.5±0.81ms ? ?/sec 1.01 104.9±1.44ms ? ?/sec
logical_select_one_from_700 1.00 324.4±1.30µs ? ?/sec 1.01 328.7±1.49µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 283.1±0.70µs ? ?/sec 1.02 289.7±0.64µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 271.4±0.89µs ? ?/sec 1.01 274.3±0.65µs ? ?/sec
logical_union_4_branches 1.00 418.1±1.07µs ? ?/sec 1.01 422.4±0.89µs ? ?/sec
logical_union_8_branches 1.00 801.9±1.60µs ? ?/sec 1.01 807.3±1.91µs ? ?/sec
logical_wide_aggregate_100_exprs 1.00 4.5±0.01ms ? ?/sec 1.00 4.5±0.01ms ? ?/sec
logical_wide_case_50_exprs 1.00 2.4±0.00ms ? ?/sec 1.01 2.4±0.00ms ? ?/sec
logical_wide_filter_200_predicates 1.00 1302.5±8.84µs ? ?/sec 1.01 1318.2±8.07µs ? ?/sec
logical_wide_filter_50_predicates 1.00 390.3±2.47µs ? ?/sec 1.01 394.0±2.15µs ? ?/sec
optimizer_correlated_exists 1.00 248.0±1.27µs ? ?/sec 1.01 251.3±1.06µs ? ?/sec
optimizer_join_4_with_agg_filter 1.02 469.1±2.33µs ? ?/sec 1.00 460.4±1.04µs ? ?/sec
optimizer_join_chain_4 1.01 181.9±0.56µs ? ?/sec 1.00 179.9±0.30µs ? ?/sec
optimizer_join_chain_8 1.00 566.7±1.26µs ? ?/sec 1.00 569.4±1.35µs ? ?/sec
optimizer_select_all_from_1000 1.00 6.8±0.01ms ? ?/sec 1.01 6.8±0.02ms ? ?/sec
optimizer_select_one_from_700 1.00 259.1±1.26µs ? ?/sec 1.02 264.8±0.75µs ? ?/sec
optimizer_tpcds_all 1.00 311.4±0.38ms ? ?/sec 1.00 309.8±0.39ms ? ?/sec
optimizer_tpch_all 1.01 17.5±0.03ms ? ?/sec 1.00 17.2±0.03ms ? ?/sec
optimizer_wide_aggregate_100 1.00 2.3±0.00ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
optimizer_wide_filter_200 1.00 3.6±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_intersection 1.01 604.9±1.93µs ? ?/sec 1.00 601.8±1.91µs ? ?/sec
physical_join_consider_sort 1.00 1054.9±2.33µs ? ?/sec 1.01 1068.7±3.51µs ? ?/sec
physical_join_distinct 1.00 265.9±0.86µs ? ?/sec 1.01 268.9±0.49µs ? ?/sec
physical_many_self_joins 1.00 7.7±0.02ms ? ?/sec 1.02 7.9±0.01ms ? ?/sec
physical_plan_clickbench_all 1.01 128.3±0.42ms ? ?/sec 1.00 126.8±0.56ms ? ?/sec
physical_plan_clickbench_q1 1.00 1385.7±10.79µs ? ?/sec 1.03 1434.1±11.72µs ? ?/sec
physical_plan_clickbench_q10 1.01 2.1±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q11 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_clickbench_q12 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q13 1.00 2.1±0.01ms ? ?/sec 1.00 2.0±0.02ms ? ?/sec
physical_plan_clickbench_q14 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q15 1.01 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q16 1.02 1830.2±11.42µs ? ?/sec 1.00 1799.2±6.51µs ? ?/sec
physical_plan_clickbench_q17 1.01 1865.9±7.96µs ? ?/sec 1.00 1845.5±8.11µs ? ?/sec
physical_plan_clickbench_q18 1.01 1696.5±8.52µs ? ?/sec 1.00 1679.5±6.55µs ? ?/sec
physical_plan_clickbench_q19 1.01 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q2 1.00 1774.0±7.46µs ? ?/sec 1.02 1805.9±9.54µs ? ?/sec
physical_plan_clickbench_q20 1.01 1541.9±8.90µs ? ?/sec 1.00 1529.8±6.14µs ? ?/sec
physical_plan_clickbench_q21 1.01 1788.1±9.08µs ? ?/sec 1.00 1773.5±6.61µs ? ?/sec
physical_plan_clickbench_q22 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q23 1.01 2.4±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q24 1.02 6.7±0.01ms ? ?/sec 1.00 6.6±0.01ms ? ?/sec
physical_plan_clickbench_q25 1.02 1934.4±19.16µs ? ?/sec 1.00 1902.7±7.38µs ? ?/sec
physical_plan_clickbench_q26 1.01 1750.8±8.01µs ? ?/sec 1.00 1735.7±7.58µs ? ?/sec
physical_plan_clickbench_q27 1.00 1946.1±8.15µs ? ?/sec 1.00 1946.1±7.30µs ? ?/sec
physical_plan_clickbench_q28 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q29 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q3 1.00 1666.0±9.27µs ? ?/sec 1.02 1701.6±6.63µs ? ?/sec
physical_plan_clickbench_q30 1.02 15.4±0.04ms ? ?/sec 1.00 15.2±0.03ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q33 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q34 1.00 1826.3±8.42µs ? ?/sec 1.00 1830.4±6.25µs ? ?/sec
physical_plan_clickbench_q35 1.00 1884.5±8.75µs ? ?/sec 1.00 1876.7±6.80µs ? ?/sec
physical_plan_clickbench_q36 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q37 1.01 2.6±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q38 1.01 2.6±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q39 1.01 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q4 1.00 1486.0±7.85µs ? ?/sec 1.02 1522.2±12.10µs ? ?/sec
physical_plan_clickbench_q40 1.01 3.4±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
physical_plan_clickbench_q41 1.01 2.9±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
physical_plan_clickbench_q43 1.00 3.1±0.01ms ? ?/sec 1.01 3.2±0.01ms ? ?/sec
physical_plan_clickbench_q44 1.00 1596.8±9.20µs ? ?/sec 1.00 1589.2±6.54µs ? ?/sec
physical_plan_clickbench_q45 1.00 1578.5±8.13µs ? ?/sec 1.01 1595.9±6.41µs ? ?/sec
physical_plan_clickbench_q46 1.00 1896.6±8.74µs ? ?/sec 1.01 1911.8±7.36µs ? ?/sec
physical_plan_clickbench_q47 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q48 1.00 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_clickbench_q49 1.00 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_clickbench_q5 1.00 1637.0±8.38µs ? ?/sec 1.01 1657.1±13.26µs ? ?/sec
physical_plan_clickbench_q50 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q51 1.02 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_clickbench_q6 1.00 1642.5±8.68µs ? ?/sec 1.00 1647.7±7.80µs ? ?/sec
physical_plan_clickbench_q7 1.02 1463.4±8.35µs ? ?/sec 1.00 1438.1±6.17µs ? ?/sec
physical_plan_clickbench_q8 1.01 1955.1±9.89µs ? ?/sec 1.00 1927.7±7.68µs ? ?/sec
physical_plan_clickbench_q9 1.00 1971.0±9.51µs ? ?/sec 1.00 1964.9±10.23µs ? ?/sec
physical_plan_struct_join_agg_sort 1.01 1321.7±3.11µs ? ?/sec 1.00 1314.0±3.38µs ? ?/sec
physical_plan_tpcds_all 1.01 719.2±0.53ms ? ?/sec 1.00 713.5±3.62ms ? ?/sec
physical_plan_tpch_all 1.00 45.1±0.07ms ? ?/sec 1.00 44.9±0.23ms ? ?/sec
physical_plan_tpch_q1 1.03 1587.7±2.28µs ? ?/sec 1.00 1547.4±3.11µs ? ?/sec
physical_plan_tpch_q10 1.03 2.9±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
physical_plan_tpch_q11 1.03 2.3±0.00ms ? ?/sec 1.00 2.2±0.00ms ? ?/sec
physical_plan_tpch_q12 1.02 1285.6±2.19µs ? ?/sec 1.00 1256.1±3.05µs ? ?/sec
physical_plan_tpch_q13 1.01 1067.0±2.08µs ? ?/sec 1.00 1052.3±2.99µs ? ?/sec
physical_plan_tpch_q14 1.05 1449.7±2.78µs ? ?/sec 1.00 1382.2±3.07µs ? ?/sec
physical_plan_tpch_q16 1.02 1639.2±3.19µs ? ?/sec 1.00 1613.4±2.32µs ? ?/sec
physical_plan_tpch_q17 1.02 1675.1±2.45µs ? ?/sec 1.00 1642.4±10.00µs ? ?/sec
physical_plan_tpch_q18 1.01 2.0±0.01ms ? ?/sec 1.00 1988.8±13.95µs ? ?/sec
physical_plan_tpch_q19 1.02 1933.3±13.94µs ? ?/sec 1.00 1891.3±3.72µs ? ?/sec
physical_plan_tpch_q2 1.02 3.7±0.01ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
physical_plan_tpch_q20 1.01 2.2±0.00ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
physical_plan_tpch_q21 1.00 2.9±0.00ms ? ?/sec 1.00 2.9±0.00ms ? ?/sec
physical_plan_tpch_q22 1.01 1527.5±2.38µs ? ?/sec 1.00 1505.3±3.68µs ? ?/sec
physical_plan_tpch_q3 1.03 1909.9±6.22µs ? ?/sec 1.00 1860.6±4.58µs ? ?/sec
physical_plan_tpch_q4 1.01 1221.8±2.45µs ? ?/sec 1.00 1205.1±3.50µs ? ?/sec
physical_plan_tpch_q5 1.01 2.8±0.00ms ? ?/sec 1.00 2.7±0.00ms ? ?/sec
physical_plan_tpch_q6 1.02 634.2±1.63µs ? ?/sec 1.00 621.9±6.27µs ? ?/sec
physical_plan_tpch_q7 1.02 2.9±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
physical_plan_tpch_q8 1.01 3.9±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
physical_plan_tpch_q9 1.02 2.7±0.00ms ? ?/sec 1.00 2.7±0.00ms ? ?/sec
physical_select_aggregates_from_200 1.00 15.4±0.02ms ? ?/sec 1.00 15.4±0.02ms ? ?/sec
physical_select_all_from_1000 1.00 114.4±0.17ms ? ?/sec 1.00 114.4±0.19ms ? ?/sec
physical_select_one_from_700 1.00 765.1±2.60µs ? ?/sec 1.03 785.2±2.31µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.02 4.2±0.00ms ? ?/sec 1.00 4.1±0.00ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.04 9.1±0.01ms ? ?/sec 1.00 8.7±0.01ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.01 95.0±0.27ms ? ?/sec 1.00 94.3±0.27ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.09 394.5±0.98ms ? ?/sec 1.00 361.3±1.11ms ? ?/sec
physical_theta_join_consider_sort 1.00 1071.6±2.53µs ? ?/sec 1.02 1091.7±2.45µs ? ?/sec
physical_unnest_to_join 1.00 636.2±6.39µs ? ?/sec 1.01 641.1±1.76µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.01 717.8±1.13µs ? ?/sec 1.00 709.2±2.04µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.02 1425.2±5.09µs ? ?/sec 1.00 1403.1±2.67µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.02 442.3±1.02µs ? ?/sec 1.00 432.7±1.43µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.03 546.0±1.16µs ? ?/sec 1.00 528.8±1.47µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.03 585.2±1.56µs ? ?/sec 1.00 568.4±1.76µs ? ?/sec
with_param_values_many_columns 1.00 430.1±1.93µs ? ?/sec 1.00 431.4±2.70µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2305.5s |
| Peak memory | 131.3 MiB |
| Avg memory | 70.1 MiB |
| CPU user | 1892.2s |
| CPU sys | 1.3s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2295.5s |
| Peak memory | 131.3 MiB |
| Avg memory | 70.0 MiB |
| CPU user | 1883.4s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5358892436
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5364843024
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5364827227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364827227-1830-7w57k 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (724812422d6d2501886c269f3a1795bd8ad3b409) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..724812422d6d2501886c269f3a1795bd8ad3b409)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5324430119
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24445?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `98.07692%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.25%. Comparing base ([`6eaca8b`](https://app.codecov.io/gh/apache/datafusion/commit/6eaca8bfe8421faca1413f29f71fcaf0ccaf631a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d1e682e`](https://app.codecov.io/gh/apache/datafusion/commit/d1e682ecdd95e594dcde196fe32b98548efde19c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24445?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-plan/src/projection.rs](https://app.codecov.io/gh/apache/datafusion/pull/24445?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fprojection.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9wcm9qZWN0aW9uLnJz) | 97.56% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24445 +/- ##
==========================================
- Coverage 81.26% 81.25% -0.01%
==========================================
Files 1112 1112
Lines 391578 391626 +48
Branches 391578 391626 +48
==========================================
+ Hits 318197 318233 +36
- Misses 54677 54682 +5
- Partials 18704 18711 +7
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24445?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5338860435
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5338111275)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing qizhu/reuse-eq-group-on-child-swap (09d9cd2d037621aac741f053f87e185cea92fe67) to c429919 (merge-base) [diff](https://github.com/apache/datafusion/compare/c429919c7c8bd81e4e687531a609e2380ed60649..09d9cd2d037621aac741f053f87e185cea92fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD qizhu_reuse-eq-group-on-child-swap
----- ---- ----------------------------------
logical_aggregate_with_join 1.00 456.3±1.34µs ? ?/sec 1.00 455.8±1.31µs ? ?/sec
logical_correlated_subquery_exists 1.00 286.6±0.80µs ? ?/sec 1.00 286.1±1.12µs ? ?/sec
logical_correlated_subquery_in 1.00 288.4±0.53µs ? ?/sec 1.00 287.8±1.14µs ? ?/sec
logical_distinct_many_columns 1.00 574.9±2.73µs ? ?/sec 1.00 576.0±3.38µs ? ?/sec
logical_join_4_with_agg_and_filter 1.01 256.4±1.26µs ? ?/sec 1.00 253.8±1.52µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.00 427.1±1.45µs ? ?/sec 1.00 426.9±2.41µs ? ?/sec
logical_join_chain_16 1.00 673.5±3.77µs ? ?/sec 1.00 675.6±3.54µs ? ?/sec
logical_join_chain_4 1.00 123.5±0.41µs ? ?/sec 1.00 123.0±0.77µs ? ?/sec
logical_join_chain_8 1.00 250.1±0.80µs ? ?/sec 1.00 251.3±1.26µs ? ?/sec
logical_multiple_subqueries 1.00 526.4±1.28µs ? ?/sec 1.00 524.9±3.61µs ? ?/sec
logical_nested_cte_4_levels 1.05 274.0±1.52µs ? ?/sec 1.00 261.2±1.00µs ? ?/sec
logical_plan_struct_join_agg_sort 1.04 189.4±0.94µs ? ?/sec 1.00 182.8±0.93µs ? ?/sec
logical_plan_tpcds_all 1.00 95.2±0.17ms ? ?/sec 1.00 95.1±0.92ms ? ?/sec
logical_plan_tpch_all 1.01 6.8±0.03ms ? ?/sec 1.00 6.7±0.02ms ? ?/sec
logical_scalar_subquery 1.00 313.4±1.19µs ? ?/sec 1.00 312.1±2.06µs ? ?/sec
logical_select_all_from_1000 1.02 107.0±1.72ms ? ?/sec 1.00 104.8±0.21ms ? ?/sec
logical_select_one_from_700 1.00 326.2±1.63µs ? ?/sec 1.00 326.9±2.06µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 287.5±0.76µs ? ?/sec 1.00 285.3±0.70µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.01 274.8±1.08µs ? ?/sec 1.00 272.5±0.53µs ? ?/sec
logical_union_4_branches 1.02 428.5±2.55µs ? ?/sec 1.00 420.2±1.14µs ? ?/sec
logical_union_8_branches 1.02 823.1±7.11µs ? ?/sec 1.00 807.0±3.73µs ? ?/sec
logical_wide_aggregate_100_exprs 1.00 4.5±0.01ms ? ?/sec 1.02 4.6±0.02ms ? ?/sec
logical_wide_case_50_exprs 1.00 2.4±0.00ms ? ?/sec 1.03 2.5±0.02ms ? ?/sec
logical_wide_filter_200_predicates 1.00 1324.2±7.37µs ? ?/sec 1.01 1334.6±9.36µs ? ?/sec
logical_wide_filter_50_predicates 1.00 396.4±2.63µs ? ?/sec 1.00 395.4±2.50µs ? ?/sec
optimizer_correlated_exists 1.01 246.7±0.75µs ? ?/sec 1.00 245.3±0.52µs ? ?/sec
optimizer_join_4_with_agg_filter 1.02 466.6±1.86µs ? ?/sec 1.00 457.2±1.20µs ? ?/sec
optimizer_join_chain_4 1.01 182.8±0.60µs ? ?/sec 1.00 181.5±0.53µs ? ?/sec
optimizer_join_chain_8 1.01 567.4±0.90µs ? ?/sec 1.00 562.6±1.53µs ? ?/sec
optimizer_select_all_from_1000 1.00 6.9±0.01ms ? ?/sec 1.00 6.8±0.02ms ? ?/sec
optimizer_select_one_from_700 1.00 257.0±0.63µs ? ?/sec 1.00 258.1±0.69µs ? ?/sec
optimizer_tpcds_all 1.00 312.5±0.38ms ? ?/sec 1.00 313.0±0.30ms ? ?/sec
optimizer_tpch_all 1.00 17.5±0.04ms ? ?/sec 1.00 17.5±0.03ms ? ?/sec
optimizer_wide_aggregate_100 1.01 2.3±0.01ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
optimizer_wide_filter_200 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_intersection 1.01 606.7±2.52µs ? ?/sec 1.00 598.1±2.55µs ? ?/sec
physical_join_consider_sort 1.02 1075.5±3.54µs ? ?/sec 1.00 1051.6±2.47µs ? ?/sec
physical_join_distinct 1.01 266.8±0.61µs ? ?/sec 1.00 264.7±0.71µs ? ?/sec
physical_many_self_joins 1.02 7.9±0.02ms ? ?/sec 1.00 7.7±0.03ms ? ?/sec
physical_plan_clickbench_all 1.00 130.0±0.55ms ? ?/sec 1.22 158.5±3.09ms ? ?/sec
physical_plan_clickbench_q1 1.01 1430.5±8.52µs ? ?/sec 1.00 1422.8±19.54µs ? ?/sec
physical_plan_clickbench_q10 1.00 2.1±0.01ms ? ?/sec 1.27 2.7±0.15ms ? ?/sec
physical_plan_clickbench_q11 1.00 2.3±0.01ms ? ?/sec 1.23 2.8±0.09ms ? ?/sec
physical_plan_clickbench_q12 1.00 2.3±0.01ms ? ?/sec 1.22 2.9±0.10ms ? ?/sec
physical_plan_clickbench_q13 1.00 2.1±0.02ms ? ?/sec 1.23 2.6±0.10ms ? ?/sec
physical_plan_clickbench_q14 1.00 2.2±0.01ms ? ?/sec 1.25 2.8±0.10ms ? ?/sec
physical_plan_clickbench_q15 1.00 2.2±0.01ms ? ?/sec 1.25 2.7±0.08ms ? ?/sec
physical_plan_clickbench_q16 1.00 1838.1±8.37µs ? ?/sec 1.27 2.3±0.09ms ? ?/sec
physical_plan_clickbench_q17 1.00 1888.8±7.27µs ? ?/sec 1.27 2.4±0.08ms ? ?/sec
physical_plan_clickbench_q18 1.00 1714.4±7.25µs ? ?/sec 1.25 2.1±0.08ms ? ?/sec
physical_plan_clickbench_q19 1.00 2.1±0.01ms ? ?/sec 1.27 2.7±0.08ms ? ?/sec
physical_plan_clickbench_q2 1.01 1816.4±7.42µs ? ?/sec 1.00 1803.6±17.70µs ? ?/sec
physical_plan_clickbench_q20 1.00 1566.5±7.58µs ? ?/sec 1.25 1953.5±86.24µs ? ?/sec
physical_plan_clickbench_q21 1.00 1811.6±8.08µs ? ?/sec 1.29 2.3±0.09ms ? ?/sec
physical_plan_clickbench_q22 1.00 2.2±0.01ms ? ?/sec 1.28 2.8±0.08ms ? ?/sec
physical_plan_clickbench_q23 1.00 2.4±0.01ms ? ?/sec 1.26 3.0±0.06ms ? ?/sec
physical_plan_clickbench_q24 1.00 6.8±0.02ms ? ?/sec 1.09 7.4±0.07ms ? ?/sec
physical_plan_clickbench_q25 1.00 1941.1±6.52µs ? ?/sec 1.27 2.5±0.07ms ? ?/sec
physical_plan_clickbench_q26 1.00 1775.9±18.27µs ? ?/sec 1.27 2.3±0.09ms ? ?/sec
physical_plan_clickbench_q27 1.00 1968.7±9.38µs ? ?/sec 1.26 2.5±0.08ms ? ?/sec
physical_plan_clickbench_q28 1.00 2.4±0.01ms ? ?/sec 1.28 3.0±0.08ms ? ?/sec
physical_plan_clickbench_q29 1.00 2.5±0.01ms ? ?/sec 1.27 3.2±0.08ms ? ?/sec
physical_plan_clickbench_q3 1.00 1695.1±7.79µs ? ?/sec 1.00 1696.4±22.73µs ? ?/sec
physical_plan_clickbench_q30 1.00 15.8±0.07ms ? ?/sec 1.02 16.2±0.10ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.5±0.01ms ? ?/sec 1.24 3.1±0.08ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.5±0.01ms ? ?/sec 1.26 3.2±0.07ms ? ?/sec
physical_plan_clickbench_q33 1.00 2.1±0.01ms ? ?/sec 1.26 2.7±0.09ms ? ?/sec
physical_plan_clickbench_q34 1.00 1861.0±7.57µs ? ?/sec 1.28 2.4±0.07ms ? ?/sec
physical_plan_clickbench_q35 1.00 1869.9±6.75µs ? ?/sec 1.31 2.4±0.11ms ? ?/sec
physical_plan_clickbench_q36 1.00 2.2±0.01ms ? ?/sec 1.30 2.8±0.07ms ? ?/sec
physical_plan_clickbench_q37 1.00 2.6±0.01ms ? ?/sec 1.28 3.3±0.08ms ? ?/sec
physical_plan_clickbench_q38 1.00 2.6±0.01ms ? ?/sec 1.28 3.3±0.08ms ? ?/sec
physical_plan_clickbench_q39 1.00 2.7±0.01ms ? ?/sec 1.28 3.4±0.13ms ? ?/sec
physical_plan_clickbench_q4 1.00 1514.4±10.96µs ? ?/sec 1.00 1508.6±35.52µs ? ?/sec
physical_plan_clickbench_q40 1.00 3.4±0.01ms ? ?/sec 1.23 4.1±0.07ms ? ?/sec
physical_plan_clickbench_q41 1.00 2.9±0.01ms ? ?/sec 1.23 3.6±0.07ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.1±0.01ms ? ?/sec 1.23 3.8±0.07ms ? ?/sec
physical_plan_clickbench_q43 1.00 3.2±0.01ms ? ?/sec 1.23 4.0±0.07ms ? ?/sec
physical_plan_clickbench_q44 1.00 1624.6±6.35µs ? ?/sec 1.24 2.0±0.08ms ? ?/sec
physical_plan_clickbench_q45 1.00 1629.3±6.65µs ? ?/sec 1.24 2.0±0.07ms ? ?/sec
physical_plan_clickbench_q46 1.00 1953.3±8.23µs ? ?/sec 1.26 2.5±0.08ms ? ?/sec
physical_plan_clickbench_q47 1.00 2.6±0.01ms ? ?/sec 1.25 3.3±0.08ms ? ?/sec
physical_plan_clickbench_q48 1.00 2.8±0.01ms ? ?/sec 1.23 3.5±0.06ms ? ?/sec
physical_plan_clickbench_q49 1.00 2.9±0.01ms ? ?/sec 1.23 3.5±0.06ms ? ?/sec
physical_plan_clickbench_q5 1.00 1654.3±6.41µs ? ?/sec 1.16 1915.1±124.58µs ? ?/sec
physical_plan_clickbench_q50 1.00 2.7±0.01ms ? ?/sec 1.25 3.4±0.06ms ? ?/sec
physical_plan_clickbench_q51 1.00 2.1±0.01ms ? ?/sec 1.25 2.6±0.11ms ? ?/sec
physical_plan_clickbench_q6 1.00 1652.1±7.85µs ? ?/sec 1.08 1786.6±55.49µs ? ?/sec
physical_plan_clickbench_q7 1.00 1471.6±8.24µs ? ?/sec 1.07 1575.1±41.72µs ? ?/sec
physical_plan_clickbench_q8 1.00 1966.7±21.79µs ? ?/sec 1.11 2.2±0.08ms ? ?/sec
physical_plan_clickbench_q9 1.00 1972.3±9.23µs ? ?/sec 1.08 2.1±0.07ms ? ?/sec
physical_plan_struct_join_agg_sort 1.02 1360.3±39.05µs ? ?/sec 1.00 1329.9±9.33µs ? ?/sec
physical_plan_tpcds_all 1.06 759.7±8.58ms ? ?/sec 1.00 714.9±6.47ms ? ?/sec
physical_plan_tpch_all 1.07 47.8±0.29ms ? ?/sec 1.00 44.6±0.11ms ? ?/sec
physical_plan_tpch_q1 1.01 1609.1±8.21µs ? ?/sec 1.00 1585.9±3.94µs ? ?/sec
physical_plan_tpch_q10 1.06 3.0±0.04ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.03 2.4±0.04ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_tpch_q12 1.01 1290.0±10.12µs ? ?/sec 1.00 1274.6±2.23µs ? ?/sec
physical_plan_tpch_q13 1.01 1072.2±8.00µs ? ?/sec 1.00 1060.6±2.75µs ? ?/sec
physical_plan_tpch_q14 1.04 1493.7±46.23µs ? ?/sec 1.00 1433.6±3.77µs ? ?/sec
physical_plan_tpch_q16 1.01 1653.7±13.31µs ? ?/sec 1.00 1635.6±3.75µs ? ?/sec
physical_plan_tpch_q17 1.03 1702.0±10.75µs ? ?/sec 1.00 1658.9±5.37µs ? ?/sec
physical_plan_tpch_q18 1.02 2.0±0.02ms ? ?/sec 1.00 1978.8±3.71µs ? ?/sec
physical_plan_tpch_q19 1.01 1937.7±11.24µs ? ?/sec 1.00 1911.4±2.97µs ? ?/sec
physical_plan_tpch_q2 1.08 3.9±0.10ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
physical_plan_tpch_q20 1.05 2.3±0.04ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_tpch_q21 1.03 2.9±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_tpch_q22 1.02 1545.3±11.21µs ? ?/sec 1.00 1510.4±5.30µs ? ?/sec
physical_plan_tpch_q3 1.04 1942.0±18.60µs ? ?/sec 1.00 1873.1±4.44µs ? ?/sec
physical_plan_tpch_q4 1.02 1240.3±6.13µs ? ?/sec 1.00 1218.0±2.76µs ? ?/sec
physical_plan_tpch_q5 1.05 2.9±0.03ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
physical_plan_tpch_q6 1.01 644.6±2.80µs ? ?/sec 1.00 636.2±2.39µs ? ?/sec
physical_plan_tpch_q7 1.06 3.0±0.06ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
physical_plan_tpch_q8 1.09 4.2±0.03ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
physical_plan_tpch_q9 1.08 2.9±0.03ms ? ?/sec 1.00 2.7±0.00ms ? ?/sec
physical_select_aggregates_from_200 1.01 15.8±0.03ms ? ?/sec 1.00 15.6±0.06ms ? ?/sec
physical_select_all_from_1000 1.00 116.3±0.44ms ? ?/sec 1.02 118.4±1.86ms ? ?/sec
physical_select_one_from_700 1.00 771.9±2.47µs ? ?/sec 1.00 770.1±3.15µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.05 4.4±0.03ms ? ?/sec 1.00 4.2±0.01ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.08 9.4±0.03ms ? ?/sec 1.00 8.7±0.02ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.04 99.1±0.43ms ? ?/sec 1.00 95.4±0.30ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.13 416.0±1.48ms ? ?/sec 1.00 369.8±1.17ms ? ?/sec
physical_theta_join_consider_sort 1.00 1081.7±4.51µs ? ?/sec 1.00 1080.0±3.34µs ? ?/sec
physical_unnest_to_join 1.00 637.6±1.88µs ? ?/sec 1.00 638.0±2.16µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.01 729.9±1.64µs ? ?/sec 1.00 721.8±1.74µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.02 1436.8±3.68µs ? ?/sec 1.00 1413.1±4.37µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.01 450.8±1.56µs ? ?/sec 1.00 444.5±1.36µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.01 551.9±1.51µs ? ?/sec 1.00 545.5±1.79µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 592.1±1.88µs ? ?/sec 1.00 594.9±2.17µs ? ?/sec
with_param_values_many_columns 1.00 436.7±2.34µs ? ?/sec 1.00 437.8±3.03µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2580.6s |
| Peak memory | 132.0 MiB |
| Avg memory | 63.4 MiB |
| CPU user | 1903.4s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2470.5s |
| Peak memory | 131.1 MiB |
| Avg memory | 63.2 MiB |
| CPU user | 1843.6s |
| CPU sys | 1.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5358891983
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5352242796
Benchmark for [this request](https://github.com/apache/datafusion/pull/24445#issuecomment-5351980794) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `sql_planner`
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5358934171
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5358892436)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5358892436-1814-rltnh 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (321dae0effa0b4bf70ea1bc8302c5b9159920bae) to dbdc627 (merge-base) [diff](https://github.com/apache/datafusion/compare/dbdc627deb98778dbf23cf7ccc2df7c66e1d9f55..321dae0effa0b4bf70ea1bc8302c5b9159920bae)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5359773665
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5358891983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing qizhu/reuse-eq-group-on-child-swap (321dae0effa0b4bf70ea1bc8302c5b9159920bae) to dbdc627 (merge-base) [diff](https://github.com/apache/datafusion/compare/dbdc627deb98778dbf23cf7ccc2df7c66e1d9f55..321dae0effa0b4bf70ea1bc8302c5b9159920bae)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD qizhu_reuse-eq-group-on-child-swap
----- ---- ----------------------------------
logical_aggregate_with_join 1.00 443.2±1.53µs ? ?/sec 1.03 455.2±3.08µs ? ?/sec
logical_correlated_subquery_exists 1.00 278.6±0.81µs ? ?/sec 1.02 284.8±0.84µs ? ?/sec
logical_correlated_subquery_in 1.00 280.8±1.07µs ? ?/sec 1.02 285.9±0.70µs ? ?/sec
logical_distinct_many_columns 1.00 564.5±1.40µs ? ?/sec 1.02 573.3±1.36µs ? ?/sec
logical_join_4_with_agg_and_filter 1.00 245.2±1.08µs ? ?/sec 1.02 250.0±1.03µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.00 413.0±1.42µs ? ?/sec 1.03 423.7±1.70µs ? ?/sec
logical_join_chain_16 1.00 665.5±3.14µs ? ?/sec 1.02 681.2±6.37µs ? ?/sec
logical_join_chain_4 1.00 119.6±0.41µs ? ?/sec 1.02 121.8±0.49µs ? ?/sec
logical_join_chain_8 1.00 246.4±1.29µs ? ?/sec 1.01 249.9±1.41µs ? ?/sec
logical_multiple_subqueries 1.00 504.0±2.45µs ? ?/sec 1.03 517.8±5.44µs ? ?/sec
logical_nested_cte_4_levels 1.00 252.7±1.09µs ? ?/sec 1.01 256.4±1.55µs ? ?/sec
logical_plan_struct_join_agg_sort 1.00 169.8±1.25µs ? ?/sec 1.03 174.8±1.54µs ? ?/sec
logical_plan_tpcds_all 1.00 92.5±0.16ms ? ?/sec 1.01 93.0±0.19ms ? ?/sec
logical_plan_tpch_all 1.00 6.4±0.02ms ? ?/sec 1.01 6.5±0.03ms ? ?/sec
logical_scalar_subquery 1.00 301.6±1.42µs ? ?/sec 1.02 306.4±1.76µs ? ?/sec
logical_select_all_from_1000 1.00 104.2±0.14ms ? ?/sec 1.01 105.3±0.17ms ? ?/sec
logical_select_one_from_700 1.00 324.4±2.22µs ? ?/sec 1.01 327.5±1.80µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 281.9±0.98µs ? ?/sec 1.01 286.0±0.65µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 269.4±0.97µs ? ?/sec 1.02 273.8±0.87µs ? ?/sec
logical_union_4_branches 1.00 416.0±1.56µs ? ?/sec 1.01 421.6±3.29µs ? ?/sec
logical_union_8_branches 1.00 797.1±2.49µs ? ?/sec 1.02 810.7±2.38µs ? ?/sec
logical_wide_aggregate_100_exprs 1.00 4.5±0.01ms ? ?/sec 1.00 4.5±0.01ms ? ?/sec
logical_wide_case_50_exprs 1.00 2.4±0.00ms ? ?/sec 1.02 2.4±0.00ms ? ?/sec
logical_wide_filter_200_predicates 1.00 1314.2±7.70µs ? ?/sec 1.03 1351.9±10.70µs ? ?/sec
logical_wide_filter_50_predicates 1.00 387.8±2.39µs ? ?/sec 1.01 393.2±2.85µs ? ?/sec
optimizer_correlated_exists 1.00 242.1±0.99µs ? ?/sec 1.04 250.9±0.97µs ? ?/sec
optimizer_join_4_with_agg_filter 1.00 465.9±1.73µs ? ?/sec 1.02 474.8±1.44µs ? ?/sec
optimizer_join_chain_4 1.00 181.4±0.53µs ? ?/sec 1.02 184.7±0.29µs ? ?/sec
optimizer_join_chain_8 1.00 565.8±1.18µs ? ?/sec 1.01 572.1±1.19µs ? ?/sec
optimizer_select_all_from_1000 1.04 7.1±0.01ms ? ?/sec 1.00 6.8±0.01ms ? ?/sec
optimizer_select_one_from_700 1.00 253.9±0.95µs ? ?/sec 1.03 260.7±0.71µs ? ?/sec
optimizer_tpcds_all 1.00 312.5±1.80ms ? ?/sec 1.00 313.9±0.35ms ? ?/sec
optimizer_tpch_all 1.00 17.5±0.05ms ? ?/sec 1.01 17.7±0.04ms ? ?/sec
optimizer_wide_aggregate_100 1.00 2.3±0.00ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
optimizer_wide_filter_200 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_intersection 1.00 587.1±2.27µs ? ?/sec 1.02 601.3±1.67µs ? ?/sec
physical_join_consider_sort 1.00 1037.0±2.33µs ? ?/sec 1.04 1081.1±6.21µs ? ?/sec
physical_join_distinct 1.00 261.9±1.24µs ? ?/sec 1.02 267.3±0.78µs ? ?/sec
physical_many_self_joins 1.00 7.6±0.03ms ? ?/sec 1.02 7.8±0.02ms ? ?/sec
physical_plan_clickbench_all 1.00 130.1±0.45ms ? ?/sec 1.00 129.7±0.47ms ? ?/sec
physical_plan_clickbench_q1 1.00 1411.3±8.46µs ? ?/sec 1.00 1418.1±11.82µs ? ?/sec
physical_plan_clickbench_q10 1.00 2.1±0.01ms ? ?/sec 1.02 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q11 1.00 2.2±0.01ms ? ?/sec 1.02 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q12 1.00 2.3±0.01ms ? ?/sec 1.03 2.4±0.02ms ? ?/sec
physical_plan_clickbench_q13 1.00 2.1±0.01ms ? ?/sec 1.03 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q14 1.00 2.2±0.01ms ? ?/sec 1.03 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q15 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q16 1.00 1849.5±6.34µs ? ?/sec 1.00 1847.8±6.52µs ? ?/sec
physical_plan_clickbench_q17 1.00 1901.1±8.02µs ? ?/sec 1.00 1897.7±7.84µs ? ?/sec
physical_plan_clickbench_q18 1.00 1716.1±6.76µs ? ?/sec 1.00 1721.6±6.66µs ? ?/sec
physical_plan_clickbench_q19 1.00 2.1±0.01ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q2 1.00 1783.6±8.07µs ? ?/sec 1.01 1801.2±7.15µs ? ?/sec
physical_plan_clickbench_q20 1.00 1560.8±6.71µs ? ?/sec 1.01 1568.9±7.40µs ? ?/sec
physical_plan_clickbench_q21 1.00 1810.2±6.50µs ? ?/sec 1.00 1816.5±7.12µs ? ?/sec
physical_plan_clickbench_q22 1.00 2.2±0.01ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q23 1.00 2.4±0.02ms ? ?/sec 1.01 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q24 1.01 6.8±0.04ms ? ?/sec 1.00 6.7±0.02ms ? ?/sec
physical_plan_clickbench_q25 1.02 1963.0±17.79µs ? ?/sec 1.00 1916.2±9.31µs ? ?/sec
physical_plan_clickbench_q26 1.00 1752.4±6.21µs ? ?/sec 1.00 1751.7±6.75µs ? ?/sec
physical_plan_clickbench_q27 1.00 1935.6±6.73µs ? ?/sec 1.00 1943.4±7.02µs ? ?/sec
physical_plan_clickbench_q28 1.00 2.3±0.01ms ? ?/sec 1.01 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q29 1.00 2.5±0.02ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q3 1.00 1674.5±9.41µs ? ?/sec 1.02 1703.7±7.77µs ? ?/sec
physical_plan_clickbench_q30 1.02 15.6±0.09ms ? ?/sec 1.00 15.3±0.08ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.5±0.01ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.5±0.01ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q33 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q34 1.00 1813.5±8.09µs ? ?/sec 1.02 1844.6±7.32µs ? ?/sec
physical_plan_clickbench_q35 1.00 1856.9±8.24µs ? ?/sec 1.02 1892.0±9.07µs ? ?/sec
physical_plan_clickbench_q36 1.00 2.2±0.02ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q37 1.00 2.5±0.01ms ? ?/sec 1.02 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q38 1.00 2.5±0.02ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q39 1.00 2.6±0.01ms ? ?/sec 1.03 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q4 1.00 1490.1±7.65µs ? ?/sec 1.01 1510.8±8.32µs ? ?/sec
physical_plan_clickbench_q40 1.00 3.3±0.01ms ? ?/sec 1.04 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q41 1.00 2.9±0.01ms ? ?/sec 1.04 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.0±0.01ms ? ?/sec 1.01 3.0±0.01ms ? ?/sec
physical_plan_clickbench_q43 1.00 3.2±0.02ms ? ?/sec 1.03 3.3±0.01ms ? ?/sec
physical_plan_clickbench_q44 1.00 1593.5±6.51µs ? ?/sec 1.03 1635.6±7.40µs ? ?/sec
physical_plan_clickbench_q45 1.00 1598.5±6.41µs ? ?/sec 1.01 1608.2±7.55µs ? ?/sec
physical_plan_clickbench_q46 1.01 1946.8±7.82µs ? ?/sec 1.00 1935.4±9.57µs ? ?/sec
physical_plan_clickbench_q47 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q48 1.00 2.8±0.01ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q49 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q5 1.00 1628.7±8.27µs ? ?/sec 1.03 1677.1±7.78µs ? ?/sec
physical_plan_clickbench_q50 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q51 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q6 1.00 1635.0±7.97µs ? ?/sec 1.03 1684.2±7.29µs ? ?/sec
physical_plan_clickbench_q7 1.00 1449.4±6.13µs ? ?/sec 1.03 1498.5±7.27µs ? ?/sec
physical_plan_clickbench_q8 1.00 1942.6±10.53µs ? ?/sec 1.03 2.0±0.01ms ? ?/sec
physical_plan_clickbench_q9 1.00 1950.7±9.67µs ? ?/sec 1.02 1998.3±6.99µs ? ?/sec
physical_plan_struct_join_agg_sort 1.00 1318.7±3.48µs ? ?/sec 1.00 1324.6±4.87µs ? ?/sec
physical_plan_tpcds_all 1.00 721.2±3.50ms ? ?/sec 1.01 725.3±1.02ms ? ?/sec
physical_plan_tpch_all 1.00 44.7±0.09ms ? ?/sec 1.01 45.3±0.11ms ? ?/sec
physical_plan_tpch_q1 1.00 1567.4±2.39µs ? ?/sec 1.01 1580.5±4.62µs ? ?/sec
physical_plan_tpch_q10 1.00 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_tpch_q11 1.00 2.3±0.01ms ? ?/sec 1.01 2.3±0.00ms ? ?/sec
physical_plan_tpch_q12 1.00 1257.5±2.33µs ? ?/sec 1.02 1286.4±4.80µs ? ?/sec
physical_plan_tpch_q13 1.00 1048.9±2.71µs ? ?/sec 1.01 1059.0±3.34µs ? ?/sec
physical_plan_tpch_q14 1.00 1434.2±15.14µs ? ?/sec 1.02 1465.3±11.44µs ? ?/sec
physical_plan_tpch_q16 1.00 1628.3±2.78µs ? ?/sec 1.03 1673.3±12.95µs ? ?/sec
physical_plan_tpch_q17 1.00 1660.5±3.16µs ? ?/sec 1.03 1703.8±3.30µs ? ?/sec
physical_plan_tpch_q18 1.00 1979.1±4.40µs ? ?/sec 1.03 2.0±0.00ms ? ?/sec
physical_plan_tpch_q19 1.00 1892.9±3.17µs ? ?/sec 1.03 1946.2±4.65µs ? ?/sec
physical_plan_tpch_q2 1.00 3.6±0.00ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_plan_tpch_q20 1.00 2.1±0.01ms ? ?/sec 1.03 2.2±0.00ms ? ?/sec
physical_plan_tpch_q21 1.00 2.9±0.02ms ? ?/sec 1.02 2.9±0.01ms ? ?/sec
physical_plan_tpch_q22 1.00 1503.7±2.95µs ? ?/sec 1.01 1525.4±3.61µs ? ?/sec
physical_plan_tpch_q3 1.00 1870.9±5.19µs ? ?/sec 1.01 1897.4±2.95µs ? ?/sec
physical_plan_tpch_q4 1.00 1210.2±2.98µs ? ?/sec 1.02 1234.7±2.36µs ? ?/sec
physical_plan_tpch_q5 1.00 2.7±0.00ms ? ?/sec 1.01 2.8±0.01ms ? ?/sec
physical_plan_tpch_q6 1.00 619.5±1.57µs ? ?/sec 1.03 639.2±2.56µs ? ?/sec
physical_plan_tpch_q7 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_tpch_q8 1.00 3.9±0.02ms ? ?/sec 1.01 3.9±0.01ms ? ?/sec
physical_plan_tpch_q9 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.00 15.5±0.04ms ? ?/sec 1.01 15.6±0.07ms ? ?/sec
physical_select_all_from_1000 1.00 115.5±0.16ms ? ?/sec 1.01 116.9±0.23ms ? ?/sec
physical_select_one_from_700 1.00 760.8±2.43µs ? ?/sec 1.03 784.2±1.91µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.01 4.2±0.02ms ? ?/sec 1.00 4.2±0.01ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.04 9.1±0.04ms ? ?/sec 1.00 8.7±0.05ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.01 96.1±0.38ms ? ?/sec 1.00 94.7±0.28ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.09 400.8±1.97ms ? ?/sec 1.00 367.0±2.92ms ? ?/sec
physical_theta_join_consider_sort 1.00 1059.5±2.23µs ? ?/sec 1.03 1096.4±7.48µs ? ?/sec
physical_unnest_to_join 1.00 625.2±1.91µs ? ?/sec 1.02 636.9±2.14µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 715.2±1.76µs ? ?/sec 1.01 724.9±1.44µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.01 1432.3±2.70µs ? ?/sec 1.00 1419.3±1.80µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 434.9±1.25µs ? ?/sec 1.02 443.0±4.54µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 537.0±2.24µs ? ?/sec 1.01 540.7±1.37µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 584.7±1.83µs ? ?/sec 1.01 589.2±1.44µs ? ?/sec
with_param_values_many_columns 1.00 433.8±3.12µs ? ?/sec 1.01 439.5±1.89µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2265.5s |
| Peak memory | 129.1 MiB |
| Avg memory | 69.7 MiB |
| CPU user | 1893.5s |
| CPU sys | 1.3s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2310.5s |
| Peak memory | 131.9 MiB |
| Avg memory | 69.5 MiB |
| CPU user | 1880.9s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5339090159
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5351995130
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5351980794)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5351980794-1791-vrpsw 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (eea206ca0417a7ef33d599fdac372e68a7add6de) to f1f0449 (merge-base) [diff](https://github.com/apache/datafusion/compare/f1f0449a5327a03cc99817fcd0ac8531f0c015ec..eea206ca0417a7ef33d599fdac372e68a7add6de)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5364827227
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #24445:
URL: https://github.com/apache/datafusion/pull/24445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5351275370
On the perf question: `physical_sorted_union_order_by_50_uint64` is the shape this touches, and the bot has it at ~10% across three runs on different commits:
```
82edbc3b 404.4ms -> 368.9ms 1.10
09d9cd2d 416.0ms -> 369.8ms 1.13
95cc5360 410.3ms -> 373.9ms 1.10
```
The smaller variants move less, which is what the mechanism predicts: more columns in the sort order means a larger equivalence group to re-project on each rebuild.
The query is not public and I could not reduce it to an MRE, but I should also be straight about why its number was so much larger. That codebase predates `EnsureRequirements` and still runs `EnforceSorting` six times plus `EnforceDistribution` twice, so the projection chain is rebuilt eight times per plan rather than once. The saving is per rebuild, so consolidating those passes -- which upstream already did -- caps most of it. Quoting 45% here without that context was misleading and I will rewrite the description around the numbers above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5358934282
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5358891983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5358891983-1813-ft82t 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (321dae0effa0b4bf70ea1bc8302c5b9159920bae) to dbdc627 (merge-base) [diff](https://github.com/apache/datafusion/compare/dbdc627deb98778dbf23cf7ccc2df7c66e1d9f55..321dae0effa0b4bf70ea1bc8302c5b9159920bae)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5359774505
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5358892436)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing qizhu/reuse-eq-group-on-child-swap (321dae0effa0b4bf70ea1bc8302c5b9159920bae) to dbdc627 (merge-base) [diff](https://github.com/apache/datafusion/compare/dbdc627deb98778dbf23cf7ccc2df7c66e1d9f55..321dae0effa0b4bf70ea1bc8302c5b9159920bae)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD qizhu_reuse-eq-group-on-child-swap
----- ---- ----------------------------------
logical_aggregate_with_join 1.00 449.2±1.71µs ? ?/sec 1.04 465.8±12.34µs ? ?/sec
logical_correlated_subquery_exists 1.00 282.5±0.91µs ? ?/sec 1.02 286.8±7.31µs ? ?/sec
logical_correlated_subquery_in 1.00 284.8±0.96µs ? ?/sec 1.02 291.9±6.21µs ? ?/sec
logical_distinct_many_columns 1.00 570.2±1.34µs ? ?/sec 1.00 569.9±1.04µs ? ?/sec
logical_join_4_with_agg_and_filter 1.00 257.9±1.07µs ? ?/sec 1.01 261.0±1.49µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.00 430.1±2.70µs ? ?/sec 1.01 433.9±4.25µs ? ?/sec
logical_join_chain_16 1.00 675.1±2.97µs ? ?/sec 1.01 680.6±5.85µs ? ?/sec
logical_join_chain_4 1.00 122.9±0.52µs ? ?/sec 1.02 125.3±1.30µs ? ?/sec
logical_join_chain_8 1.00 250.9±1.04µs ? ?/sec 1.02 254.7±2.85µs ? ?/sec
logical_multiple_subqueries 1.00 518.6±1.92µs ? ?/sec 1.02 527.7±6.44µs ? ?/sec
logical_nested_cte_4_levels 1.00 264.8±6.84µs ? ?/sec 1.00 264.5±1.68µs ? ?/sec
logical_plan_struct_join_agg_sort 1.00 178.9±0.74µs ? ?/sec 1.01 181.2±0.92µs ? ?/sec
logical_plan_tpcds_all 1.00 93.4±0.20ms ? ?/sec 1.01 94.1±0.23ms ? ?/sec
logical_plan_tpch_all 1.00 6.5±0.02ms ? ?/sec 1.01 6.6±0.02ms ? ?/sec
logical_scalar_subquery 1.00 305.3±1.25µs ? ?/sec 1.03 315.1±6.73µs ? ?/sec
logical_select_all_from_1000 1.00 104.4±0.37ms ? ?/sec 1.01 105.4±0.31ms ? ?/sec
logical_select_one_from_700 1.00 325.1±1.48µs ? ?/sec 1.03 333.9±16.11µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 283.8±0.91µs ? ?/sec 1.03 292.9±12.27µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 272.1±0.83µs ? ?/sec 1.03 281.6±12.12µs ? ?/sec
logical_union_4_branches 1.00 424.7±6.36µs ? ?/sec 1.00 424.7±1.47µs ? ?/sec
logical_union_8_branches 1.00 809.4±6.64µs ? ?/sec 1.01 814.6±1.85µs ? ?/sec
logical_wide_aggregate_100_exprs 1.00 4.5±0.01ms ? ?/sec 1.00 4.5±0.01ms ? ?/sec
logical_wide_case_50_exprs 1.00 2.4±0.01ms ? ?/sec 1.01 2.4±0.01ms ? ?/sec
logical_wide_filter_200_predicates 1.00 1337.3±7.78µs ? ?/sec 1.00 1342.9±11.90µs ? ?/sec
logical_wide_filter_50_predicates 1.00 395.7±2.34µs ? ?/sec 1.00 394.1±7.27µs ? ?/sec
optimizer_correlated_exists 1.00 243.7±1.12µs ? ?/sec 1.02 249.2±0.71µs ? ?/sec
optimizer_join_4_with_agg_filter 1.00 472.1±1.01µs ? ?/sec 1.03 487.4±1.75µs ? ?/sec
optimizer_join_chain_4 1.00 184.9±0.47µs ? ?/sec 1.02 187.8±0.39µs ? ?/sec
optimizer_join_chain_8 1.00 568.9±1.08µs ? ?/sec 1.01 572.7±1.89µs ? ?/sec
optimizer_select_all_from_1000 1.04 7.1±0.03ms ? ?/sec 1.00 6.8±0.01ms ? ?/sec
optimizer_select_one_from_700 1.00 255.0±1.04µs ? ?/sec 1.02 261.0±0.74µs ? ?/sec
optimizer_tpcds_all 1.00 313.1±1.01ms ? ?/sec 1.01 315.5±0.45ms ? ?/sec
optimizer_tpch_all 1.00 17.6±0.10ms ? ?/sec 1.01 17.8±0.03ms ? ?/sec
optimizer_wide_aggregate_100 1.00 2.3±0.00ms ? ?/sec 1.01 2.3±0.01ms ? ?/sec
optimizer_wide_filter_200 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_intersection 1.00 601.0±1.01µs ? ?/sec 1.01 608.0±1.52µs ? ?/sec
physical_join_consider_sort 1.00 1048.7±2.18µs ? ?/sec 1.01 1061.6±3.71µs ? ?/sec
physical_join_distinct 1.00 265.6±0.70µs ? ?/sec 1.03 273.5±11.94µs ? ?/sec
physical_many_self_joins 1.00 7.6±0.05ms ? ?/sec 1.02 7.8±0.03ms ? ?/sec
physical_plan_clickbench_all 1.00 129.9±0.41ms ? ?/sec 1.01 131.0±0.49ms ? ?/sec
physical_plan_clickbench_q1 1.01 1417.6±10.09µs ? ?/sec 1.00 1405.5±11.32µs ? ?/sec
physical_plan_clickbench_q10 1.00 2.1±0.02ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q11 1.00 2.2±0.01ms ? ?/sec 1.01 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q12 1.00 2.3±0.01ms ? ?/sec 1.01 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q13 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q14 1.00 2.2±0.01ms ? ?/sec 1.02 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q15 1.01 2.2±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q16 1.00 1825.3±14.09µs ? ?/sec 1.01 1835.9±12.10µs ? ?/sec
physical_plan_clickbench_q17 1.00 1872.7±9.64µs ? ?/sec 1.01 1886.4±7.67µs ? ?/sec
physical_plan_clickbench_q18 1.00 1700.3±7.35µs ? ?/sec 1.01 1714.3±7.43µs ? ?/sec
physical_plan_clickbench_q19 1.00 2.1±0.01ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q2 1.00 1815.4±9.93µs ? ?/sec 1.00 1810.9±11.14µs ? ?/sec
physical_plan_clickbench_q20 1.02 1579.8±12.89µs ? ?/sec 1.00 1553.0±9.96µs ? ?/sec
physical_plan_clickbench_q21 1.00 1800.7±15.17µs ? ?/sec 1.00 1798.7±8.19µs ? ?/sec
physical_plan_clickbench_q22 1.00 2.2±0.02ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q23 1.00 2.4±0.04ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q24 1.00 6.7±0.04ms ? ?/sec 1.00 6.8±0.13ms ? ?/sec
physical_plan_clickbench_q25 1.00 1925.2±13.14µs ? ?/sec 1.00 1926.7±11.93µs ? ?/sec
physical_plan_clickbench_q26 1.00 1767.9±10.01µs ? ?/sec 1.00 1760.8±9.51µs ? ?/sec
physical_plan_clickbench_q27 1.00 1953.8±20.39µs ? ?/sec 1.02 1989.2±8.30µs ? ?/sec
physical_plan_clickbench_q28 1.00 2.4±0.01ms ? ?/sec 1.02 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q29 1.00 2.5±0.01ms ? ?/sec 1.02 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q3 1.02 1737.1±13.35µs ? ?/sec 1.00 1707.2±10.94µs ? ?/sec
physical_plan_clickbench_q30 1.02 15.6±0.08ms ? ?/sec 1.00 15.3±0.14ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.5±0.02ms ? ?/sec 1.03 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.5±0.02ms ? ?/sec 1.01 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q33 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q34 1.00 1831.5±9.76µs ? ?/sec 1.01 1843.2±6.20µs ? ?/sec
physical_plan_clickbench_q35 1.00 1873.0±7.05µs ? ?/sec 1.05 1961.4±14.85µs ? ?/sec
physical_plan_clickbench_q36 1.00 2.2±0.01ms ? ?/sec 1.05 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q37 1.00 2.6±0.02ms ? ?/sec 1.02 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q38 1.00 2.6±0.03ms ? ?/sec 1.02 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q39 1.00 2.6±0.01ms ? ?/sec 1.02 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q4 1.00 1531.7±10.03µs ? ?/sec 1.00 1525.2±7.05µs ? ?/sec
physical_plan_clickbench_q40 1.00 3.4±0.01ms ? ?/sec 1.02 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q41 1.00 2.9±0.02ms ? ?/sec 1.02 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.1±0.03ms ? ?/sec 1.02 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q43 1.00 3.2±0.03ms ? ?/sec 1.02 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q44 1.00 1607.5±9.58µs ? ?/sec 1.03 1660.2±11.29µs ? ?/sec
physical_plan_clickbench_q45 1.00 1612.9±11.14µs ? ?/sec 1.00 1613.6±6.11µs ? ?/sec
physical_plan_clickbench_q46 1.00 1930.6±7.70µs ? ?/sec 1.00 1935.1±8.28µs ? ?/sec
physical_plan_clickbench_q47 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q48 1.00 2.9±0.02ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q49 1.01 2.9±0.02ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q5 1.00 1648.9±7.24µs ? ?/sec 1.01 1662.5±7.92µs ? ?/sec
physical_plan_clickbench_q50 1.00 2.7±0.03ms ? ?/sec 1.01 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q51 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q6 1.00 1655.5±6.39µs ? ?/sec 1.01 1669.0±7.37µs ? ?/sec
physical_plan_clickbench_q7 1.00 1498.4±8.47µs ? ?/sec 1.01 1511.8±10.19µs ? ?/sec
physical_plan_clickbench_q8 1.00 1984.6±16.87µs ? ?/sec 1.00 1984.2±15.51µs ? ?/sec
physical_plan_clickbench_q9 1.00 1984.9±14.76µs ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_struct_join_agg_sort 1.00 1328.2±3.66µs ? ?/sec 1.01 1334.9±4.78µs ? ?/sec
physical_plan_tpcds_all 1.01 729.1±5.81ms ? ?/sec 1.00 724.1±5.01ms ? ?/sec
physical_plan_tpch_all 1.02 46.6±0.34ms ? ?/sec 1.00 45.5±0.07ms ? ?/sec
physical_plan_tpch_q1 1.00 1609.1±9.54µs ? ?/sec 1.01 1617.2±4.54µs ? ?/sec
physical_plan_tpch_q10 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_tpch_q11 1.00 2.3±0.00ms ? ?/sec 1.01 2.3±0.00ms ? ?/sec
physical_plan_tpch_q12 1.00 1297.2±2.99µs ? ?/sec 1.02 1317.5±2.32µs ? ?/sec
physical_plan_tpch_q13 1.00 1074.9±2.95µs ? ?/sec 1.01 1087.2±1.99µs ? ?/sec
physical_plan_tpch_q14 1.00 1441.1±6.02µs ? ?/sec 1.00 1443.7±3.94µs ? ?/sec
physical_plan_tpch_q16 1.00 1665.2±4.01µs ? ?/sec 1.01 1684.0±4.05µs ? ?/sec
physical_plan_tpch_q17 1.00 1698.4±4.28µs ? ?/sec 1.00 1701.1±4.58µs ? ?/sec
physical_plan_tpch_q18 1.00 2.0±0.00ms ? ?/sec 1.01 2.1±0.00ms ? ?/sec
physical_plan_tpch_q19 1.00 1929.1±5.67µs ? ?/sec 1.01 1948.4±3.07µs ? ?/sec
physical_plan_tpch_q2 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_plan_tpch_q20 1.00 2.2±0.00ms ? ?/sec 1.01 2.2±0.00ms ? ?/sec
physical_plan_tpch_q21 1.00 2.9±0.01ms ? ?/sec 1.01 2.9±0.01ms ? ?/sec
physical_plan_tpch_q22 1.02 1568.4±3.29µs ? ?/sec 1.00 1536.3±2.84µs ? ?/sec
physical_plan_tpch_q3 1.00 1913.5±4.06µs ? ?/sec 1.01 1935.3±3.40µs ? ?/sec
physical_plan_tpch_q4 1.00 1223.0±2.34µs ? ?/sec 1.02 1251.9±4.10µs ? ?/sec
physical_plan_tpch_q5 1.00 2.8±0.01ms ? ?/sec 1.01 2.8±0.00ms ? ?/sec
physical_plan_tpch_q6 1.00 638.8±1.36µs ? ?/sec 1.02 651.2±1.25µs ? ?/sec
physical_plan_tpch_q7 1.00 2.9±0.00ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_tpch_q8 1.00 3.9±0.01ms ? ?/sec 1.01 3.9±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 2.8±0.01ms ? ?/sec 1.01 2.8±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.00 15.5±0.04ms ? ?/sec 1.01 15.7±0.09ms ? ?/sec
physical_select_all_from_1000 1.00 115.8±0.55ms ? ?/sec 1.03 119.7±1.63ms ? ?/sec
physical_select_one_from_700 1.00 770.2±1.42µs ? ?/sec 1.01 775.4±1.98µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.01 4.2±0.01ms ? ?/sec 1.00 4.2±0.01ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.03 9.1±0.02ms ? ?/sec 1.00 8.8±0.09ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.02 97.0±0.50ms ? ?/sec 1.00 95.5±0.82ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.09 401.9±3.79ms ? ?/sec 1.00 367.7±1.63ms ? ?/sec
physical_theta_join_consider_sort 1.00 1069.2±2.67µs ? ?/sec 1.02 1085.8±2.44µs ? ?/sec
physical_unnest_to_join 1.00 634.5±2.11µs ? ?/sec 1.01 643.2±2.41µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 729.6±1.69µs ? ?/sec 1.00 730.4±1.37µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.01 1429.3±4.01µs ? ?/sec 1.00 1415.3±5.72µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.01 450.8±1.17µs ? ?/sec 1.00 448.2±1.58µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 553.5±1.25µs ? ?/sec 1.00 555.8±2.24µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.01 593.3±2.15µs ? ?/sec 1.00 586.9±1.96µs ? ?/sec
with_param_values_many_columns 1.00 431.4±2.18µs ? ?/sec 1.02 440.2±2.22µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2275.5s |
| Peak memory | 130.0 MiB |
| Avg memory | 70.5 MiB |
| CPU user | 1892.2s |
| CPU sys | 1.3s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2310.5s |
| Peak memory | 132.0 MiB |
| Avg memory | 70.3 MiB |
| CPU user | 1884.4s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#discussion_r3826972346
##########
datafusion/physical-expr/src/equivalence/properties/mod.rs:
##########
@@ -1169,6 +1169,23 @@ impl EquivalenceProperties {
/// `output_schema`.
pub fn project(&self, mapping: &ProjectionMapping, output_schema: SchemaRef) -> Self {
let eq_group = self.eq_group.project(mapping);
+ self.project_with_eq_group(mapping, output_schema, eq_group)
+ }
+
+ /// Same as [`Self::project`], but takes an already-projected equivalence
+ /// group instead of computing one.
+ ///
+ /// [`EquivalenceGroup::project`] is a pure function of the group and the
+ /// mapping, so a caller that knows both are unchanged since the last
+ /// projection can hand back the previous result rather than recomputing an
+ /// identical one. Orderings are still derived here: they are precisely what
+ /// changes when a sort is introduced below this node.
+ pub fn project_with_eq_group(
Review Comment:
Done, that shape is better and it removes the problem rather than asserting around it. `project_reusing` now takes `previous` and `cached`, compares the groups itself and falls back to a full projection, so the `debug_assert` and the unchecked entry point it needed are both gone.
One knock-on worth flagging: the decision moving into `physical-expr` means `ProjectionExec` no longer knows whether reuse happened, and `cfg(test)` does not reach across crates, so the counter you asked for earlier could not stay where it was. The tests split by what they prove — `physical-expr` asserts the reuse fired, `physical-plan` asserts the resulting properties are right. Disabling the reuse still fails the former.
I did not need `projected_from`: since the signature takes `previous`, the comparison has both sides already.
On the `Arc` — you are right and my stated reason was wrong; I checked your three points and they hold, so I have corrected the description. Keeping it separate for a different reason: `Arc::make_mut` adds a cost at every mutation, and `ptr_eq` changes *which* cases hit rather than only how fast the check is, so it wants its own before/after rather than riding along in a PR whose numbers are already settled. Happy to do it next, or here if you would rather see them together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5365508977
Thanks again @adriangb and @Dandandan for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5364955364
> thanks @zhuqi-lucas, assuming the bench still looks good lgtm
Thank you @adriangb for review and good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#discussion_r3822873955
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -194,6 +219,55 @@ impl ProjectionExec {
})
}
+ /// Like [`Self::try_from_projector`], but reuses `eq_group` as the output
+ /// equivalence group instead of projecting the input's group again.
+ ///
+ /// [`EquivalenceGroup::project`] is a pure function of the group and the
+ /// mapping, so reuse is sound exactly when both are unchanged.
+ ///
+ /// The caller establishes the first by comparing the old and new child
+ /// groups. The second holds because the mapping comes from
+ /// `projector.projection()`, carried over untouched, and from the child's
+ /// schema, which `ProjectionMapping::try_new` consults only for field names
+ /// and indices -- never for types or nullability. So a child differing only
+ /// in nullability keeps the same mapping. A child that renamed or reordered
+ /// those fields would change the group too, since its members are `Column`s
+ /// carrying those names, and the comparison above would reject it; were one
+ /// to slip through anyway, `try_new`'s name assertion errors out rather than
+ /// letting a stale group into the plan.
+ fn try_from_projector_reusing_eq_group(
+ projector: Projector,
+ input: Arc<dyn ExecutionPlan>,
+ eq_group: EquivalenceGroup,
+ ) -> Result<Self> {
Review Comment:
This duplicates `compute_properties` and can drift if `PlanProperties` grows a new field.
The duplication exists only because reuse is expressed as a second constructor. But the two paths differ in exactly one expression — how eq_properties is obtained. Everything else (output_partitioning, pipeline_behavior, boundedness, PlanProperties::new, the Self { … } literal) is copied. So push the difference down into compute_properties as a parameter and delete the second constructor:
```rust
fn compute_properties(
input: &Arc<dyn ExecutionPlan>,
projection_mapping: &ProjectionMapping,
schema: SchemaRef,
reused_eq_group: Option<EquivalenceGroup>, // <- the only new knob
) -> Result<PlanProperties> {
let input_eq_properties = input.equivalence_properties();
// The only thing reuse changes. Everything below is common, so the two
// paths cannot drift.
let eq_properties = match reused_eq_group {
Some(eq_group) => input_eq_properties
.project_with_eq_group(projection_mapping, schema, eq_group),
None => input_eq_properties.project(projection_mapping, schema),
};
// ... unchanged: partitioning, PlanProperties::new
}
fn try_from_projector(projector, input) -> Result<Self> {
Self::try_from_projector_with_eq_group(projector, input, None)
}
fn try_from_projector_with_eq_group(
projector: Projector,
input: Arc<dyn ExecutionPlan>,
reused_eq_group: Option<EquivalenceGroup>,
) -> Result<Self> { /* the single existing body */ }
```
`replace_children(Recompute)` then loses its early return and its duplicated call. The guard just produces the Option:
```rust
let child = children.swap_remove(0);
let reused_eq_group = if self.input.equivalence_properties().eq_group()
.has_same_classes(child.equivalence_properties().eq_group())
{
#[cfg(test)]
eq_group_reuse_probe::record_hit();
Some(self.cache.equivalence_properties().eq_group().clone())
} else {
None
};
ProjectionExec::try_from_projector_with_eq_group(self.projector.clone(), child, reused_eq_group)
.map(|p| Arc::new(p) as _)
```
We could also move the guard into EquivalenceProperties instead of the group:
```rust
pub fn project_reusing(
&self,
mapping: &ProjectionMapping,
output_schema: SchemaRef,
// (the eq properties this was last projected from, the group that produced)
previous: Option<(&EquivalenceGroup, &EquivalenceGroup)>,
) -> Self
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#discussion_r3823023448
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -194,6 +219,55 @@ impl ProjectionExec {
})
}
+ /// Like [`Self::try_from_projector`], but reuses `eq_group` as the output
+ /// equivalence group instead of projecting the input's group again.
+ ///
+ /// [`EquivalenceGroup::project`] is a pure function of the group and the
+ /// mapping, so reuse is sound exactly when both are unchanged.
+ ///
+ /// The caller establishes the first by comparing the old and new child
+ /// groups. The second holds because the mapping comes from
+ /// `projector.projection()`, carried over untouched, and from the child's
+ /// schema, which `ProjectionMapping::try_new` consults only for field names
+ /// and indices -- never for types or nullability. So a child differing only
+ /// in nullability keeps the same mapping. A child that renamed or reordered
+ /// those fields would change the group too, since its members are `Column`s
+ /// carrying those names, and the comparison above would reject it; were one
+ /// to slip through anyway, `try_new`'s name assertion errors out rather than
+ /// letting a stale group into the plan.
+ fn try_from_projector_reusing_eq_group(
+ projector: Projector,
+ input: Arc<dyn ExecutionPlan>,
+ eq_group: EquivalenceGroup,
+ ) -> Result<Self> {
Review Comment:
Done, and it reads much better. `compute_properties` takes the optional group, `try_from_projector` delegates with `None`, and there is one body instead of two. `replace_children` loses the early return and the second call site as you sketched.
Left the guard in `ProjectionExec` rather than moving it into `EquivalenceProperties`. The soundness argument leans on facts the plan node owns -- that the projector was carried over untouched, so the mapping is unchanged -- and the `(previous_group, produced_group)` pair reads as a cache key that `EquivalenceProperties` would have no way to validate. Happy to move it if you would rather it live there.
No behaviour change; deleting the fast path still fails the two tests that assert it is taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#discussion_r3818560383
##########
datafusion/physical-expr/src/equivalence/properties/mod.rs:
##########
@@ -1169,6 +1169,23 @@ impl EquivalenceProperties {
/// `output_schema`.
pub fn project(&self, mapping: &ProjectionMapping, output_schema: SchemaRef) -> Self {
let eq_group = self.eq_group.project(mapping);
+ self.project_with_eq_group(mapping, output_schema, eq_group)
+ }
+
+ /// Same as [`Self::project`], but takes an already-projected equivalence
+ /// group instead of computing one.
+ ///
+ /// [`EquivalenceGroup::project`] is a pure function of the group and the
+ /// mapping, so a caller that knows both are unchanged since the last
+ /// projection can hand back the previous result rather than recomputing an
+ /// identical one. Orderings are still derived here: they are precisely what
+ /// changes when a sort is introduced below this node.
+ pub fn project_with_eq_group(
Review Comment:
Good catch. It cannot be a hard error without performing the projection the caller is skipping, so it is a `debug_assert!` now. `project` routes around it through a private unchecked entry point, since it builds the group itself and satisfies the precondition by construction. Added a `should_panic` test so the assertion cannot quietly stop being wired up.
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -2215,4 +2289,281 @@ mod tests {
Ok(())
}
+
+ /// `EmptyExec(a, b, c)` under a filter that equates `lhs` and `rhs`, so the
+ /// child carries a non-trivial equivalence group.
+ fn filtered_source(lhs: &str, rhs: &str) -> Result<Arc<dyn ExecutionPlan>> {
+ filtered_source_with_nullability(lhs, rhs, false)
+ }
+
+ /// As [`filtered_source`], but `nullable` varies the schema's nullability
+ /// while leaving field names and order alone.
+ fn filtered_source_with_nullability(
+ lhs: &str,
+ rhs: &str,
+ nullable: bool,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Int32, nullable),
+ Field::new("b", DataType::Int32, nullable),
+ Field::new("c", DataType::Int32, nullable),
+ ]));
+ let input: Arc<dyn ExecutionPlan> = Arc::new(EmptyExec::new(Arc::clone(&schema)));
+ let predicate = binary(
+ col(lhs, &schema)?,
+ Operator::Eq,
+ col(rhs, &schema)?,
+ &schema,
+ )?;
+ Ok(Arc::new(FilterExec::try_new(predicate, input)?))
+ }
+
+ /// `[a AS x, b AS y, c AS z]` against `filtered_source`'s schema.
+ fn renaming_exprs(schema: &SchemaRef) -> Result<Vec<ProjectionExpr>> {
+ [("a", "x"), ("b", "y"), ("c", "z")]
+ .into_iter()
+ .map(|(source, alias)| {
+ Ok(ProjectionExpr {
+ expr: col(source, schema)?,
+ alias: alias.to_string(),
+ })
+ })
+ .collect()
+ }
+
+ fn assert_same_properties(actual: &dyn ExecutionPlan, expected: &ProjectionExec) {
+ let actual_props = actual.properties().equivalence_properties();
+ let expected_props = expected.properties().equivalence_properties();
+ assert!(
+ actual_props
+ .eq_group()
+ .has_same_classes(expected_props.eq_group()),
+ "equivalence group: {:?} vs {:?}",
+ actual_props.eq_group(),
+ expected_props.eq_group()
+ );
+ assert_eq!(
+ actual_props.oeq_class(),
+ expected_props.oeq_class(),
+ "orderings"
+ );
+ assert_eq!(
+ actual_props.constraints(),
+ expected_props.constraints(),
+ "constraints"
+ );
+ assert_eq!(actual_props.schema(), expected_props.schema(), "schema");
+ // `Partitioning` has no `PartialEq`, so compare the partition count and
+ // the explicit `Display` form. Derived `Debug` would change with any
+ // field addition, making this brittle for no gain.
+ let actual_partitioning = actual.properties().output_partitioning();
+ let expected_partitioning = expected.properties().output_partitioning();
+ assert_eq!(
+ actual_partitioning.partition_count(),
+ expected_partitioning.partition_count(),
+ "partition count"
+ );
+ assert_eq!(
+ actual_partitioning.to_string(),
+ expected_partitioning.to_string(),
+ "partitioning"
+ );
+ }
+
+ #[test]
+ fn test_sort_below_changes_orderings_but_not_the_equivalence_group() -> Result<()> {
+ // The premise the fast path rests on. If a sort ever starts altering
+ // the equivalence group, reusing the cached group becomes unsound and
+ // this test is the one that should fail first.
+ let child = filtered_source("a", "b")?;
+ let ordering = LexOrdering::new(vec![PhysicalSortExpr::new_default(col(
+ "c",
+ &child.schema(),
+ )?)])
+ .expect("non-empty ordering");
+ let sorted = SortExec::new(ordering, Arc::clone(&child));
+
+ let child_props = child.properties().equivalence_properties();
+ let sorted_props = sorted.properties().equivalence_properties();
+
+ assert!(
+ !child_props.eq_group().is_empty(),
+ "the filter did not produce an equivalence class"
+ );
+ assert!(
+ child_props
+ .eq_group()
+ .has_same_classes(sorted_props.eq_group()),
+ "sorting altered the equivalence group"
+ );
+ assert_ne!(
+ child_props.oeq_class(),
+ sorted_props.oeq_class(),
+ "sorting did not alter the orderings"
+ );
+
+ Ok(())
+ }
+
+ #[test]
+ fn test_replace_children_reuses_eq_group_when_only_orderings_change() -> Result<()> {
Review Comment:
You were right, they passed with the fast path deleted. There is a `cfg(test)` probe now and the tests assert the reuse count: one for the two cases that should reuse, zero for the case where the child equivalence group changed. Deleting the fast path fails the first two. The probe is thread local rather than a global counter, since tests run in parallel and a shared count would let one test see another's hits.
##########
datafusion/physical-expr/src/equivalence/class.rs:
##########
@@ -310,6 +310,33 @@ pub struct EquivalenceGroup {
}
impl EquivalenceGroup {
+ /// A cheap, deliberately conservative check that two groups hold the same
+ /// equivalence classes.
+ ///
+ /// This is not `PartialEq`, and the distinction is the point. `classes` is a
+ /// `Vec` whose order carries no meaning -- `remove_class_at_idx` uses
+ /// `swap_remove` -- so two groups describing exactly the same equalities can
+ /// hold their classes in different orders and this returns `false` for them.
+ /// Naming it `PartialEq` would invite callers to read it as semantic equality,
+ /// which it is not.
+ ///
+ /// The comparison is positional because it runs on a hot path:
+ /// `ProjectionExec` consults it every time a rule replaces its child. Set
+ /// semantics would mean scanning the other group per class, and the quadratic
+ /// blowup costs far more than the recomputation it is trying to avoid --
+ /// measured at 8x for 32 classes and 15x for 64.
Review Comment:
Agreed, dropped them. The reason the comparison is positional does not depend on the exact numbers, so the doc just states the shape of the cost now and the measurements stay in the commit that introduced it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5351980794
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5325475432
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5324655706)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing qizhu/reuse-eq-group-on-child-swap (82edbc3b2c6a77922a40d5f467565abc48456c07) to 6eaca8b (merge-base) [diff](https://github.com/apache/datafusion/compare/6eaca8bfe8421faca1413f29f71fcaf0ccaf631a..82edbc3b2c6a77922a40d5f467565abc48456c07)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD qizhu_reuse-eq-group-on-child-swap
----- ---- ----------------------------------
logical_aggregate_with_join 1.00 450.4±1.62µs ? ?/sec 1.00 449.9±1.39µs ? ?/sec
logical_correlated_subquery_exists 1.00 283.1±0.75µs ? ?/sec 1.00 284.0±1.50µs ? ?/sec
logical_correlated_subquery_in 1.00 285.9±0.83µs ? ?/sec 1.00 286.1±0.66µs ? ?/sec
logical_distinct_many_columns 1.00 575.9±1.44µs ? ?/sec 1.00 573.5±1.10µs ? ?/sec
logical_join_4_with_agg_and_filter 1.00 254.5±2.47µs ? ?/sec 1.00 253.6±1.86µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.00 427.1±4.67µs ? ?/sec 1.00 428.7±2.52µs ? ?/sec
logical_join_chain_16 1.00 676.0±3.97µs ? ?/sec 1.00 673.2±4.15µs ? ?/sec
logical_join_chain_4 1.00 122.9±0.52µs ? ?/sec 1.00 122.4±0.41µs ? ?/sec
logical_join_chain_8 1.00 250.0±1.30µs ? ?/sec 1.00 248.9±1.07µs ? ?/sec
logical_multiple_subqueries 1.00 517.9±2.24µs ? ?/sec 1.01 523.8±8.38µs ? ?/sec
logical_nested_cte_4_levels 1.00 260.5±1.35µs ? ?/sec 1.02 264.9±0.89µs ? ?/sec
logical_plan_struct_join_agg_sort 1.03 182.2±1.00µs ? ?/sec 1.00 177.8±1.29µs ? ?/sec
logical_plan_tpcds_all 1.00 93.9±0.18ms ? ?/sec 1.00 94.2±0.18ms ? ?/sec
logical_plan_tpch_all 1.00 6.6±0.03ms ? ?/sec 1.00 6.6±0.02ms ? ?/sec
logical_scalar_subquery 1.00 306.9±4.18µs ? ?/sec 1.01 308.6±1.23µs ? ?/sec
logical_select_all_from_1000 1.00 105.0±0.11ms ? ?/sec 1.00 104.5±0.11ms ? ?/sec
logical_select_one_from_700 1.00 326.2±2.24µs ? ?/sec 1.00 324.9±2.12µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 283.9±0.59µs ? ?/sec 1.00 283.7±0.84µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 271.9±0.79µs ? ?/sec 1.00 271.0±0.74µs ? ?/sec
logical_union_4_branches 1.00 422.0±4.00µs ? ?/sec 1.01 425.5±1.19µs ? ?/sec
logical_union_8_branches 1.00 804.7±2.32µs ? ?/sec 1.01 808.8±2.56µs ? ?/sec
logical_wide_aggregate_100_exprs 1.00 4.5±0.01ms ? ?/sec 1.00 4.5±0.01ms ? ?/sec
logical_wide_case_50_exprs 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
logical_wide_filter_200_predicates 1.00 1307.6±8.06µs ? ?/sec 1.01 1316.7±7.90µs ? ?/sec
logical_wide_filter_50_predicates 1.00 387.7±2.57µs ? ?/sec 1.00 389.5±2.24µs ? ?/sec
optimizer_correlated_exists 1.01 249.8±0.56µs ? ?/sec 1.00 248.3±0.94µs ? ?/sec
optimizer_join_4_with_agg_filter 1.00 483.4±1.09µs ? ?/sec 1.01 488.7±1.54µs ? ?/sec
optimizer_join_chain_4 1.01 188.7±0.41µs ? ?/sec 1.00 187.7±0.57µs ? ?/sec
optimizer_join_chain_8 1.00 574.4±1.30µs ? ?/sec 1.00 576.5±1.49µs ? ?/sec
optimizer_select_all_from_1000 1.00 6.8±0.01ms ? ?/sec 1.00 6.8±0.01ms ? ?/sec
optimizer_select_one_from_700 1.01 254.2±0.58µs ? ?/sec 1.00 252.9±0.72µs ? ?/sec
optimizer_tpcds_all 1.01 320.0±1.79ms ? ?/sec 1.00 318.4±0.36ms ? ?/sec
optimizer_tpch_all 1.00 18.1±0.04ms ? ?/sec 1.00 18.1±0.04ms ? ?/sec
optimizer_wide_aggregate_100 1.00 2.3±0.00ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
optimizer_wide_filter_200 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_intersection 1.00 597.5±1.68µs ? ?/sec 1.01 601.2±2.49µs ? ?/sec
physical_join_consider_sort 1.00 1055.3±7.26µs ? ?/sec 1.00 1050.3±3.73µs ? ?/sec
physical_join_distinct 1.01 265.4±0.64µs ? ?/sec 1.00 263.9±0.66µs ? ?/sec
physical_many_self_joins 1.00 7.6±0.02ms ? ?/sec 1.00 7.6±0.02ms ? ?/sec
physical_plan_clickbench_all 1.00 129.9±0.41ms ? ?/sec 1.00 129.7±0.42ms ? ?/sec
physical_plan_clickbench_q1 1.00 1407.3±7.49µs ? ?/sec 1.00 1400.8±8.07µs ? ?/sec
physical_plan_clickbench_q10 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q11 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q12 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q13 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q14 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q15 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q16 1.00 1820.3±7.74µs ? ?/sec 1.00 1824.5±7.84µs ? ?/sec
physical_plan_clickbench_q17 1.00 1869.4±7.14µs ? ?/sec 1.00 1876.9±6.83µs ? ?/sec
physical_plan_clickbench_q18 1.00 1696.7±8.08µs ? ?/sec 1.01 1717.6±8.05µs ? ?/sec
physical_plan_clickbench_q19 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q2 1.01 1815.2±7.62µs ? ?/sec 1.00 1804.7±7.01µs ? ?/sec
physical_plan_clickbench_q20 1.00 1539.6±7.60µs ? ?/sec 1.02 1563.6±7.41µs ? ?/sec
physical_plan_clickbench_q21 1.00 1790.6±6.82µs ? ?/sec 1.01 1811.5±7.29µs ? ?/sec
physical_plan_clickbench_q22 1.00 2.2±0.01ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q23 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q24 1.02 6.8±0.03ms ? ?/sec 1.00 6.7±0.02ms ? ?/sec
physical_plan_clickbench_q25 1.03 1970.7±10.12µs ? ?/sec 1.00 1915.9±7.72µs ? ?/sec
physical_plan_clickbench_q26 1.00 1769.6±7.47µs ? ?/sec 1.00 1765.4±8.05µs ? ?/sec
physical_plan_clickbench_q27 1.01 1977.7±7.51µs ? ?/sec 1.00 1967.0±6.20µs ? ?/sec
physical_plan_clickbench_q28 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q29 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q3 1.01 1705.0±7.21µs ? ?/sec 1.00 1692.6±7.48µs ? ?/sec
physical_plan_clickbench_q30 1.02 15.7±0.05ms ? ?/sec 1.00 15.4±0.26ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q33 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q34 1.00 1828.3±7.16µs ? ?/sec 1.01 1838.2±16.27µs ? ?/sec
physical_plan_clickbench_q35 1.00 1876.9±6.49µs ? ?/sec 1.00 1871.8±8.93µs ? ?/sec
physical_plan_clickbench_q36 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q37 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q38 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q39 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q4 1.01 1516.5±6.21µs ? ?/sec 1.00 1505.9±8.56µs ? ?/sec
physical_plan_clickbench_q40 1.00 3.4±0.01ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
physical_plan_clickbench_q41 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.1±0.01ms ? ?/sec 1.01 3.1±0.01ms ? ?/sec
physical_plan_clickbench_q43 1.00 3.2±0.01ms ? ?/sec 1.02 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q44 1.00 1586.4±7.10µs ? ?/sec 1.02 1623.4±9.13µs ? ?/sec
physical_plan_clickbench_q45 1.00 1593.1±7.22µs ? ?/sec 1.02 1625.4±8.23µs ? ?/sec
physical_plan_clickbench_q46 1.00 1921.2±6.73µs ? ?/sec 1.02 1957.5±8.66µs ? ?/sec
physical_plan_clickbench_q47 1.01 2.7±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q48 1.01 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_clickbench_q49 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_clickbench_q5 1.00 1660.0±8.10µs ? ?/sec 1.00 1652.0±13.88µs ? ?/sec
physical_plan_clickbench_q50 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q51 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q6 1.00 1639.0±5.92µs ? ?/sec 1.02 1668.4±14.79µs ? ?/sec
physical_plan_clickbench_q7 1.00 1455.1±6.90µs ? ?/sec 1.01 1468.6±10.57µs ? ?/sec
physical_plan_clickbench_q8 1.00 1953.6±6.20µs ? ?/sec 1.01 1966.8±12.57µs ? ?/sec
physical_plan_clickbench_q9 1.00 1972.9±8.63µs ? ?/sec 1.00 1982.0±14.67µs ? ?/sec
physical_plan_struct_join_agg_sort 1.01 1357.6±2.65µs ? ?/sec 1.00 1343.8±3.40µs ? ?/sec
physical_plan_tpcds_all 1.02 737.9±3.39ms ? ?/sec 1.00 726.9±1.07ms ? ?/sec
physical_plan_tpch_all 1.01 45.9±0.08ms ? ?/sec 1.00 45.6±0.07ms ? ?/sec
physical_plan_tpch_q1 1.01 1615.5±3.02µs ? ?/sec 1.00 1603.8±2.89µs ? ?/sec
physical_plan_tpch_q10 1.02 2.9±0.01ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.01 2.3±0.05ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
physical_plan_tpch_q12 1.00 1295.8±18.16µs ? ?/sec 1.01 1307.1±2.43µs ? ?/sec
physical_plan_tpch_q13 1.00 1070.7±5.06µs ? ?/sec 1.02 1088.4±4.40µs ? ?/sec
physical_plan_tpch_q14 1.00 1473.9±3.34µs ? ?/sec 1.00 1476.0±6.32µs ? ?/sec
physical_plan_tpch_q16 1.00 1665.4±11.68µs ? ?/sec 1.01 1676.8±6.94µs ? ?/sec
physical_plan_tpch_q17 1.00 1705.2±4.98µs ? ?/sec 1.00 1710.2±8.83µs ? ?/sec
physical_plan_tpch_q18 1.00 2.0±0.00ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q19 1.00 1937.0±2.90µs ? ?/sec 1.01 1964.8±10.73µs ? ?/sec
physical_plan_tpch_q2 1.02 3.7±0.03ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
physical_plan_tpch_q20 1.00 2.2±0.00ms ? ?/sec 1.01 2.2±0.01ms ? ?/sec
physical_plan_tpch_q21 1.00 2.9±0.00ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_tpch_q22 1.00 1535.1±3.40µs ? ?/sec 1.00 1535.5±3.15µs ? ?/sec
physical_plan_tpch_q3 1.01 1955.9±13.44µs ? ?/sec 1.00 1927.4±3.56µs ? ?/sec
physical_plan_tpch_q4 1.01 1253.5±6.83µs ? ?/sec 1.00 1240.4±3.91µs ? ?/sec
physical_plan_tpch_q5 1.01 2.8±0.01ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
physical_plan_tpch_q6 1.01 657.0±1.34µs ? ?/sec 1.00 649.0±1.86µs ? ?/sec
physical_plan_tpch_q7 1.01 2.9±0.01ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 4.0±0.01ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_tpch_q9 1.02 2.8±0.00ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.00 15.6±0.03ms ? ?/sec 1.00 15.6±0.04ms ? ?/sec
physical_select_all_from_1000 1.00 116.0±0.13ms ? ?/sec 1.00 115.5±0.14ms ? ?/sec
physical_select_one_from_700 1.00 765.6±1.84µs ? ?/sec 1.00 763.0±3.32µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.01 4.3±0.01ms ? ?/sec 1.00 4.2±0.01ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.05 9.2±0.02ms ? ?/sec 1.00 8.8±0.02ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.01 96.8±0.22ms ? ?/sec 1.00 95.5±0.30ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.10 404.4±1.87ms ? ?/sec 1.00 368.9±0.93ms ? ?/sec
physical_theta_join_consider_sort 1.00 1077.1±7.15µs ? ?/sec 1.00 1077.8±2.61µs ? ?/sec
physical_unnest_to_join 1.00 637.3±1.99µs ? ?/sec 1.00 639.9±1.78µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.01 729.1±1.53µs ? ?/sec 1.00 723.2±1.58µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.02 1436.8±4.48µs ? ?/sec 1.00 1410.1±3.29µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.01 455.9±1.51µs ? ?/sec 1.00 452.6±1.17µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 553.6±1.84µs ? ?/sec 1.00 550.9±1.59µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.01 592.4±1.87µs ? ?/sec 1.00 589.3±2.07µs ? ?/sec
with_param_values_many_columns 1.01 437.1±2.57µs ? ?/sec 1.00 433.1±2.17µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2400.5s |
| Peak memory | 127.8 MiB |
| Avg memory | 65.1 MiB |
| CPU user | 1897.4s |
| CPU sys | 1.3s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2395.5s |
| Peak memory | 130.1 MiB |
| Avg memory | 66.6 MiB |
| CPU user | 1888.5s |
| CPU sys | 1.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#discussion_r3823023448
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -194,6 +219,55 @@ impl ProjectionExec {
})
}
+ /// Like [`Self::try_from_projector`], but reuses `eq_group` as the output
+ /// equivalence group instead of projecting the input's group again.
+ ///
+ /// [`EquivalenceGroup::project`] is a pure function of the group and the
+ /// mapping, so reuse is sound exactly when both are unchanged.
+ ///
+ /// The caller establishes the first by comparing the old and new child
+ /// groups. The second holds because the mapping comes from
+ /// `projector.projection()`, carried over untouched, and from the child's
+ /// schema, which `ProjectionMapping::try_new` consults only for field names
+ /// and indices -- never for types or nullability. So a child differing only
+ /// in nullability keeps the same mapping. A child that renamed or reordered
+ /// those fields would change the group too, since its members are `Column`s
+ /// carrying those names, and the comparison above would reject it; were one
+ /// to slip through anyway, `try_new`'s name assertion errors out rather than
+ /// letting a stale group into the plan.
+ fn try_from_projector_reusing_eq_group(
+ projector: Projector,
+ input: Arc<dyn ExecutionPlan>,
+ eq_group: EquivalenceGroup,
+ ) -> Result<Self> {
Review Comment:
Thanks @adriangb , very good suggestion!
Done, and it reads much better. `compute_properties` takes the optional group, `try_from_projector` delegates with `None`, and there is one body instead of two. `replace_children` loses the early return and the second call site as you sketched.
Left the guard in `ProjectionExec` rather than moving it into `EquivalenceProperties`. The soundness argument leans on facts the plan node owns -- that the projector was carried over untouched, so the mapping is unchanged -- and the `(previous_group, produced_group)` pair reads as a cache key that `EquivalenceProperties` would have no way to validate. Happy to move it if you would rather it live there.
No behaviour change; deleting the fast path still fails the two tests that assert it is taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5338124571
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5338111275)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5338111275-1718-59rxw 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (09d9cd2d037621aac741f053f87e185cea92fe67) to c429919 (merge-base) [diff](https://github.com/apache/datafusion/compare/c429919c7c8bd81e4e687531a609e2380ed60649..09d9cd2d037621aac741f053f87e185cea92fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5338111275
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#issuecomment-5339115860
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24445#issuecomment-5339090159)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5339090159-1734-d742x 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing qizhu/reuse-eq-group-on-child-swap (95cc5360b98bf7d506c9c004ccfe69917fda6c9e) to c429919 (merge-base) [diff](https://github.com/apache/datafusion/compare/c429919c7c8bd81e4e687531a609e2380ed60649..95cc5360b98bf7d506c9c004ccfe69917fda6c9e)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse the projected equivalence group when only child orderings change [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #24445:
URL: https://github.com/apache/datafusion/pull/24445#discussion_r3814253573
##########
datafusion/physical-expr/src/equivalence/properties/mod.rs:
##########
@@ -1169,6 +1169,23 @@ impl EquivalenceProperties {
/// `output_schema`.
pub fn project(&self, mapping: &ProjectionMapping, output_schema: SchemaRef) -> Self {
let eq_group = self.eq_group.project(mapping);
+ self.project_with_eq_group(mapping, output_schema, eq_group)
+ }
+
+ /// Same as [`Self::project`], but takes an already-projected equivalence
+ /// group instead of computing one.
+ ///
+ /// [`EquivalenceGroup::project`] is a pure function of the group and the
+ /// mapping, so a caller that knows both are unchanged since the last
+ /// projection can hand back the previous result rather than recomputing an
+ /// identical one. Orderings are still derived here: they are precisely what
+ /// changes when a sort is introduced below this node.
+ pub fn project_with_eq_group(
Review Comment:
This is a public function that has an unchecked precondition: a caller passing a group that isn't `self.eq_group.project(mapping)` silently gets wrong equivalence properties. Should we make that an error or something?
##########
datafusion/physical-expr/src/equivalence/class.rs:
##########
@@ -310,6 +310,33 @@ pub struct EquivalenceGroup {
}
impl EquivalenceGroup {
+ /// A cheap, deliberately conservative check that two groups hold the same
+ /// equivalence classes.
+ ///
+ /// This is not `PartialEq`, and the distinction is the point. `classes` is a
+ /// `Vec` whose order carries no meaning -- `remove_class_at_idx` uses
+ /// `swap_remove` -- so two groups describing exactly the same equalities can
+ /// hold their classes in different orders and this returns `false` for them.
+ /// Naming it `PartialEq` would invite callers to read it as semantic equality,
+ /// which it is not.
+ ///
+ /// The comparison is positional because it runs on a hot path:
+ /// `ProjectionExec` consults it every time a rule replaces its child. Set
+ /// semantics would mean scanning the other group per class, and the quadratic
+ /// blowup costs far more than the recomputation it is trying to avoid --
+ /// measured at 8x for 32 classes and 15x for 64.
Review Comment:
I worry these measured multipliers will bitrot
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -2215,4 +2289,281 @@ mod tests {
Ok(())
}
+
+ /// `EmptyExec(a, b, c)` under a filter that equates `lhs` and `rhs`, so the
+ /// child carries a non-trivial equivalence group.
+ fn filtered_source(lhs: &str, rhs: &str) -> Result<Arc<dyn ExecutionPlan>> {
+ filtered_source_with_nullability(lhs, rhs, false)
+ }
+
+ /// As [`filtered_source`], but `nullable` varies the schema's nullability
+ /// while leaving field names and order alone.
+ fn filtered_source_with_nullability(
+ lhs: &str,
+ rhs: &str,
+ nullable: bool,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Int32, nullable),
+ Field::new("b", DataType::Int32, nullable),
+ Field::new("c", DataType::Int32, nullable),
+ ]));
+ let input: Arc<dyn ExecutionPlan> = Arc::new(EmptyExec::new(Arc::clone(&schema)));
+ let predicate = binary(
+ col(lhs, &schema)?,
+ Operator::Eq,
+ col(rhs, &schema)?,
+ &schema,
+ )?;
+ Ok(Arc::new(FilterExec::try_new(predicate, input)?))
+ }
+
+ /// `[a AS x, b AS y, c AS z]` against `filtered_source`'s schema.
+ fn renaming_exprs(schema: &SchemaRef) -> Result<Vec<ProjectionExpr>> {
+ [("a", "x"), ("b", "y"), ("c", "z")]
+ .into_iter()
+ .map(|(source, alias)| {
+ Ok(ProjectionExpr {
+ expr: col(source, schema)?,
+ alias: alias.to_string(),
+ })
+ })
+ .collect()
+ }
+
+ fn assert_same_properties(actual: &dyn ExecutionPlan, expected: &ProjectionExec) {
+ let actual_props = actual.properties().equivalence_properties();
+ let expected_props = expected.properties().equivalence_properties();
+ assert!(
+ actual_props
+ .eq_group()
+ .has_same_classes(expected_props.eq_group()),
+ "equivalence group: {:?} vs {:?}",
+ actual_props.eq_group(),
+ expected_props.eq_group()
+ );
+ assert_eq!(
+ actual_props.oeq_class(),
+ expected_props.oeq_class(),
+ "orderings"
+ );
+ assert_eq!(
+ actual_props.constraints(),
+ expected_props.constraints(),
+ "constraints"
+ );
+ assert_eq!(actual_props.schema(), expected_props.schema(), "schema");
+ // `Partitioning` has no `PartialEq`, so compare the partition count and
+ // the explicit `Display` form. Derived `Debug` would change with any
+ // field addition, making this brittle for no gain.
+ let actual_partitioning = actual.properties().output_partitioning();
+ let expected_partitioning = expected.properties().output_partitioning();
+ assert_eq!(
+ actual_partitioning.partition_count(),
+ expected_partitioning.partition_count(),
+ "partition count"
+ );
+ assert_eq!(
+ actual_partitioning.to_string(),
+ expected_partitioning.to_string(),
+ "partitioning"
+ );
+ }
+
+ #[test]
+ fn test_sort_below_changes_orderings_but_not_the_equivalence_group() -> Result<()> {
+ // The premise the fast path rests on. If a sort ever starts altering
+ // the equivalence group, reusing the cached group becomes unsound and
+ // this test is the one that should fail first.
+ let child = filtered_source("a", "b")?;
+ let ordering = LexOrdering::new(vec![PhysicalSortExpr::new_default(col(
+ "c",
+ &child.schema(),
+ )?)])
+ .expect("non-empty ordering");
+ let sorted = SortExec::new(ordering, Arc::clone(&child));
+
+ let child_props = child.properties().equivalence_properties();
+ let sorted_props = sorted.properties().equivalence_properties();
+
+ assert!(
+ !child_props.eq_group().is_empty(),
+ "the filter did not produce an equivalence class"
+ );
+ assert!(
+ child_props
+ .eq_group()
+ .has_same_classes(sorted_props.eq_group()),
+ "sorting altered the equivalence group"
+ );
+ assert_ne!(
+ child_props.oeq_class(),
+ sorted_props.oeq_class(),
+ "sorting did not alter the orderings"
+ );
+
+ Ok(())
+ }
+
+ #[test]
+ fn test_replace_children_reuses_eq_group_when_only_orderings_change() -> Result<()> {
Review Comment:
I don't think this are hitting the fast path. I.e. if we deleted the fast path this test would still pass. Could we add a `cfg(test)` counter or something that *asserts* that we are actually hitting the fast path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org