You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/11/13 19:49:52 UTC
[PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
alamb opened a new pull request, #18678:
URL: https://github.com/apache/datafusion/pull/18678
## Which issue does this PR close?
- Related to https://github.com/apache/datafusion/issues/17261
## Rationale for this change
Each time a LogicalPlan is rewritten to eliminate a Union, we traverse the entire plan tree and copy some non trivial parts of it
Thus it is faster to plan when we have fewer passes over the plan tree
the EliminateNestedUnion and EliminateOneUnion rules both do similar things, and
the EliminateNestedUnion rule is very simple. So let's combine them into a
single rule that does both things in one pass over the plan tree.
## What changes are included in this PR?
Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules into a single pass
## Are these changes tested?
Yes with existing tests
I will also run planning benchmarks
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3534430922
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group alamb_consolidate_passes main
----- ------------------------ ----
logical_aggregate_with_join 1.02 633.8±2.89µs ? ?/sec 1.00 620.3±2.48µs ? ?/sec
logical_select_all_from_1000 1.05 11.3±0.04ms ? ?/sec 1.00 10.7±0.04ms ? ?/sec
logical_select_one_from_700 1.03 422.3±3.13µs ? ?/sec 1.00 409.6±2.06µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.03 378.3±2.03µs ? ?/sec 1.00 366.2±1.59µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.03 363.3±2.52µs ? ?/sec 1.00 352.6±0.84µs ? ?/sec
physical_intersection 1.01 833.4±6.92µs ? ?/sec 1.00 821.9±4.76µs ? ?/sec
physical_join_consider_sort 1.02 1391.4±4.32µs ? ?/sec 1.00 1360.4±17.11µs ? ?/sec
physical_join_distinct 1.03 353.9±2.35µs ? ?/sec 1.00 344.7±3.77µs ? ?/sec
physical_many_self_joins 1.05 9.7±0.04ms ? ?/sec 1.00 9.2±0.04ms ? ?/sec
physical_plan_clickbench_all 1.00 181.3±1.80ms ? ?/sec 1.00 180.7±1.80ms ? ?/sec
physical_plan_clickbench_q1 1.01 2.4±0.02ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.01 3.2±0.04ms ? ?/sec 1.00 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q11 1.01 3.5±0.03ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.6±0.03ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.01 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.01 3.5±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.01 3.3±0.04ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q17 1.02 3.3±0.04ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q18 1.02 2.8±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.01 3.7±0.04ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.8±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q20 1.03 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.01 2.8±0.02ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q22 1.01 3.4±0.04ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q23 1.01 3.7±0.03ms ? ?/sec 1.00 3.7±0.06ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.2±0.03ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.8±0.02ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.1±0.04ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q3 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.7±0.14ms ? ?/sec 1.00 12.7±0.15ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q32 1.01 3.7±0.05ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.01 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.01 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.7±0.02ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
physical_plan_clickbench_q37 1.01 3.9±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.9±0.04ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q39 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q4 1.01 2.5±0.05ms ? ?/sec 1.00 2.4±0.04ms ? ?/sec
physical_plan_clickbench_q40 1.01 4.5±0.04ms ? ?/sec 1.00 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.9±0.05ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.9±0.04ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.3±0.04ms ? ?/sec 1.00 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.6±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.6±0.02ms ? ?/sec 1.01 2.6±0.21ms ? ?/sec
physical_plan_clickbench_q46 1.01 3.0±0.08ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.6±0.04ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.3±0.04ms ? ?/sec 1.00 4.3±0.05ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.6±0.05ms ? ?/sec 1.00 4.6±0.08ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.7±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.2±0.05ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.02 2.7±0.02ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q7 1.01 2.4±0.02ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.02 3.3±0.06ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.1±0.03ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpcds_all 1.00 1028.1±7.99ms ? ?/sec 1.00 1033.1±3.96ms ? ?/sec
physical_plan_tpch_all 1.00 63.5±0.38ms ? ?/sec 1.01 64.2±0.44ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.9±0.02ms ? ?/sec 1.01 4.0±0.02ms ? ?/sec
physical_plan_tpch_q11 1.00 3.5±0.01ms ? ?/sec 1.02 3.6±0.02ms ? ?/sec
physical_plan_tpch_q12 1.00 1784.6±8.23µs ? ?/sec 1.02 1811.8±8.49µs ? ?/sec
physical_plan_tpch_q13 1.00 1443.4±15.30µs ? ?/sec 1.01 1463.8±7.75µs ? ?/sec
physical_plan_tpch_q14 1.00 1953.7±11.66µs ? ?/sec 1.01 1964.8±16.95µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
physical_plan_tpch_q17 1.00 2.6±0.02ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.6±0.01ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.00 3.2±0.01ms ? ?/sec 1.01 3.2±0.01ms ? ?/sec
physical_plan_tpch_q2 1.00 5.9±0.04ms ? ?/sec 1.00 5.9±0.02ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.02ms ? ?/sec 1.02 3.2±0.02ms ? ?/sec
physical_plan_tpch_q21 1.00 4.2±0.01ms ? ?/sec 1.01 4.2±0.01ms ? ?/sec
physical_plan_tpch_q22 1.00 2.8±0.02ms ? ?/sec 1.01 2.8±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.6±0.01ms ? ?/sec 1.01 2.7±0.05ms ? ?/sec
physical_plan_tpch_q4 1.00 1483.1±12.99µs ? ?/sec 1.01 1494.2±8.60µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.01ms ? ?/sec 1.01 3.2±0.01ms ? ?/sec
physical_plan_tpch_q6 1.00 854.1±4.46µs ? ?/sec 1.00 857.6±7.64µs ? ?/sec
physical_plan_tpch_q7 1.00 4.1±0.01ms ? ?/sec 1.01 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.4±0.02ms ? ?/sec 1.01 5.5±0.03ms ? ?/sec
physical_plan_tpch_q9 1.00 4.0±0.02ms ? ?/sec 1.01 4.0±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.00 16.8±0.08ms ? ?/sec 1.00 16.8±0.08ms ? ?/sec
physical_select_all_from_1000 1.03 24.2±0.07ms ? ?/sec 1.00 23.4±0.31ms ? ?/sec
physical_select_one_from_700 1.05 1092.4±7.24µs ? ?/sec 1.00 1042.7±6.10µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 6.0±0.04ms ? ?/sec 1.00 6.0±0.03ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 12.9±0.08ms ? ?/sec 1.01 13.0±0.09ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 162.4±1.03ms ? ?/sec 1.00 163.0±1.11ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.00 377.8±2.37ms ? ?/sec 1.00 379.5±3.37ms ? ?/sec
physical_theta_join_consider_sort 1.02 1753.3±9.37µs ? ?/sec 1.00 1725.3±8.68µs ? ?/sec
physical_unnest_to_join 1.01 1846.0±7.53µs ? ?/sec 1.00 1819.5±6.46µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1081.3±7.00µs ? ?/sec 1.01 1089.4±4.19µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 2.3±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 649.6±5.00µs ? ?/sec 1.01 655.6±5.10µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 802.2±4.60µs ? ?/sec 1.01 814.2±5.82µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 863.3±4.75µs ? ?/sec 1.01 873.3±6.52µs ? ?/sec
with_param_values_many_columns 1.01 655.4±3.63µs ? ?/sec 1.00 647.5±4.48µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3529745666
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/consolidate_passes (70b34d95877d836aba07c5cc482d3d5bf74f27fe) to 377c0fce481d561d58e3a6fad2dca18cb1d58384 [diff](https://github.com/apache/datafusion/compare/377c0fce481d561d58e3a6fad2dca18cb1d58384..70b34d95877d836aba07c5cc482d3d5bf74f27fe)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_consolidate_passes
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2527740008
##########
datafusion/optimizer/src/optimizer.rs:
##########
@@ -242,7 +241,6 @@ impl Optimizer {
Arc::new(EliminateLimit::new()),
Arc::new(PropagateEmptyRelation::new()),
// Must be after PropagateEmptyRelation
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2529460947
##########
datafusion/optimizer/src/optimize_unions.rs:
##########
@@ -26,19 +26,25 @@ use itertools::Itertools;
use std::sync::Arc;
#[derive(Default, Debug)]
-/// An optimization rule that replaces nested unions with a single union.
-pub struct EliminateNestedUnion;
+/// An optimization rule that
+/// 1. replaces nested unions with a single union.
+/// 2. removes unions with a single input.
+pub struct OptimizeUnions;
-impl EliminateNestedUnion {
+// Backwards compatibility name
+#[deprecated(since = "52.0.0", note = "Please use OptimizeUnions instead")]
+pub type EliminateNestedUnion = OptimizeUnions;
Review Comment:
Do we need to consider the mod too? Since it'll now be `optimize_unions::EliminateNestedUnion` which is breaking anyway 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3538889537
I got ambitious (it happens once every few moons 🤣) and ran the benchmark locally just to see if I saw the same uptick in the first few benchmarks as @alamb's results seem to have. Looks like my machine has about +/- 3-5% overall which is good to know.
```
group alamb_consolidate_passes main
----- ------------------------ ----
logical_aggregate_with_join 1.02 273.6±27.76µs ? ?/sec 1.00 269.5±5.71µs ? ?/sec
logical_select_all_from_1000 1.00 6.0±0.14ms ? ?/sec 1.01 6.1±0.18ms ? ?/sec
logical_select_one_from_700 1.00 200.0±9.57µs ? ?/sec 1.00 200.0±14.22µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 163.7±1.69µs ? ?/sec 1.00 163.8±1.96µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 156.9±1.69µs ? ?/sec 1.00 157.0±6.68µs ? ?/sec
physical_intersection 1.01 345.9±5.18µs ? ?/sec 1.00 343.7±9.57µs ? ?/sec
physical_join_consider_sort 1.00 619.4±44.70µs ? ?/sec 1.02 628.7±17.02µs ? ?/sec
physical_join_distinct 1.00 152.6±3.26µs ? ?/sec 1.00 152.1±1.75µs ? ?/sec
physical_many_self_joins 1.00 4.6±0.47ms ? ?/sec 1.02 4.7±0.23ms ? ?/sec
physical_plan_clickbench_all 1.00 111.4±7.00ms ? ?/sec 1.01 112.6±5.26ms ? ?/sec
physical_plan_clickbench_q1 1.00 1453.7±89.45µs ? ?/sec 1.03 1494.5±95.53µs ? ?/sec
physical_plan_clickbench_q10 1.00 1944.8±119.64µs ? ?/sec 1.01 1973.4±118.93µs ? ?/sec
physical_plan_clickbench_q11 1.07 2.2±0.19ms ? ?/sec 1.00 2.0±0.09ms ? ?/sec
physical_plan_clickbench_q12 1.04 2.3±0.14ms ? ?/sec 1.00 2.2±0.10ms ? ?/sec
physical_plan_clickbench_q13 1.02 1975.3±96.53µs ? ?/sec 1.00 1927.2±116.17µs ? ?/sec
physical_plan_clickbench_q14 1.01 2.1±0.18ms ? ?/sec 1.00 2.1±0.07ms ? ?/sec
physical_plan_clickbench_q15 1.03 2.0±0.11ms ? ?/sec 1.00 1948.4±111.61µs ? ?/sec
physical_plan_clickbench_q16 1.03 1912.4±115.83µs ? ?/sec 1.00 1850.9±107.62µs ? ?/sec
physical_plan_clickbench_q17 1.00 1955.4±134.96µs ? ?/sec 1.03 2.0±0.24ms ? ?/sec
physical_plan_clickbench_q18 1.01 1659.7±191.65µs ? ?/sec 1.00 1640.4±111.59µs ? ?/sec
physical_plan_clickbench_q19 1.00 2.1±0.10ms ? ?/sec 1.03 2.2±0.11ms ? ?/sec
physical_plan_clickbench_q2 1.03 1726.2±188.88µs ? ?/sec 1.00 1679.0±117.41µs ? ?/sec
physical_plan_clickbench_q20 1.00 1467.6±57.85µs ? ?/sec 1.05 1536.2±109.10µs ? ?/sec
physical_plan_clickbench_q21 1.00 1696.9±107.09µs ? ?/sec 1.03 1744.0±123.88µs ? ?/sec
physical_plan_clickbench_q22 1.01 2.1±0.11ms ? ?/sec 1.00 2.1±0.09ms ? ?/sec
physical_plan_clickbench_q23 1.01 2.3±0.30ms ? ?/sec 1.00 2.2±0.12ms ? ?/sec
physical_plan_clickbench_q24 1.03 2.5±0.14ms ? ?/sec 1.00 2.4±0.12ms ? ?/sec
physical_plan_clickbench_q25 1.00 1771.7±89.43µs ? ?/sec 1.02 1811.0±109.44µs ? ?/sec
physical_plan_clickbench_q26 1.00 1677.6±158.90µs ? ?/sec 1.01 1700.7±170.85µs ? ?/sec
physical_plan_clickbench_q27 1.00 1845.1±101.08µs ? ?/sec 1.02 1888.0±120.95µs ? ?/sec
physical_plan_clickbench_q28 1.01 2.2±0.14ms ? ?/sec 1.00 2.2±0.08ms ? ?/sec
physical_plan_clickbench_q29 1.03 2.4±0.12ms ? ?/sec 1.00 2.3±0.05ms ? ?/sec
physical_plan_clickbench_q3 1.03 1635.5±90.18µs ? ?/sec 1.00 1589.2±98.94µs ? ?/sec
physical_plan_clickbench_q30 1.00 6.9±0.17ms ? ?/sec 1.00 7.0±0.14ms ? ?/sec
physical_plan_clickbench_q31 1.01 2.3±0.15ms ? ?/sec 1.00 2.2±0.08ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.2±0.11ms ? ?/sec 1.00 2.2±0.12ms ? ?/sec
physical_plan_clickbench_q33 1.00 1877.0±52.88µs ? ?/sec 1.05 1967.7±145.66µs ? ?/sec
physical_plan_clickbench_q34 1.00 1770.3±88.15µs ? ?/sec 1.00 1771.0±112.51µs ? ?/sec
physical_plan_clickbench_q35 1.04 1930.0±120.80µs ? ?/sec 1.00 1852.2±108.99µs ? ?/sec
physical_plan_clickbench_q36 1.00 2.1±0.07ms ? ?/sec 1.09 2.3±0.19ms ? ?/sec
physical_plan_clickbench_q37 1.00 2.2±0.12ms ? ?/sec 1.09 2.4±0.20ms ? ?/sec
physical_plan_clickbench_q38 1.01 2.3±0.10ms ? ?/sec 1.00 2.3±0.14ms ? ?/sec
physical_plan_clickbench_q39 1.00 2.2±0.14ms ? ?/sec 1.01 2.2±0.13ms ? ?/sec
physical_plan_clickbench_q4 1.00 1471.1±101.71µs ? ?/sec 1.03 1509.3±181.76µs ? ?/sec
physical_plan_clickbench_q40 1.00 2.7±0.12ms ? ?/sec 1.01 2.7±0.23ms ? ?/sec
physical_plan_clickbench_q41 1.02 2.3±0.10ms ? ?/sec 1.00 2.3±0.04ms ? ?/sec
physical_plan_clickbench_q42 1.04 2.3±0.11ms ? ?/sec 1.00 2.2±0.06ms ? ?/sec
physical_plan_clickbench_q43 1.00 2.6±0.10ms ? ?/sec 1.03 2.7±0.15ms ? ?/sec
physical_plan_clickbench_q44 1.00 1545.6±94.86µs ? ?/sec 1.02 1573.1±78.07µs ? ?/sec
physical_plan_clickbench_q45 1.00 1490.3±89.23µs ? ?/sec 1.05 1568.7±85.20µs ? ?/sec
physical_plan_clickbench_q46 1.00 1847.2±161.94µs ? ?/sec 1.05 1933.4±131.56µs ? ?/sec
physical_plan_clickbench_q47 1.00 2.1±0.16ms ? ?/sec 1.00 2.1±0.10ms ? ?/sec
physical_plan_clickbench_q48 1.06 2.7±0.24ms ? ?/sec 1.00 2.5±0.06ms ? ?/sec
physical_plan_clickbench_q49 1.00 2.7±0.12ms ? ?/sec 1.01 2.7±0.13ms ? ?/sec
physical_plan_clickbench_q5 1.03 1610.2±102.46µs ? ?/sec 1.00 1570.2±68.28µs ? ?/sec
physical_plan_clickbench_q50 1.02 2.5±0.10ms ? ?/sec 1.00 2.4±0.14ms ? ?/sec
physical_plan_clickbench_q51 1.00 1929.7±121.60µs ? ?/sec 1.05 2.0±0.16ms ? ?/sec
physical_plan_clickbench_q6 1.00 1637.2±105.52µs ? ?/sec 1.00 1633.3±88.27µs ? ?/sec
physical_plan_clickbench_q7 1.06 1444.5±80.80µs ? ?/sec 1.00 1362.1±83.40µs ? ?/sec
physical_plan_clickbench_q8 1.00 1934.7±108.92µs ? ?/sec 1.00 1939.7±74.83µs ? ?/sec
physical_plan_clickbench_q9 1.04 1943.4±123.35µs ? ?/sec 1.00 1876.7±116.01µs ? ?/sec
physical_plan_tpcds_all 1.00 528.5±9.70ms ? ?/sec 1.00 526.1±8.43ms ? ?/sec
physical_plan_tpch_all 1.00 34.0±1.74ms ? ?/sec 1.00 34.1±2.50ms ? ?/sec
physical_plan_tpch_q1 1.00 1028.0±37.98µs ? ?/sec 1.00 1026.5±12.61µs ? ?/sec
physical_plan_tpch_q10 1.05 2.1±0.19ms ? ?/sec 1.00 2.0±0.02ms ? ?/sec
physical_plan_tpch_q11 1.01 1808.0±93.49µs ? ?/sec 1.00 1788.2±68.54µs ? ?/sec
physical_plan_tpch_q12 1.00 926.1±12.29µs ? ?/sec 1.01 931.7±39.60µs ? ?/sec
physical_plan_tpch_q13 1.05 774.8±136.70µs ? ?/sec 1.00 734.7±13.43µs ? ?/sec
physical_plan_tpch_q14 1.01 1000.6±48.25µs ? ?/sec 1.00 986.8±13.31µs ? ?/sec
physical_plan_tpch_q16 1.01 1291.5±40.21µs ? ?/sec 1.00 1276.8±20.84µs ? ?/sec
physical_plan_tpch_q17 1.03 1342.2±139.29µs ? ?/sec 1.00 1300.6±16.13µs ? ?/sec
physical_plan_tpch_q18 1.02 1388.5±218.01µs ? ?/sec 1.00 1358.9±16.44µs ? ?/sec
physical_plan_tpch_q19 1.00 1653.9±24.40µs ? ?/sec 1.00 1658.7±83.47µs ? ?/sec
physical_plan_tpch_q2 1.02 3.2±0.15ms ? ?/sec 1.00 3.1±0.06ms ? ?/sec
physical_plan_tpch_q20 1.02 1645.3±90.76µs ? ?/sec 1.00 1620.7±13.85µs ? ?/sec
physical_plan_tpch_q21 1.00 2.2±0.05ms ? ?/sec 1.00 2.2±0.10ms ? ?/sec
physical_plan_tpch_q22 1.01 1457.4±20.57µs ? ?/sec 1.00 1445.7±32.89µs ? ?/sec
physical_plan_tpch_q3 1.06 1437.5±303.11µs ? ?/sec 1.00 1353.6±31.64µs ? ?/sec
physical_plan_tpch_q4 1.00 766.8±35.64µs ? ?/sec 1.00 766.4±30.13µs ? ?/sec
physical_plan_tpch_q5 1.01 1655.5±27.03µs ? ?/sec 1.00 1640.6±27.60µs ? ?/sec
physical_plan_tpch_q6 1.01 401.5±33.15µs ? ?/sec 1.00 396.1±17.64µs ? ?/sec
physical_plan_tpch_q7 1.00 2.2±0.04ms ? ?/sec 1.01 2.2±0.10ms ? ?/sec
physical_plan_tpch_q8 1.00 2.8±0.07ms ? ?/sec 1.01 2.9±0.32ms ? ?/sec
physical_plan_tpch_q9 1.01 2.1±0.12ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.00 8.7±0.66ms ? ?/sec 1.03 8.9±0.61ms ? ?/sec
physical_select_all_from_1000 1.00 13.1±0.83ms ? ?/sec 1.03 13.6±0.23ms ? ?/sec
physical_select_one_from_700 1.00 491.0±3.82µs ? ?/sec 1.00 491.1±17.92µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.01 3.2±0.11ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.01 6.9±0.52ms ? ?/sec 1.00 6.8±0.23ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 90.0±3.81ms ? ?/sec 1.00 90.1±2.20ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.01 190.9±6.67ms ? ?/sec 1.00 190.0±5.93ms ? ?/sec
physical_theta_join_consider_sort 1.00 803.2±32.78µs ? ?/sec 1.01 809.2±24.75µs ? ?/sec
physical_unnest_to_join 1.00 865.7±27.61µs ? ?/sec 1.05 907.1±35.43µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 523.6±8.13µs ? ?/sec 1.03 537.3±15.27µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 1181.5±28.42µs ? ?/sec 1.00 1185.2±18.33µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 289.2±4.99µs ? ?/sec 1.00 288.9±13.27µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 371.4±6.94µs ? ?/sec 1.03 384.2±16.19µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 406.8±7.05µs ? ?/sec 1.01 410.2±6.51µs ? ?/sec
with_param_values_many_columns 1.01 358.5±9.43µs ? ?/sec 1.00 354.1±5.07µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3530148208
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group alamb_consolidate_passes main
----- ------------------------ ----
logical_aggregate_with_join 1.03 635.6±16.39µs ? ?/sec 1.00 618.0±4.30µs ? ?/sec
logical_select_all_from_1000 1.02 11.3±0.03ms ? ?/sec 1.00 11.1±0.13ms ? ?/sec
logical_select_one_from_700 1.02 421.1±2.02µs ? ?/sec 1.00 410.8±2.52µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.02 377.0±6.47µs ? ?/sec 1.00 368.7±1.68µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.03 364.3±7.19µs ? ?/sec 1.00 354.2±3.03µs ? ?/sec
physical_intersection 1.02 838.6±3.82µs ? ?/sec 1.00 818.7±3.83µs ? ?/sec
physical_join_consider_sort 1.03 1389.4±17.64µs ? ?/sec 1.00 1353.4±4.70µs ? ?/sec
physical_join_distinct 1.03 354.8±2.21µs ? ?/sec 1.00 343.8±1.10µs ? ?/sec
physical_many_self_joins 1.05 9.6±0.03ms ? ?/sec 1.00 9.2±0.04ms ? ?/sec
physical_plan_clickbench_all 1.00 179.5±1.03ms ? ?/sec 1.00 179.7±1.16ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.3±0.02ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.5±0.02ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.2±0.03ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.4±0.03ms ? ?/sec 1.01 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.3±0.03ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.1±0.02ms ? ?/sec 1.01 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.2±0.03ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.7±0.02ms ? ?/sec 1.01 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.6±0.03ms ? ?/sec 1.02 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.5±0.02ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.8±0.02ms ? ?/sec 1.01 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.6±0.03ms ? ?/sec 1.01 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.1±0.03ms ? ?/sec 1.00 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q25 1.00 2.9±0.03ms ? ?/sec 1.01 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.0±0.02ms ? ?/sec 1.01 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.7±0.04ms ? ?/sec 1.01 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.0±0.04ms ? ?/sec 1.00 4.0±0.03ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.5±0.19ms ? ?/sec 1.00 12.5±0.10ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.7±0.05ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.2±0.02ms ? ?/sec 1.01 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.00 2.9±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.0±0.03ms ? ?/sec 1.01 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.7±0.08ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.8±0.04ms ? ?/sec 1.01 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.8±0.02ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.4±0.02ms ? ?/sec 1.01 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.5±0.03ms ? ?/sec 1.00 4.5±0.03ms ? ?/sec
physical_plan_clickbench_q41 1.01 3.9±0.12ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.8±0.03ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.3±0.04ms ? ?/sec 1.00 4.3±0.04ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.0±0.02ms ? ?/sec 1.01 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.5±0.03ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.3±0.04ms ? ?/sec 1.01 4.3±0.04ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.6±0.05ms ? ?/sec 1.01 4.6±0.04ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.6±0.02ms ? ?/sec 1.01 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.1±0.03ms ? ?/sec 1.01 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.4±0.02ms ? ?/sec 1.01 2.4±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpcds_all 1.00 1020.3±3.07ms ? ?/sec 1.01 1025.6±3.21ms ? ?/sec
physical_plan_tpch_all 1.00 63.4±0.38ms ? ?/sec 1.01 63.9±0.17ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.9±0.02ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
physical_plan_tpch_q11 1.00 3.5±0.01ms ? ?/sec 1.01 3.5±0.02ms ? ?/sec
physical_plan_tpch_q12 1.00 1782.8±7.26µs ? ?/sec 1.01 1797.2±12.68µs ? ?/sec
physical_plan_tpch_q13 1.00 1440.6±6.00µs ? ?/sec 1.01 1448.7±7.61µs ? ?/sec
physical_plan_tpch_q14 1.00 1948.7±8.25µs ? ?/sec 1.00 1954.3±9.71µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
physical_plan_tpch_q17 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q19 1.00 3.2±0.01ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_tpch_q2 1.00 5.8±0.02ms ? ?/sec 1.01 5.9±0.02ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.01ms ? ?/sec 1.01 3.2±0.01ms ? ?/sec
physical_plan_tpch_q21 1.00 4.2±0.02ms ? ?/sec 1.01 4.2±0.01ms ? ?/sec
physical_plan_tpch_q22 1.00 2.8±0.02ms ? ?/sec 1.01 2.8±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.6±0.01ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1476.3±7.51µs ? ?/sec 1.01 1493.7±12.77µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q6 1.00 859.0±7.18µs ? ?/sec 1.00 857.3±5.13µs ? ?/sec
physical_plan_tpch_q7 1.00 4.1±0.02ms ? ?/sec 1.01 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.4±0.03ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 4.0±0.02ms ? ?/sec 1.01 4.0±0.10ms ? ?/sec
physical_select_aggregates_from_200 1.00 16.8±0.08ms ? ?/sec 1.00 16.8±0.08ms ? ?/sec
physical_select_all_from_1000 1.04 24.2±0.11ms ? ?/sec 1.00 23.4±0.09ms ? ?/sec
physical_select_one_from_700 1.05 1091.4±6.93µs ? ?/sec 1.00 1038.6±3.77µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 5.9±0.02ms ? ?/sec 1.01 6.0±0.02ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 12.8±0.05ms ? ?/sec 1.01 12.9±0.07ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 161.0±0.91ms ? ?/sec 1.01 161.8±0.89ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.00 372.3±2.12ms ? ?/sec 1.01 374.2±2.63ms ? ?/sec
physical_theta_join_consider_sort 1.01 1750.1±8.17µs ? ?/sec 1.00 1727.1±29.38µs ? ?/sec
physical_unnest_to_join 1.01 1846.3±9.28µs ? ?/sec 1.00 1820.5±7.24µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1082.9±7.05µs ? ?/sec 1.01 1090.9±5.89µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 655.6±5.10µs ? ?/sec 1.00 657.6±4.14µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 804.9±5.80µs ? ?/sec 1.01 812.3±7.39µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 869.2±7.98µs ? ?/sec 1.01 873.7±5.19µs ? ?/sec
with_param_values_many_columns 1.01 655.8±4.35µs ? ?/sec 1.00 650.7±6.00µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3529951917
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/consolidate_passes (70b34d95877d836aba07c5cc482d3d5bf74f27fe) to 377c0fce481d561d58e3a6fad2dca18cb1d58384 [diff](https://github.com/apache/datafusion/compare/377c0fce481d561d58e3a6fad2dca18cb1d58384..70b34d95877d836aba07c5cc482d3d5bf74f27fe)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_consolidate_passes
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2524812652
##########
datafusion/optimizer/src/eliminate_nested_union.rs:
##########
@@ -420,4 +423,28 @@ mod tests {
TableScan: table_1
")
}
+
+ #[test]
+ fn eliminate_one_union() -> Result<()> {
Review Comment:
single input unions can't be created easily, so this test looks a little different
##########
datafusion/optimizer/src/eliminate_one_union.rs:
##########
@@ -1,121 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! [`EliminateOneUnion`] eliminates single element `Union`
-
-use crate::{OptimizerConfig, OptimizerRule};
-use datafusion_common::{tree_node::Transformed, Result};
-use datafusion_expr::logical_plan::{LogicalPlan, Union};
-use std::sync::Arc;
-
-use crate::optimizer::ApplyOrder;
-
-#[derive(Default, Debug)]
-/// An optimization rule that eliminates union with one element.
-pub struct EliminateOneUnion;
-
-impl EliminateOneUnion {
- #[allow(missing_docs)]
- pub fn new() -> Self {
- Self {}
- }
-}
-
-impl OptimizerRule for EliminateOneUnion {
- fn name(&self) -> &str {
- "eliminate_one_union"
- }
-
- fn supports_rewrite(&self) -> bool {
- true
- }
-
- fn rewrite(
- &self,
- plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
- ) -> Result<Transformed<LogicalPlan>> {
- match plan {
- LogicalPlan::Union(Union { mut inputs, .. }) if inputs.len() == 1 => Ok(
- Transformed::yes(Arc::unwrap_or_clone(inputs.pop().unwrap())),
- ),
- _ => Ok(Transformed::no(plan)),
- }
- }
-
- fn apply_order(&self) -> Option<ApplyOrder> {
- Some(ApplyOrder::TopDown)
- }
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::test::*;
- use arrow::datatypes::{DataType, Field, Schema};
- use datafusion_common::ToDFSchema;
- use datafusion_expr::{
- expr_rewriter::coerce_plan_expr_for_schema, logical_plan::table_scan,
- };
- use std::sync::Arc;
-
- fn schema() -> Schema {
- Schema::new(vec![
- Field::new("id", DataType::Int32, false),
- Field::new("key", DataType::Utf8, false),
- Field::new("value", DataType::Int32, false),
- ])
- }
-
- fn assert_optimized_plan_equal(plan: LogicalPlan, expected: &str) -> Result<()> {
- assert_optimized_plan_with_rules(
- vec![Arc::new(EliminateOneUnion::new())],
- plan,
- expected,
- true,
- )
- }
-
- #[test]
- fn eliminate_nothing() -> Result<()> {
- let plan_builder = table_scan(Some("table"), &schema(), None)?;
-
- let plan = plan_builder.clone().union(plan_builder.build()?)?.build()?;
-
- let expected = "\
- Union\
- \n TableScan: table\
- \n TableScan: table";
- assert_optimized_plan_equal(plan, expected)
- }
-
- #[test]
- fn eliminate_one_union() -> Result<()> {
Review Comment:
I ported this test over
##########
datafusion/optimizer/src/eliminate_one_union.rs:
##########
@@ -1,121 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! [`EliminateOneUnion`] eliminates single element `Union`
-
-use crate::{OptimizerConfig, OptimizerRule};
-use datafusion_common::{tree_node::Transformed, Result};
-use datafusion_expr::logical_plan::{LogicalPlan, Union};
-use std::sync::Arc;
-
-use crate::optimizer::ApplyOrder;
-
-#[derive(Default, Debug)]
-/// An optimization rule that eliminates union with one element.
-pub struct EliminateOneUnion;
-
-impl EliminateOneUnion {
- #[allow(missing_docs)]
- pub fn new() -> Self {
- Self {}
- }
-}
-
-impl OptimizerRule for EliminateOneUnion {
- fn name(&self) -> &str {
- "eliminate_one_union"
- }
-
- fn supports_rewrite(&self) -> bool {
- true
- }
-
- fn rewrite(
- &self,
- plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
- ) -> Result<Transformed<LogicalPlan>> {
- match plan {
- LogicalPlan::Union(Union { mut inputs, .. }) if inputs.len() == 1 => Ok(
- Transformed::yes(Arc::unwrap_or_clone(inputs.pop().unwrap())),
- ),
- _ => Ok(Transformed::no(plan)),
- }
- }
-
- fn apply_order(&self) -> Option<ApplyOrder> {
- Some(ApplyOrder::TopDown)
- }
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::test::*;
- use arrow::datatypes::{DataType, Field, Schema};
- use datafusion_common::ToDFSchema;
- use datafusion_expr::{
- expr_rewriter::coerce_plan_expr_for_schema, logical_plan::table_scan,
- };
- use std::sync::Arc;
-
- fn schema() -> Schema {
- Schema::new(vec![
- Field::new("id", DataType::Int32, false),
- Field::new("key", DataType::Utf8, false),
- Field::new("value", DataType::Int32, false),
- ])
- }
-
- fn assert_optimized_plan_equal(plan: LogicalPlan, expected: &str) -> Result<()> {
- assert_optimized_plan_with_rules(
- vec![Arc::new(EliminateOneUnion::new())],
- plan,
- expected,
- true,
- )
- }
-
- #[test]
- fn eliminate_nothing() -> Result<()> {
- let plan_builder = table_scan(Some("table"), &schema(), None)?;
Review Comment:
there is already the same test in `datafusion/optimizer/src/eliminate_nested_union.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3533951304
> I find it odd that the perf slightly decreased in some benchmarks. I'll try and reproduce it today locally.
i think looking at the two runs (of the planning benchmark) I would say that there is no noticable performance difference (kind of as expected). I will rerun to see if we can see a repeating pattern emerge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2525629884
##########
datafusion/optimizer/src/eliminate_nested_union.rs:
##########
@@ -55,6 +55,9 @@ impl OptimizerRule for EliminateNestedUnion {
_config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
match plan {
+ LogicalPlan::Union(Union { mut inputs, .. }) if inputs.len() == 1 => Ok(
Review Comment:
Worth adding a minor comment here, considering its not exactly related to nested unions?
(Or could rename away from `EliminateNestedUnions` to something like `SimplifyUnions` to be all encompassing)
##########
datafusion/optimizer/src/optimizer.rs:
##########
@@ -242,7 +241,6 @@ impl Optimizer {
Arc::new(EliminateLimit::new()),
Arc::new(PropagateEmptyRelation::new()),
// Must be after PropagateEmptyRelation
Review Comment:
It seems this comment was added with `EliminateOneUnion` so need to remove it, reference: https://github.com/apache/datafusion/commit/704e034abbc57645ae594288c49c550350540022#diff-42d66905c3fa6b245c3493fb4df4816e0fde3036941315ab42198fb16f3907df
(Any impact that the logic is now before `PropagateEmptyRelation`? Or running multiple passes of optimizer makes order redundant anyway?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3534558782
I'm going to go with the benchmarks being within a range of noise. The first 10 or so do seem to be slightly higher on average though that could just be because they were the first to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3533631745
I find it odd that the perf slightly decreased in some benchmarks. I'll try and reproduce it today locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3534218597
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/consolidate_passes (70b34d95877d836aba07c5cc482d3d5bf74f27fe) to 377c0fce481d561d58e3a6fad2dca18cb1d58384 [diff](https://github.com/apache/datafusion/compare/377c0fce481d561d58e3a6fad2dca18cb1d58384..70b34d95877d836aba07c5cc482d3d5bf74f27fe)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_consolidate_passes
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2528722132
##########
datafusion/optimizer/src/optimize_unions.rs:
##########
@@ -55,6 +55,9 @@ impl OptimizerRule for EliminateNestedUnion {
_config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
match plan {
+ LogicalPlan::Union(Union { mut inputs, .. }) if inputs.len() == 1 => Ok(
Review Comment:
> (Or could rename away from EliminateNestedUnions to something like SimplifyUnions to be all encompassing)
I like this idea -- done in 9041a171c12c2ed4ddac5a69a63cb8851ee27252
##########
datafusion/optimizer/src/optimizer.rs:
##########
@@ -242,7 +241,6 @@ impl Optimizer {
Arc::new(EliminateLimit::new()),
Arc::new(PropagateEmptyRelation::new()),
// Must be after PropagateEmptyRelation
Review Comment:
Since all tests pass with `OptimizeUnions` in its current location I think we can just remove this comment as stale. I have done so in 4ed07868f7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3533974411
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/consolidate_passes (70b34d95877d836aba07c5cc482d3d5bf74f27fe) to 377c0fce481d561d58e3a6fad2dca18cb1d58384 [diff](https://github.com/apache/datafusion/compare/377c0fce481d561d58e3a6fad2dca18cb1d58384..70b34d95877d836aba07c5cc482d3d5bf74f27fe)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_consolidate_passes
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3541961716
Thanks @Omega359 and @Jefffrey -- while this isn't perfect, it makes things simpler and seems to help planning so I'll merge it in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #18678:
URL: https://github.com/apache/datafusion/pull/18678
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3534218470
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group alamb_consolidate_passes main
----- ------------------------ ----
logical_aggregate_with_join 1.03 634.9±3.83µs ? ?/sec 1.00 616.5±3.46µs ? ?/sec
logical_select_all_from_1000 1.05 11.3±0.03ms ? ?/sec 1.00 10.8±0.05ms ? ?/sec
logical_select_one_from_700 1.04 425.6±4.80µs ? ?/sec 1.00 408.4±1.83µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.04 380.5±5.90µs ? ?/sec 1.00 365.7±1.65µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.04 365.6±2.06µs ? ?/sec 1.00 353.0±1.84µs ? ?/sec
physical_intersection 1.01 836.7±5.37µs ? ?/sec 1.00 824.8±5.22µs ? ?/sec
physical_join_consider_sort 1.03 1401.2±6.00µs ? ?/sec 1.00 1356.9±14.41µs ? ?/sec
physical_join_distinct 1.04 358.2±3.83µs ? ?/sec 1.00 343.5±1.16µs ? ?/sec
physical_many_self_joins 1.06 9.8±0.09ms ? ?/sec 1.00 9.2±0.04ms ? ?/sec
physical_plan_clickbench_all 1.00 181.2±1.82ms ? ?/sec 1.01 183.5±2.52ms ? ?/sec
physical_plan_clickbench_q1 1.02 2.4±0.02ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.2±0.04ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.01 3.4±0.03ms ? ?/sec 1.00 3.4±0.06ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.6±0.04ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.01 3.4±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.01 3.3±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q16 1.01 3.1±0.02ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q18 1.01 2.8±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.6±0.05ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.8±0.02ms ? ?/sec 1.00 2.8±0.04ms ? ?/sec
physical_plan_clickbench_q20 1.01 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.01 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.01 3.4±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q24 1.01 4.2±0.04ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.0±0.04ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.02 2.8±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q27 1.02 3.0±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.1±0.03ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q3 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.6±0.18ms ? ?/sec 1.00 12.7±0.13ms ? ?/sec
physical_plan_clickbench_q31 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q33 1.01 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.01 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.8±0.03ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.9±0.03ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q39 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.06ms ? ?/sec
physical_plan_clickbench_q4 1.01 2.4±0.02ms ? ?/sec 1.00 2.4±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.5±0.04ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.01 3.9±0.04ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.9±0.06ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.3±0.06ms ? ?/sec 1.00 4.3±0.05ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.6±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.02 2.6±0.03ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.01 3.0±0.03ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.6±0.04ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.3±0.05ms ? ?/sec 1.00 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.6±0.06ms ? ?/sec 1.00 4.6±0.04ms ? ?/sec
physical_plan_clickbench_q5 1.04 2.8±0.17ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q50 1.01 4.2±0.04ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q7 1.02 2.4±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_tpcds_all 1.00 1026.0±3.33ms ? ?/sec 1.00 1030.6±5.90ms ? ?/sec
physical_plan_tpch_all 1.00 63.8±0.35ms ? ?/sec 1.01 64.2±0.38ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.9±0.02ms ? ?/sec 1.01 4.0±0.04ms ? ?/sec
physical_plan_tpch_q11 1.00 3.5±0.02ms ? ?/sec 1.02 3.6±0.02ms ? ?/sec
physical_plan_tpch_q12 1.00 1785.8±9.92µs ? ?/sec 1.01 1804.7±8.39µs ? ?/sec
physical_plan_tpch_q13 1.00 1435.7±4.95µs ? ?/sec 1.01 1453.6±8.56µs ? ?/sec
physical_plan_tpch_q14 1.00 1960.9±8.24µs ? ?/sec 1.00 1962.5±11.18µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.03ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_tpch_q17 1.00 2.6±0.01ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.6±0.01ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.00 3.2±0.01ms ? ?/sec 1.01 3.2±0.01ms ? ?/sec
physical_plan_tpch_q2 1.00 5.8±0.03ms ? ?/sec 1.01 5.9±0.04ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.02ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_tpch_q21 1.00 4.2±0.02ms ? ?/sec 1.01 4.2±0.02ms ? ?/sec
physical_plan_tpch_q22 1.00 2.8±0.02ms ? ?/sec 1.01 2.8±0.02ms ? ?/sec
physical_plan_tpch_q3 1.00 2.6±0.01ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1479.2±8.25µs ? ?/sec 1.01 1487.9±8.44µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q6 1.00 853.4±5.77µs ? ?/sec 1.01 859.6±6.86µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.02ms ? ?/sec 1.01 4.2±0.03ms ? ?/sec
physical_plan_tpch_q8 1.00 5.4±0.04ms ? ?/sec 1.02 5.5±0.05ms ? ?/sec
physical_plan_tpch_q9 1.00 4.0±0.01ms ? ?/sec 1.01 4.0±0.03ms ? ?/sec
physical_select_aggregates_from_200 1.00 16.8±0.07ms ? ?/sec 1.00 16.8±0.09ms ? ?/sec
physical_select_all_from_1000 1.04 24.4±0.13ms ? ?/sec 1.00 23.5±0.12ms ? ?/sec
physical_select_one_from_700 1.04 1088.1±7.37µs ? ?/sec 1.00 1041.5±5.99µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 6.0±0.02ms ? ?/sec 1.01 6.0±0.02ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 12.9±0.06ms ? ?/sec 1.01 13.0±0.11ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.02 166.3±1.21ms ? ?/sec 1.00 163.9±1.17ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.01 382.9±2.59ms ? ?/sec 1.00 378.2±2.42ms ? ?/sec
physical_theta_join_consider_sort 1.02 1768.3±7.54µs ? ?/sec 1.00 1726.2±18.24µs ? ?/sec
physical_unnest_to_join 1.02 1861.1±8.86µs ? ?/sec 1.00 1822.5±14.28µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1086.2±6.91µs ? ?/sec 1.00 1084.5±6.39µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.01 2.3±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.01 654.4±11.60µs ? ?/sec 1.00 650.5±2.53µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 809.3±4.54µs ? ?/sec 1.00 807.8±4.59µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 866.9±4.97µs ? ?/sec 1.00 869.1±4.69µs ? ?/sec
with_param_values_many_columns 1.02 660.4±6.06µs ? ?/sec 1.00 648.9±7.85µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2529850335
##########
datafusion/optimizer/src/optimize_unions.rs:
##########
@@ -26,19 +26,25 @@ use itertools::Itertools;
use std::sync::Arc;
#[derive(Default, Debug)]
-/// An optimization rule that replaces nested unions with a single union.
-pub struct EliminateNestedUnion;
+/// An optimization rule that
+/// 1. replaces nested unions with a single union.
+/// 2. removes unions with a single input.
+pub struct OptimizeUnions;
-impl EliminateNestedUnion {
+// Backwards compatibility name
+#[deprecated(since = "52.0.0", note = "Please use OptimizeUnions instead")]
+pub type EliminateNestedUnion = OptimizeUnions;
Review Comment:
That is a good point -- I will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#issuecomment-3529951810
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group alamb_consolidate_passes main
----- ------------------------ ----
logical_aggregate_with_join 1.04 636.8±3.23µs ? ?/sec 1.00 614.8±2.03µs ? ?/sec
logical_select_all_from_1000 1.04 11.3±0.04ms ? ?/sec 1.00 10.8±0.22ms ? ?/sec
logical_select_one_from_700 1.04 423.1±6.11µs ? ?/sec 1.00 408.4±1.75µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.04 381.3±6.03µs ? ?/sec 1.00 365.9±1.40µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.04 366.0±1.93µs ? ?/sec 1.00 352.7±1.41µs ? ?/sec
physical_intersection 1.02 841.2±6.12µs ? ?/sec 1.00 821.2±4.48µs ? ?/sec
physical_join_consider_sort 1.03 1390.1±5.90µs ? ?/sec 1.00 1355.6±8.11µs ? ?/sec
physical_join_distinct 1.03 354.9±1.35µs ? ?/sec 1.00 344.5±2.17µs ? ?/sec
physical_many_self_joins 1.04 9.6±0.04ms ? ?/sec 1.00 9.2±0.03ms ? ?/sec
physical_plan_clickbench_all 1.01 180.9±4.09ms ? ?/sec 1.00 179.6±1.43ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.3±0.02ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.5±0.06ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.01 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q15 1.01 3.3±0.02ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q16 1.01 3.1±0.03ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.7±0.03ms ? ?/sec 1.00 2.7±0.05ms ? ?/sec
physical_plan_clickbench_q19 1.01 3.6±0.03ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.8±0.03ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.01 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.01 2.8±0.02ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.6±0.02ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.1±0.04ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.00 2.9±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q26 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.0±0.02ms ? ?/sec 1.00 4.0±0.03ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.6±0.25ms ? ?/sec 1.00 12.5±0.30ms ? ?/sec
physical_plan_clickbench_q31 1.01 3.7±0.03ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q33 1.01 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.01 2.9±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q36 1.01 3.7±0.02ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.8±0.03ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.8±0.03ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.4±0.02ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.5±0.03ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.9±0.03ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.8±0.04ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.2±0.06ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.6±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.5±0.03ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.3±0.03ms ? ?/sec 1.00 4.3±0.04ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.5±0.04ms ? ?/sec 1.01 4.6±0.05ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.6±0.01ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.1±0.03ms ? ?/sec 1.01 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.03ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.01 2.4±0.03ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.1±0.02ms ? ?/sec 1.01 3.1±0.02ms ? ?/sec
physical_plan_tpcds_all 1.00 1021.6±3.93ms ? ?/sec 1.00 1024.1±3.16ms ? ?/sec
physical_plan_tpch_all 1.00 63.4±0.32ms ? ?/sec 1.01 63.9±0.27ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.9±0.01ms ? ?/sec 1.01 3.9±0.01ms ? ?/sec
physical_plan_tpch_q11 1.00 3.5±0.04ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
physical_plan_tpch_q12 1.00 1792.8±13.18µs ? ?/sec 1.00 1797.8±11.76µs ? ?/sec
physical_plan_tpch_q13 1.00 1442.9±10.57µs ? ?/sec 1.01 1456.1±5.92µs ? ?/sec
physical_plan_tpch_q14 1.00 1956.4±10.56µs ? ?/sec 1.00 1961.4±12.22µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
physical_plan_tpch_q17 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.6±0.01ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_tpch_q19 1.00 3.2±0.02ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_tpch_q2 1.00 5.8±0.04ms ? ?/sec 1.01 5.9±0.02ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.01ms ? ?/sec 1.01 3.2±0.01ms ? ?/sec
physical_plan_tpch_q21 1.00 4.2±0.02ms ? ?/sec 1.01 4.2±0.01ms ? ?/sec
physical_plan_tpch_q22 1.00 2.8±0.06ms ? ?/sec 1.01 2.8±0.02ms ? ?/sec
physical_plan_tpch_q3 1.00 2.6±0.01ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1478.4±8.39µs ? ?/sec 1.01 1488.4±9.29µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.10ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_tpch_q6 1.00 854.4±3.68µs ? ?/sec 1.00 857.4±8.24µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.01ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.4±0.02ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 4.0±0.02ms ? ?/sec 1.00 4.0±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.00 16.8±0.07ms ? ?/sec 1.01 16.9±0.07ms ? ?/sec
physical_select_all_from_1000 1.04 24.2±0.07ms ? ?/sec 1.00 23.4±0.07ms ? ?/sec
physical_select_one_from_700 1.04 1087.9±15.35µs ? ?/sec 1.00 1042.1±4.39µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 5.9±0.02ms ? ?/sec 1.00 6.0±0.02ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 12.8±0.07ms ? ?/sec 1.00 12.9±0.05ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 161.8±1.25ms ? ?/sec 1.00 162.2±1.05ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.00 372.3±2.57ms ? ?/sec 1.00 373.5±2.12ms ? ?/sec
physical_theta_join_consider_sort 1.01 1754.4±7.31µs ? ?/sec 1.00 1729.1±7.06µs ? ?/sec
physical_unnest_to_join 1.01 1844.5±7.78µs ? ?/sec 1.00 1821.9±7.58µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1086.4±7.41µs ? ?/sec 1.00 1085.2±6.24µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 2.2±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 654.7±5.07µs ? ?/sec 1.00 653.0±6.10µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.01 811.3±7.04µs ? ?/sec 1.00 806.9±4.80µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 870.7±8.54µs ? ?/sec 1.01 875.3±37.57µs ? ?/sec
with_param_values_many_columns 1.01 657.1±6.03µs ? ?/sec 1.00 650.3±3.45µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #18678:
URL: https://github.com/apache/datafusion/pull/18678#discussion_r2524877533
##########
datafusion/optimizer/src/eliminate_nested_union.rs:
##########
@@ -55,6 +55,9 @@ impl OptimizerRule for EliminateNestedUnion {
_config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
match plan {
+ LogicalPlan::Union(Union { mut inputs, .. }) if inputs.len() == 1 => Ok(
+ Transformed::yes(Arc::unwrap_or_clone(inputs.pop().unwrap())),
+ ),
Review Comment:
Wow, that's it? Nice find, will be curious if this shows up in planning benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org