You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/01/20 07:54:21 UTC
[PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
zhuqi-lucas opened a new pull request, #16771:
URL: https://github.com/apache/datafusion/pull/16771
## Which issue does this PR close?
Continue to reduce the Expr struct size.
- Closes [#16770](https://github.com/apache/datafusion/issues/16770)
## Rationale for this change
Continue to reduce the Expr struct size.
## What changes are included in this PR?
Continue to reduce the Expr struct size.
## Are these changes tested?
Yes
The size reduce from 128 to 112
Updated:
Continue reduce to 80 now!
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3085737075
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.01 619.3±3.99µs ? ?/sec 1.00 615.6±2.26µs ? ?/sec
logical_select_all_from_1000 1.00 11.2±0.05ms ? ?/sec 1.00 11.2±0.04ms ? ?/sec
logical_select_one_from_700 1.00 405.8±1.41µs ? ?/sec 1.00 406.5±2.02µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 367.0±7.23µs ? ?/sec 1.00 367.9±2.19µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 351.9±1.27µs ? ?/sec 1.00 352.4±2.69µs ? ?/sec
physical_intersection 1.00 823.7±13.80µs ? ?/sec 1.00 823.3±7.95µs ? ?/sec
physical_join_consider_sort 1.00 1352.6±13.56µs ? ?/sec 1.00 1354.6±6.63µs ? ?/sec
physical_join_distinct 1.00 342.1±1.02µs ? ?/sec 1.00 342.6±3.35µs ? ?/sec
physical_many_self_joins 1.00 10.0±0.05ms ? ?/sec 1.00 10.0±0.04ms ? ?/sec
physical_plan_clickbench_all 1.01 188.7±1.84ms ? ?/sec 1.00 187.7±3.23ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.5±0.02ms ? ?/sec 1.05 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.3±0.03ms ? ?/sec 1.04 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.5±0.03ms ? ?/sec 1.03 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.6±0.03ms ? ?/sec 1.04 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.3±0.03ms ? ?/sec 1.03 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.5±0.03ms ? ?/sec 1.03 3.6±0.07ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.4±0.04ms ? ?/sec 1.01 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.2±0.04ms ? ?/sec 1.02 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.3±0.03ms ? ?/sec 1.02 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.02ms ? ?/sec 1.03 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.7±0.03ms ? ?/sec 1.03 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.9±0.02ms ? ?/sec 1.04 3.0±0.07ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.6±0.02ms ? ?/sec 1.02 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.9±0.02ms ? ?/sec 1.03 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.5±0.03ms ? ?/sec 1.02 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.8±0.04ms ? ?/sec 1.02 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.01 4.3±0.03ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.0±0.03ms ? ?/sec 1.03 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.03ms ? ?/sec 1.03 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.0±0.03ms ? ?/sec 1.02 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.8±0.12ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.4±0.04ms ? ?/sec 1.02 4.5±0.04ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.8±0.03ms ? ?/sec 1.03 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.8±0.12ms ? ?/sec 1.00 12.7±0.15ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.8±0.03ms ? ?/sec 1.02 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.3±0.03ms ? ?/sec 1.02 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.0±0.03ms ? ?/sec 1.03 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.1±0.03ms ? ?/sec 1.02 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.8±0.03ms ? ?/sec 1.02 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.8±0.06ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.6±0.03ms ? ?/sec 1.02 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.6±0.05ms ? ?/sec 1.03 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.2±0.05ms ? ?/sec 1.02 4.3±0.05ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.7±0.03ms ? ?/sec 1.02 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.1±0.03ms ? ?/sec 1.03 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.7±0.02ms ? ?/sec 1.03 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.7±0.02ms ? ?/sec 1.03 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.1±0.03ms ? ?/sec 1.02 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.7±0.04ms ? ?/sec 1.02 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.3±0.03ms ? ?/sec 1.02 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.6±0.04ms ? ?/sec 1.02 4.7±0.05ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.8±0.02ms ? ?/sec 1.03 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.1±0.04ms ? ?/sec 1.01 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.03ms ? ?/sec 1.02 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.8±0.03ms ? ?/sec 1.02 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.5±0.02ms ? ?/sec 1.05 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.3±0.03ms ? ?/sec 1.05 3.5±0.05ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.2±0.04ms ? ?/sec 1.04 3.3±0.04ms ? ?/sec
physical_plan_tpcds_all 1.01 1031.8±3.36ms ? ?/sec 1.00 1021.7±3.66ms ? ?/sec
physical_plan_tpch_all 1.01 62.2±0.27ms ? ?/sec 1.00 61.7±0.30ms ? ?/sec
physical_plan_tpch_q1 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.01 3.8±0.02ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.01 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q12 1.01 1777.4±9.12µs ? ?/sec 1.00 1766.5±7.68µs ? ?/sec
physical_plan_tpch_q13 1.00 1445.9±5.98µs ? ?/sec 1.00 1445.9±6.51µs ? ?/sec
physical_plan_tpch_q14 1.00 1919.2±7.27µs ? ?/sec 1.00 1914.0±8.26µs ? ?/sec
physical_plan_tpch_q16 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q17 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_tpch_q19 1.02 3.2±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q2 1.01 5.5±0.03ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_tpch_q20 1.01 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q21 1.00 4.1±0.02ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec
physical_plan_tpch_q22 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1512.3±7.03µs ? ?/sec 1.00 1511.6±8.29µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q6 1.00 852.0±6.30µs ? ?/sec 1.00 851.5±5.11µs ? ?/sec
physical_plan_tpch_q7 1.01 4.2±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.01 5.2±0.02ms ? ?/sec 1.00 5.1±0.03ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.01 17.7±0.08ms ? ?/sec 1.00 17.5±0.07ms ? ?/sec
physical_select_all_from_1000 1.00 24.6±0.07ms ? ?/sec 1.00 24.6±0.14ms ? ?/sec
physical_select_one_from_700 1.00 1034.5±4.94µs ? ?/sec 1.00 1036.4±5.79µs ? ?/sec
physical_sorted_union_orderby 1.00 41.7±0.28ms ? ?/sec 1.00 41.6±0.31ms ? ?/sec
physical_theta_join_consider_sort 1.00 1729.3±7.23µs ? ?/sec 1.00 1735.2±20.95µs ? ?/sec
physical_unnest_to_join 1.00 1282.8±8.18µs ? ?/sec 1.00 1281.4±8.05µs ? ?/sec
with_param_values_many_columns 1.03 129.1±0.88µs ? ?/sec 1.00 125.5±0.66µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3307540344
> What is the current status of this PR? I feel like we had some good progress but then we made some changes / the benchmarks got messed up.
>
> Should we keep pursuing this? Or should we close it?
Thanks @alamb ,
We did not get good performance for this PR in recent benchmarks. I will try to fix conflicts and we may can rerun benchmark to see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3074267706
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (45ca84c2353a099cce7577bb10f353e7764f3554) to d1e6eb4b3fcf1e16391360c48bfe4bf9748ce557 [diff](https://github.com/apache/datafusion/compare/d1e6eb4b3fcf1e16391360c48bfe4bf9748ce557..45ca84c2353a099cce7577bb10f353e7764f3554)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3076797095
Strange, i can't reproduce this in my local:
```rust
with_param_values_many_columns 1.00 129.8±0.74µs ? ?/sec 1.06 137.2±0.79µs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #16771: feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size
URL: https://github.com/apache/datafusion/pull/16771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3133737059
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.00 618.7±2.76µs ? ?/sec 1.01 622.2±2.91µs ? ?/sec
logical_select_all_from_1000 1.00 11.0±0.06ms ? ?/sec 1.04 11.5±0.03ms ? ?/sec
logical_select_one_from_700 1.00 404.3±1.50µs ? ?/sec 1.00 405.5±2.19µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 366.6±5.10µs ? ?/sec 1.00 365.2±2.97µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 351.3±4.33µs ? ?/sec 1.00 351.5±1.33µs ? ?/sec
physical_intersection 1.00 818.2±3.28µs ? ?/sec 1.01 824.3±5.80µs ? ?/sec
physical_join_consider_sort 1.00 1348.5±9.27µs ? ?/sec 1.00 1353.1±7.97µs ? ?/sec
physical_join_distinct 1.00 342.7±1.13µs ? ?/sec 1.00 342.7±2.36µs ? ?/sec
physical_many_self_joins 1.01 10.0±0.09ms ? ?/sec 1.00 10.0±0.05ms ? ?/sec
physical_plan_clickbench_all 1.00 186.5±2.18ms ? ?/sec 1.01 188.0±2.28ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.5±0.02ms ? ?/sec 1.02 2.5±0.11ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.3±0.05ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.5±0.05ms ? ?/sec 1.00 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.3±0.04ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.5±0.06ms ? ?/sec 1.00 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.2±0.02ms ? ?/sec 1.01 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.3±0.02ms ? ?/sec 1.01 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.02ms ? ?/sec 1.01 2.9±0.05ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.8±0.03ms ? ?/sec 1.02 3.8±0.07ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.6±0.06ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.9±0.02ms ? ?/sec 1.01 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.5±0.03ms ? ?/sec 1.02 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.8±0.03ms ? ?/sec 1.01 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.03ms ? ?/sec 1.03 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.0±0.02ms ? ?/sec 1.01 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.03ms ? ?/sec 1.01 2.9±0.04ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.8±0.03ms ? ?/sec 1.01 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.5±0.04ms ? ?/sec 1.02 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.8±0.12ms ? ?/sec 1.04 13.3±0.15ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.8±0.03ms ? ?/sec 1.02 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.8±0.03ms ? ?/sec 1.02 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.3±0.02ms ? ?/sec 1.02 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.0±0.02ms ? ?/sec 1.01 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.1±0.02ms ? ?/sec 1.01 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.8±0.03ms ? ?/sec 1.02 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.8±0.02ms ? ?/sec 1.01 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.8±0.02ms ? ?/sec 1.02 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.6±0.03ms ? ?/sec 1.03 3.7±0.07ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.3±0.04ms ? ?/sec 1.02 4.3±0.07ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.8±0.03ms ? ?/sec 1.02 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.8±0.03ms ? ?/sec 1.01 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.1±0.03ms ? ?/sec 1.02 4.2±0.06ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.8±0.11ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.4±0.07ms ? ?/sec 1.00 4.4±0.07ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.7±0.08ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
physical_plan_clickbench_q5 1.01 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q50 1.02 4.2±0.09ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.3±0.06ms ? ?/sec 1.00 3.3±0.06ms ? ?/sec
physical_plan_clickbench_q6 1.01 2.8±0.04ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.01 2.5±0.12ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.04ms ? ?/sec
physical_plan_tpcds_all 1.00 1040.4±7.76ms ? ?/sec 1.01 1055.1±3.25ms ? ?/sec
physical_plan_tpch_all 1.00 62.7±0.40ms ? ?/sec 1.00 63.0±0.64ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.8±0.01ms ? ?/sec 1.01 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.00 3.3±0.01ms ? ?/sec 1.01 3.3±0.01ms ? ?/sec
physical_plan_tpch_q12 1.01 1800.4±14.91µs ? ?/sec 1.00 1782.0±9.52µs ? ?/sec
physical_plan_tpch_q13 1.00 1460.4±6.34µs ? ?/sec 1.01 1469.8±11.88µs ? ?/sec
physical_plan_tpch_q14 1.00 1934.2±14.43µs ? ?/sec 1.00 1941.0±8.32µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q17 1.00 2.4±0.01ms ? ?/sec 1.01 2.4±0.02ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.03 3.2±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q2 1.00 5.5±0.03ms ? ?/sec 1.01 5.5±0.04ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q21 1.00 4.1±0.03ms ? ?/sec 1.01 4.1±0.02ms ? ?/sec
physical_plan_tpch_q22 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1516.8±10.08µs ? ?/sec 1.01 1526.6±8.99µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.02ms ? ?/sec 1.02 3.2±0.01ms ? ?/sec
physical_plan_tpch_q6 1.00 850.0±5.49µs ? ?/sec 1.00 852.0±3.96µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.02ms ? ?/sec 1.02 4.3±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.2±0.04ms ? ?/sec 1.02 5.3±0.03ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.02 4.2±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.00 18.1±0.13ms ? ?/sec 1.09 19.7±0.09ms ? ?/sec
physical_select_all_from_1000 1.00 24.2±0.10ms ? ?/sec 1.03 24.9±0.09ms ? ?/sec
physical_select_one_from_700 1.00 1031.3±3.98µs ? ?/sec 1.01 1037.3±5.06µs ? ?/sec
physical_sorted_union_orderby 1.00 42.0±0.36ms ? ?/sec 1.01 42.3±0.35ms ? ?/sec
physical_theta_join_consider_sort 1.00 1720.4±23.82µs ? ?/sec 1.08 1852.0±7.39µs ? ?/sec
physical_unnest_to_join 1.00 1283.3±3.87µs ? ?/sec 1.06 1355.6±7.90µs ? ?/sec
with_param_values_many_columns 1.00 130.5±0.79µs ? ?/sec 1.08 140.3±1.16µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3141288216
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (c2c5abe2d1c7ef78f488df6599f2012b750832bd) to 3dc3aaa86e8ed815fcc0619e9bd28966633093f3 [diff](https://github.com/apache/datafusion/compare/3dc3aaa86e8ed815fcc0619e9bd28966633093f3..c2c5abe2d1c7ef78f488df6599f2012b750832bd)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3078726061
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.04 644.1±2.48µs ? ?/sec 1.00 622.3±3.87µs ? ?/sec
logical_select_all_from_1000 1.00 11.1±0.04ms ? ?/sec 1.02 11.3±0.03ms ? ?/sec
logical_select_one_from_700 1.06 430.0±2.43µs ? ?/sec 1.00 406.6±1.75µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.04 381.2±1.36µs ? ?/sec 1.00 367.1±6.18µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.04 367.2±1.42µs ? ?/sec 1.00 351.6±1.38µs ? ?/sec
physical_intersection 1.04 853.4±6.19µs ? ?/sec 1.00 823.8±8.64µs ? ?/sec
physical_join_consider_sort 1.04 1419.2±10.12µs ? ?/sec 1.00 1358.2±6.04µs ? ?/sec
physical_join_distinct 1.05 360.7±1.77µs ? ?/sec 1.00 343.0±2.09µs ? ?/sec
physical_many_self_joins 1.07 10.7±0.07ms ? ?/sec 1.00 10.0±0.04ms ? ?/sec
physical_plan_clickbench_all 1.01 185.5±2.01ms ? ?/sec 1.00 184.0±1.03ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.5±0.05ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.4±0.05ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.5±0.04ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.7±0.06ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.3±0.03ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.5±0.03ms ? ?/sec 1.01 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.04 3.5±0.05ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
physical_plan_clickbench_q16 1.01 3.3±0.04ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.3±0.03ms ? ?/sec 1.01 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.02 3.9±0.06ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.9±0.05ms ? ?/sec 1.01 3.0±0.05ms ? ?/sec
physical_plan_clickbench_q20 1.01 2.7±0.03ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q22 1.04 3.6±0.06ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q23 1.02 3.8±0.05ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.01 4.3±0.03ms ? ?/sec
physical_plan_clickbench_q25 1.02 3.1±0.10ms ? ?/sec 1.00 3.1±0.08ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.04ms ? ?/sec 1.00 2.9±0.08ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.9±0.07ms ? ?/sec 1.02 3.9±0.15ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.5±0.07ms ? ?/sec 1.01 4.5±0.07ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.8±0.05ms ? ?/sec 1.01 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.03 12.8±0.16ms ? ?/sec 1.00 12.5±0.13ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.8±0.05ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.3±0.03ms ? ?/sec 1.01 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.04 3.2±0.05ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.02 3.2±0.07ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.8±0.06ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q37 1.01 3.9±0.05ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q38 1.01 3.8±0.04ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q39 1.01 3.7±0.03ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.6±0.05ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q40 1.04 4.4±0.08ms ? ?/sec 1.00 4.3±0.03ms ? ?/sec
physical_plan_clickbench_q41 1.02 3.9±0.07ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.8±0.05ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.2±0.07ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.04 2.8±0.06ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.01 3.2±0.06ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.01 3.8±0.05ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
physical_plan_clickbench_q48 1.01 4.4±0.06ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.6±0.04ms ? ?/sec 1.00 4.6±0.04ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.8±0.04ms ? ?/sec 1.01 2.8±0.08ms ? ?/sec
physical_plan_clickbench_q50 1.05 4.3±0.09ms ? ?/sec 1.00 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q51 1.01 3.2±0.04ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.8±0.04ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.5±0.03ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.05ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.3±0.05ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_tpcds_all 1.02 1043.2±8.58ms ? ?/sec 1.00 1020.5±3.68ms ? ?/sec
physical_plan_tpch_all 1.02 62.9±0.68ms ? ?/sec 1.00 61.8±0.43ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.01 3.8±0.02ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.01 3.3±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q12 1.02 1799.4±14.71µs ? ?/sec 1.00 1768.5±12.12µs ? ?/sec
physical_plan_tpch_q13 1.00 1456.9±12.44µs ? ?/sec 1.00 1463.4±5.33µs ? ?/sec
physical_plan_tpch_q14 1.01 1924.3±21.56µs ? ?/sec 1.00 1912.4±10.19µs ? ?/sec
physical_plan_tpch_q16 1.02 2.5±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q17 1.02 2.4±0.02ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
physical_plan_tpch_q18 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.06 3.2±0.02ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q2 1.01 5.5±0.04ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_tpch_q20 1.03 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q21 1.02 4.1±0.04ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_plan_tpch_q22 1.02 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.01 2.5±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q4 1.01 1527.6±10.50µs ? ?/sec 1.00 1517.5±9.80µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q6 1.02 861.9±5.92µs ? ?/sec 1.00 842.1±5.20µs ? ?/sec
physical_plan_tpch_q7 1.01 4.2±0.05ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.01 5.2±0.05ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.01 17.8±0.09ms ? ?/sec 1.00 17.6±0.08ms ? ?/sec
physical_select_all_from_1000 1.01 24.9±0.12ms ? ?/sec 1.00 24.6±0.09ms ? ?/sec
physical_select_one_from_700 1.09 1125.1±6.41µs ? ?/sec 1.00 1036.1±11.76µs ? ?/sec
physical_sorted_union_orderby 1.00 42.4±0.39ms ? ?/sec 1.00 42.3±0.66ms ? ?/sec
physical_theta_join_consider_sort 1.04 1788.8±10.25µs ? ?/sec 1.00 1722.3±7.18µs ? ?/sec
physical_unnest_to_join 1.04 1330.1±6.58µs ? ?/sec 1.00 1274.4±6.65µs ? ?/sec
with_param_values_many_columns 1.00 130.8±0.72µs ? ?/sec 1.04 136.5±0.76µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3882861797
run sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3307508870
What is the current status of this PR? I feel like we had some good progress but then we made some changes / the benchmarks got messed up.
Should we keep pursuing this? Or should we close it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3077919308
> 👍 Just some nit for your review.
Thank you @kosiew for review and suggestion, addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3141424350
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.00 625.3±5.41µs ? ?/sec 1.02 638.6±4.90µs ? ?/sec
logical_select_all_from_1000 1.00 11.2±0.03ms ? ?/sec 1.02 11.4±0.05ms ? ?/sec
logical_select_one_from_700 1.00 407.5±1.46µs ? ?/sec 1.04 424.5±1.85µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 368.6±1.13µs ? ?/sec 1.03 379.7±1.68µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 355.7±2.48µs ? ?/sec 1.03 366.6±3.68µs ? ?/sec
physical_intersection 1.00 823.8±6.20µs ? ?/sec 1.03 845.5±4.49µs ? ?/sec
physical_join_consider_sort 1.00 1357.4±5.64µs ? ?/sec 1.04 1408.9±6.90µs ? ?/sec
physical_join_distinct 1.00 345.1±2.08µs ? ?/sec 1.03 355.8±1.11µs ? ?/sec
physical_many_self_joins 1.00 10.0±0.06ms ? ?/sec 1.06 10.6±0.07ms ? ?/sec
physical_plan_clickbench_all 1.00 189.3±1.70ms ? ?/sec 1.05 199.1±10.69ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.6±0.02ms ? ?/sec 1.02 2.6±0.07ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.4±0.03ms ? ?/sec 1.04 3.5±0.15ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.6±0.05ms ? ?/sec 1.06 3.8±0.21ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.8±0.08ms ? ?/sec 1.05 3.9±0.18ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.4±0.03ms ? ?/sec 1.05 3.5±0.16ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.6±0.05ms ? ?/sec 1.02 3.7±0.16ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.5±0.03ms ? ?/sec 1.06 3.7±0.21ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.3±0.04ms ? ?/sec 1.03 3.4±0.14ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.4±0.03ms ? ?/sec 1.05 3.6±0.15ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.02ms ? ?/sec 1.06 3.1±0.15ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.8±0.05ms ? ?/sec 1.07 4.1±0.20ms ? ?/sec
physical_plan_clickbench_q2 1.00 3.0±0.03ms ? ?/sec 1.02 3.0±0.09ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.7±0.03ms ? ?/sec 1.04 2.8±0.08ms ? ?/sec
physical_plan_clickbench_q21 1.00 3.0±0.03ms ? ?/sec 1.05 3.2±0.14ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.6±0.04ms ? ?/sec 1.07 3.8±0.18ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.9±0.04ms ? ?/sec 1.05 4.1±0.19ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.04 4.5±0.20ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.1±0.05ms ? ?/sec 1.05 3.2±0.12ms ? ?/sec
physical_plan_clickbench_q26 1.00 3.0±0.03ms ? ?/sec 1.03 3.0±0.13ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.1±0.04ms ? ?/sec 1.07 3.3±0.12ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.9±0.05ms ? ?/sec 1.07 4.2±0.22ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.6±0.05ms ? ?/sec 1.09 5.0±0.24ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.9±0.03ms ? ?/sec 1.05 3.0±0.09ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.9±0.16ms ? ?/sec 1.06 13.7±0.33ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.9±0.04ms ? ?/sec 1.08 4.2±0.21ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.9±0.04ms ? ?/sec 1.08 4.2±0.21ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.4±0.04ms ? ?/sec 1.11 3.8±0.14ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.1±0.03ms ? ?/sec 1.08 3.4±0.17ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.2±0.02ms ? ?/sec 1.06 3.4±0.11ms ? ?/sec
physical_plan_clickbench_q36 1.00 4.0±0.10ms ? ?/sec 1.03 4.1±0.18ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.9±0.05ms ? ?/sec 1.06 4.2±0.20ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.9±0.04ms ? ?/sec 1.06 4.2±0.21ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.8±0.05ms ? ?/sec 1.07 4.0±0.17ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.6±0.03ms ? ?/sec 1.03 2.7±0.11ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.4±0.08ms ? ?/sec 1.06 4.7±0.22ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.9±0.04ms ? ?/sec 1.05 4.1±0.17ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.9±0.04ms ? ?/sec 1.04 4.0±0.15ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.2±0.05ms ? ?/sec 1.08 4.6±0.22ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.8±0.03ms ? ?/sec 1.04 2.9±0.11ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.8±0.03ms ? ?/sec 1.04 2.9±0.12ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.2±0.03ms ? ?/sec 1.06 3.4±0.13ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.8±0.05ms ? ?/sec 1.09 4.1±0.20ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.5±0.05ms ? ?/sec 1.06 4.7±0.23ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.7±0.05ms ? ?/sec 1.08 5.1±0.28ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.9±0.03ms ? ?/sec 1.04 3.0±0.11ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.3±0.07ms ? ?/sec 1.06 4.5±0.15ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.3±0.03ms ? ?/sec 1.06 3.5±0.15ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.9±0.02ms ? ?/sec 1.01 2.9±0.08ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.6±0.02ms ? ?/sec 1.01 2.6±0.06ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.03ms ? ?/sec 1.01 3.4±0.10ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.3±0.03ms ? ?/sec 1.01 3.3±0.10ms ? ?/sec
physical_plan_tpcds_all 1.00 1037.0±4.58ms ? ?/sec 1.02 1054.4±9.52ms ? ?/sec
physical_plan_tpch_all 1.00 62.9±0.39ms ? ?/sec 1.00 63.0±1.20ms ? ?/sec
physical_plan_tpch_q1 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.8±0.02ms ? ?/sec 1.01 3.8±0.07ms ? ?/sec
physical_plan_tpch_q11 1.00 3.3±0.02ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_tpch_q12 1.00 1810.0±9.40µs ? ?/sec 1.00 1801.7±26.08µs ? ?/sec
physical_plan_tpch_q13 1.00 1466.8±8.13µs ? ?/sec 1.00 1464.8±10.59µs ? ?/sec
physical_plan_tpch_q14 1.01 1936.8±9.22µs ? ?/sec 1.00 1926.2±12.19µs ? ?/sec
physical_plan_tpch_q16 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_tpch_q17 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.05ms ? ?/sec
physical_plan_tpch_q19 1.02 3.2±0.01ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_tpch_q2 1.00 5.5±0.07ms ? ?/sec 1.00 5.5±0.04ms ? ?/sec
physical_plan_tpch_q20 1.01 3.1±0.02ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_tpch_q21 1.00 4.1±0.02ms ? ?/sec 1.01 4.1±0.05ms ? ?/sec
physical_plan_tpch_q22 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.04ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q4 1.01 1533.8±10.45µs ? ?/sec 1.00 1525.9±5.09µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_tpch_q6 1.02 865.4±6.90µs ? ?/sec 1.00 848.0±5.88µs ? ?/sec
physical_plan_tpch_q7 1.00 4.3±0.02ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_tpch_q8 1.00 5.2±0.03ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.02 4.2±0.09ms ? ?/sec
physical_select_aggregates_from_200 1.01 18.3±0.10ms ? ?/sec 1.00 18.1±0.09ms ? ?/sec
physical_select_all_from_1000 1.00 24.6±0.10ms ? ?/sec 1.02 25.1±0.17ms ? ?/sec
physical_select_one_from_700 1.00 1034.0±4.69µs ? ?/sec 1.07 1110.0±7.55µs ? ?/sec
physical_sorted_union_orderby 1.00 41.8±0.29ms ? ?/sec 1.03 43.0±0.41ms ? ?/sec
physical_theta_join_consider_sort 1.00 1728.4±8.41µs ? ?/sec 1.03 1777.1±5.94µs ? ?/sec
physical_unnest_to_join 1.00 1281.0±5.26µs ? ?/sec 1.02 1309.8±7.33µs ? ?/sec
with_param_values_many_columns 1.02 129.0±1.01µs ? ?/sec 1.00 126.3±0.67µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3078771877
> After opening the DF50.0.0 release issue, you can add it to the list
Thank you @xudong963 , added it in https://github.com/apache/datafusion/issues/16799#issuecomment-3078770965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3884933381
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3885846373
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (aba0a3d6bd4b8654f4025c45d805477b8cc5ee60) to f9d37cf769454fd1ce3f5af74ffd9323664a4c6d [diff](https://github.com/apache/datafusion/compare/f9d37cf769454fd1ce3f5af74ffd9323664a4c6d..aba0a3d6bd4b8654f4025c45d805477b8cc5ee60)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3074440773
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.03 637.9±3.29µs ? ?/sec 1.00 620.2±6.47µs ? ?/sec
logical_select_all_from_1000 1.00 11.4±0.10ms ? ?/sec 1.00 11.4±0.05ms ? ?/sec
logical_select_one_from_700 1.04 420.0±1.62µs ? ?/sec 1.00 405.3±1.27µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.04 378.2±1.22µs ? ?/sec 1.00 364.6±1.40µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.04 365.8±1.77µs ? ?/sec 1.00 350.8±1.20µs ? ?/sec
physical_intersection 1.03 841.6±9.73µs ? ?/sec 1.00 818.7±5.01µs ? ?/sec
physical_join_consider_sort 1.04 1403.8±15.17µs ? ?/sec 1.00 1348.2±7.87µs ? ?/sec
physical_join_distinct 1.04 355.5±1.85µs ? ?/sec 1.00 342.2±2.38µs ? ?/sec
physical_many_self_joins 1.05 10.5±0.06ms ? ?/sec 1.00 9.9±0.04ms ? ?/sec
physical_plan_clickbench_all 1.00 186.3±1.54ms ? ?/sec 1.01 189.1±2.61ms ? ?/sec
physical_plan_clickbench_q1 1.05 2.6±0.06ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_clickbench_q10 1.01 3.4±0.03ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q11 1.03 3.6±0.17ms ? ?/sec 1.00 3.5±0.05ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q13 1.01 3.4±0.03ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q14 1.02 3.6±0.08ms ? ?/sec 1.00 3.5±0.05ms ? ?/sec
physical_plan_clickbench_q15 1.02 3.5±0.03ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q16 1.03 3.3±0.09ms ? ?/sec 1.00 3.3±0.05ms ? ?/sec
physical_plan_clickbench_q17 1.02 3.4±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q18 1.04 3.0±0.09ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q19 1.01 3.8±0.04ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.02 2.7±0.07ms ? ?/sec 1.00 2.6±0.06ms ? ?/sec
physical_plan_clickbench_q21 1.01 3.0±0.02ms ? ?/sec 1.00 2.9±0.05ms ? ?/sec
physical_plan_clickbench_q22 1.02 3.6±0.04ms ? ?/sec 1.00 3.5±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.02 3.9±0.09ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.05ms ? ?/sec 1.01 4.3±0.05ms ? ?/sec
physical_plan_clickbench_q25 1.04 3.1±0.08ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.01 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.1±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.02 3.9±0.04ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.5±0.05ms ? ?/sec 1.00 4.5±0.07ms ? ?/sec
physical_plan_clickbench_q3 1.01 2.9±0.04ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q30 1.00 12.7±0.17ms ? ?/sec 1.00 12.7±0.13ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.8±0.04ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q32 1.01 3.8±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.3±0.03ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.1±0.03ms ? ?/sec 1.01 3.2±0.09ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.9±0.04ms ? ?/sec 1.00 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q37 1.02 3.9±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q38 1.02 3.9±0.03ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q39 1.02 3.7±0.04ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q4 1.01 2.6±0.03ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.02 4.3±0.05ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q41 1.01 3.9±0.03ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q42 1.02 3.8±0.04ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.01 4.2±0.06ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.7±0.02ms ? ?/sec 1.02 2.8±0.08ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.8±0.03ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q47 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q48 1.01 4.4±0.06ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q49 1.03 4.8±0.11ms ? ?/sec 1.00 4.6±0.05ms ? ?/sec
physical_plan_clickbench_q5 1.01 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q50 1.03 4.2±0.06ms ? ?/sec 1.00 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q51 1.01 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.02 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q7 1.02 2.6±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.01 3.4±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.04 3.3±0.09ms ? ?/sec 1.00 3.2±0.04ms ? ?/sec
physical_plan_tpcds_all 1.01 1036.0±8.86ms ? ?/sec 1.00 1027.5±8.74ms ? ?/sec
physical_plan_tpch_all 1.01 62.3±0.31ms ? ?/sec 1.00 61.5±0.19ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
physical_plan_tpch_q11 1.02 3.3±0.05ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q12 1.01 1781.5±9.48µs ? ?/sec 1.00 1755.6±9.08µs ? ?/sec
physical_plan_tpch_q13 1.02 1466.9±19.07µs ? ?/sec 1.00 1444.7±5.37µs ? ?/sec
physical_plan_tpch_q14 1.01 1925.3±12.20µs ? ?/sec 1.00 1908.7±8.26µs ? ?/sec
physical_plan_tpch_q16 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q17 1.01 2.4±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.04 3.2±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q2 1.01 5.5±0.02ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_tpch_q20 1.01 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q21 1.01 4.1±0.03ms ? ?/sec 1.00 4.0±0.01ms ? ?/sec
physical_plan_tpch_q22 1.02 2.7±0.03ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q4 1.01 1515.1±12.16µs ? ?/sec 1.00 1501.8±7.57µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q6 1.01 853.6±6.22µs ? ?/sec 1.00 841.8±5.44µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.2±0.03ms ? ?/sec 1.00 5.2±0.03ms ? ?/sec
physical_plan_tpch_q9 1.01 4.1±0.18ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.01 17.6±0.08ms ? ?/sec 1.00 17.5±0.05ms ? ?/sec
physical_select_all_from_1000 1.01 25.0±0.09ms ? ?/sec 1.00 24.7±0.15ms ? ?/sec
physical_select_one_from_700 1.06 1097.8±4.49µs ? ?/sec 1.00 1036.0±6.18µs ? ?/sec
physical_sorted_union_orderby 1.00 41.9±0.31ms ? ?/sec 1.00 41.9±0.30ms ? ?/sec
physical_theta_join_consider_sort 1.03 1764.4±7.18µs ? ?/sec 1.00 1718.2±15.92µs ? ?/sec
physical_unnest_to_join 1.03 1311.7±8.13µs ? ?/sec 1.00 1276.6±5.74µs ? ?/sec
with_param_values_many_columns 1.00 129.1±0.68µs ? ?/sec 1.07 138.3±1.10µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3079295059
> > My guess is that some of the new slowdown / less predictability is due to many more `Box`es (and thus allocations) -- I suggest we reconsider Boxing frequently used structures (like Column in alias, for example) -- that might make Expr larger but would result in less allocator pressure
>
> Thank you @alamb , this is a very good point, let me try this and to see the benchmark result!
@alamb Updated in latest PR, only small change now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3144609244
That is weird -- we would have to see if we can reproduce something similar locally
It may have to do with my shared VM used for benchmarking 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3882865266
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (edf1eafb3e828847ae518aec8c8c01497548fca4) to 69d0f44cc408498ac00adb85d5be3baf0ade8c1f [diff](https://github.com/apache/datafusion/compare/69d0f44cc408498ac00adb85d5be3baf0ade8c1f..edf1eafb3e828847ae518aec8c8c01497548fca4)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3886772042
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.00 628.9±5.47µs ? ?/sec 1.01 633.9±6.02µs ? ?/sec
logical_plan_struct_join_agg_sort 1.01 287.4±2.91µs ? ?/sec 1.00 285.8±2.84µs ? ?/sec
logical_select_all_from_1000 1.00 10.6±0.06ms ? ?/sec 1.04 10.9±0.26ms ? ?/sec
logical_select_one_from_700 1.00 412.7±2.02µs ? ?/sec 1.00 413.0±3.06µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 372.7±2.94µs ? ?/sec 1.01 374.7±5.95µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 358.5±3.30µs ? ?/sec 1.01 360.4±4.68µs ? ?/sec
physical_intersection 1.00 1581.0±23.07µs ? ?/sec 1.01 1592.6±37.21µs ? ?/sec
physical_join_consider_sort 1.00 2.3±0.05ms ? ?/sec 1.01 2.3±0.06ms ? ?/sec
physical_join_distinct 1.00 351.4±9.19µs ? ?/sec 1.00 351.0±1.42µs ? ?/sec
physical_many_self_joins 1.00 12.4±0.32ms ? ?/sec 1.00 12.4±0.20ms ? ?/sec
physical_plan_clickbench_all 1.01 200.1±4.56ms ? ?/sec 1.00 198.7±1.95ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.1±0.02ms ? ?/sec 1.01 2.1±0.06ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.6±0.02ms ? ?/sec 1.02 3.6±0.08ms ? ?/sec
physical_plan_clickbench_q11 1.00 4.1±0.04ms ? ?/sec 1.01 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q12 1.00 4.2±0.08ms ? ?/sec 1.01 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.08ms ? ?/sec
physical_plan_clickbench_q14 1.00 4.1±0.04ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.8±0.02ms ? ?/sec 1.01 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.6±0.08ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.7±0.07ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.6±0.04ms ? ?/sec 1.01 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.00 4.1±0.03ms ? ?/sec 1.01 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.04ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.1±0.01ms ? ?/sec 1.02 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.7±0.04ms ? ?/sec 1.01 2.8±0.01ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.9±0.04ms ? ?/sec 1.01 3.9±0.08ms ? ?/sec
physical_plan_clickbench_q23 1.00 4.1±0.05ms ? ?/sec 1.00 4.1±0.07ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.8±0.02ms ? ?/sec 1.01 4.8±0.06ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.5±0.04ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.01ms ? ?/sec 1.01 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q27 1.01 3.5±0.10ms ? ?/sec 1.00 3.5±0.07ms ? ?/sec
physical_plan_clickbench_q28 1.00 4.4±0.10ms ? ?/sec 1.00 4.4±0.05ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.6±0.02ms ? ?/sec 1.02 4.7±0.16ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.5±0.02ms ? ?/sec 1.01 2.5±0.04ms ? ?/sec
physical_plan_clickbench_q30 1.01 15.4±0.08ms ? ?/sec 1.00 15.3±0.29ms ? ?/sec
physical_plan_clickbench_q31 1.00 4.4±0.02ms ? ?/sec 1.01 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q32 1.00 4.4±0.02ms ? ?/sec 1.01 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.01 3.6±0.06ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.2±0.06ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.3±0.06ms ? ?/sec 1.01 3.3±0.01ms ? ?/sec
physical_plan_clickbench_q36 1.00 4.1±0.04ms ? ?/sec 1.01 4.2±0.11ms ? ?/sec
physical_plan_clickbench_q37 1.01 4.5±0.08ms ? ?/sec 1.00 4.5±0.02ms ? ?/sec
physical_plan_clickbench_q38 1.01 4.6±0.05ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
physical_plan_clickbench_q39 1.02 4.0±0.04ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.1±0.01ms ? ?/sec 1.01 2.2±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.01 4.8±0.06ms ? ?/sec 1.00 4.8±0.07ms ? ?/sec
physical_plan_clickbench_q41 1.02 4.2±0.10ms ? ?/sec 1.00 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q42 1.00 4.1±0.03ms ? ?/sec 1.00 4.1±0.09ms ? ?/sec
physical_plan_clickbench_q43 1.01 4.5±0.08ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.3±0.04ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.3±0.04ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.2±0.09ms ? ?/sec 1.00 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q47 1.00 4.7±0.08ms ? ?/sec 1.00 4.7±0.10ms ? ?/sec
physical_plan_clickbench_q48 1.00 5.1±0.05ms ? ?/sec 1.00 5.1±0.06ms ? ?/sec
physical_plan_clickbench_q49 1.01 5.4±0.18ms ? ?/sec 1.00 5.4±0.04ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.5±0.02ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.04 4.1±0.07ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.5±0.03ms ? ?/sec 1.02 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.07ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.04ms ? ?/sec 1.01 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.5±0.03ms ? ?/sec 1.02 3.6±0.04ms ? ?/sec
physical_plan_struct_join_agg_sort 1.00 2.6±0.03ms ? ?/sec 1.00 2.6±0.05ms ? ?/sec
physical_plan_tpcds_all 1.00 1879.8±17.62ms ? ?/sec 1.02 1913.1±20.08ms ? ?/sec
physical_plan_tpch_all 1.00 123.6±0.68ms ? ?/sec 1.02 125.5±2.53ms ? ?/sec
physical_plan_tpch_q1 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 7.1±0.09ms ? ?/sec 1.01 7.2±0.13ms ? ?/sec
physical_plan_tpch_q11 1.00 8.4±0.05ms ? ?/sec 1.01 8.4±0.09ms ? ?/sec
physical_plan_tpch_q12 1.02 3.0±0.08ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
physical_plan_tpch_q13 1.00 3.0±0.03ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q14 1.01 3.0±0.08ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
physical_plan_tpch_q16 1.01 5.2±0.21ms ? ?/sec 1.00 5.2±0.03ms ? ?/sec
physical_plan_tpch_q17 1.00 5.5±0.07ms ? ?/sec 1.00 5.5±0.03ms ? ?/sec
physical_plan_tpch_q18 1.00 5.9±0.04ms ? ?/sec 1.01 5.9±0.02ms ? ?/sec
physical_plan_tpch_q19 1.01 5.0±0.12ms ? ?/sec 1.00 5.0±0.06ms ? ?/sec
physical_plan_tpch_q2 1.01 12.3±0.41ms ? ?/sec 1.00 12.1±0.06ms ? ?/sec
physical_plan_tpch_q20 1.00 7.9±0.06ms ? ?/sec 1.01 8.0±0.20ms ? ?/sec
physical_plan_tpch_q21 1.00 10.0±0.14ms ? ?/sec 1.01 10.1±0.10ms ? ?/sec
physical_plan_tpch_q22 1.00 6.4±0.08ms ? ?/sec 1.00 6.4±0.06ms ? ?/sec
physical_plan_tpch_q3 1.00 5.5±0.05ms ? ?/sec 1.01 5.6±0.06ms ? ?/sec
physical_plan_tpch_q4 1.00 3.0±0.09ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q5 1.00 5.9±0.10ms ? ?/sec 1.01 5.9±0.05ms ? ?/sec
physical_plan_tpch_q6 1.00 1578.4±31.34µs ? ?/sec 1.00 1580.9±19.63µs ? ?/sec
physical_plan_tpch_q7 1.00 7.0±0.03ms ? ?/sec 1.02 7.2±0.05ms ? ?/sec
physical_plan_tpch_q8 1.00 9.1±0.03ms ? ?/sec 1.02 9.3±0.19ms ? ?/sec
physical_plan_tpch_q9 1.00 6.5±0.03ms ? ?/sec 1.02 6.6±0.06ms ? ?/sec
physical_select_aggregates_from_200 1.03 17.2±0.06ms ? ?/sec 1.00 16.7±0.07ms ? ?/sec
physical_select_all_from_1000 1.00 23.3±0.12ms ? ?/sec 1.02 23.9±0.21ms ? ?/sec
physical_select_one_from_700 1.00 1316.4±11.87µs ? ?/sec 1.00 1316.5±10.20µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 10.9±0.07ms ? ?/sec 1.01 11.0±0.13ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 29.5±0.90ms ? ?/sec 1.00 29.5±0.54ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 194.8±1.44ms ? ?/sec 1.01 196.9±1.28ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.00 1058.0±11.70ms ? ?/sec 1.01 1071.1±9.92ms ? ?/sec
physical_theta_join_consider_sort 1.00 2.6±0.04ms ? ?/sec 1.03 2.7±0.05ms ? ?/sec
physical_unnest_to_join 1.00 3.0±0.03ms ? ?/sec 1.01 3.1±0.02ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1574.9±21.18µs ? ?/sec 1.01 1593.1±37.04µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 2.9±0.04ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 1069.8±8.60µs ? ?/sec 1.01 1076.0±5.24µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 1254.2±10.11µs ? ?/sec 1.00 1255.2±7.90µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 1321.6±18.15µs ? ?/sec 1.00 1326.5±15.07µs ? ?/sec
with_param_values_many_columns 1.00 585.3±10.80µs ? ?/sec 1.02 594.1±9.96µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3142010548
> 🤖: Benchmark completed
>
> Details
>
> ```
> group main reduce_expr_size
> ----- ---- ----------------
> logical_aggregate_with_join 1.00 625.3±5.41µs ? ?/sec 1.02 638.6±4.90µs ? ?/sec
> logical_select_all_from_1000 1.00 11.2±0.03ms ? ?/sec 1.02 11.4±0.05ms ? ?/sec
> logical_select_one_from_700 1.00 407.5±1.46µs ? ?/sec 1.04 424.5±1.85µs ? ?/sec
> logical_trivial_join_high_numbered_columns 1.00 368.6±1.13µs ? ?/sec 1.03 379.7±1.68µs ? ?/sec
> logical_trivial_join_low_numbered_columns 1.00 355.7±2.48µs ? ?/sec 1.03 366.6±3.68µs ? ?/sec
> physical_intersection 1.00 823.8±6.20µs ? ?/sec 1.03 845.5±4.49µs ? ?/sec
> physical_join_consider_sort 1.00 1357.4±5.64µs ? ?/sec 1.04 1408.9±6.90µs ? ?/sec
> physical_join_distinct 1.00 345.1±2.08µs ? ?/sec 1.03 355.8±1.11µs ? ?/sec
> physical_many_self_joins 1.00 10.0±0.06ms ? ?/sec 1.06 10.6±0.07ms ? ?/sec
> physical_plan_clickbench_all 1.00 189.3±1.70ms ? ?/sec 1.05 199.1±10.69ms ? ?/sec
> physical_plan_clickbench_q1 1.00 2.6±0.02ms ? ?/sec 1.02 2.6±0.07ms ? ?/sec
> physical_plan_clickbench_q10 1.00 3.4±0.03ms ? ?/sec 1.04 3.5±0.15ms ? ?/sec
> physical_plan_clickbench_q11 1.00 3.6±0.05ms ? ?/sec 1.06 3.8±0.21ms ? ?/sec
> physical_plan_clickbench_q12 1.00 3.8±0.08ms ? ?/sec 1.05 3.9±0.18ms ? ?/sec
> physical_plan_clickbench_q13 1.00 3.4±0.03ms ? ?/sec 1.05 3.5±0.16ms ? ?/sec
> physical_plan_clickbench_q14 1.00 3.6±0.05ms ? ?/sec 1.02 3.7±0.16ms ? ?/sec
> physical_plan_clickbench_q15 1.00 3.5±0.03ms ? ?/sec 1.06 3.7±0.21ms ? ?/sec
> physical_plan_clickbench_q16 1.00 3.3±0.04ms ? ?/sec 1.03 3.4±0.14ms ? ?/sec
> physical_plan_clickbench_q17 1.00 3.4±0.03ms ? ?/sec 1.05 3.6±0.15ms ? ?/sec
> physical_plan_clickbench_q18 1.00 2.9±0.02ms ? ?/sec 1.06 3.1±0.15ms ? ?/sec
> physical_plan_clickbench_q19 1.00 3.8±0.05ms ? ?/sec 1.07 4.1±0.20ms ? ?/sec
> physical_plan_clickbench_q2 1.00 3.0±0.03ms ? ?/sec 1.02 3.0±0.09ms ? ?/sec
> physical_plan_clickbench_q20 1.00 2.7±0.03ms ? ?/sec 1.04 2.8±0.08ms ? ?/sec
> physical_plan_clickbench_q21 1.00 3.0±0.03ms ? ?/sec 1.05 3.2±0.14ms ? ?/sec
> physical_plan_clickbench_q22 1.00 3.6±0.04ms ? ?/sec 1.07 3.8±0.18ms ? ?/sec
> physical_plan_clickbench_q23 1.00 3.9±0.04ms ? ?/sec 1.05 4.1±0.19ms ? ?/sec
> physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.04 4.5±0.20ms ? ?/sec
> physical_plan_clickbench_q25 1.00 3.1±0.05ms ? ?/sec 1.05 3.2±0.12ms ? ?/sec
> physical_plan_clickbench_q26 1.00 3.0±0.03ms ? ?/sec 1.03 3.0±0.13ms ? ?/sec
> physical_plan_clickbench_q27 1.00 3.1±0.04ms ? ?/sec 1.07 3.3±0.12ms ? ?/sec
> physical_plan_clickbench_q28 1.00 3.9±0.05ms ? ?/sec 1.07 4.2±0.22ms ? ?/sec
> physical_plan_clickbench_q29 1.00 4.6±0.05ms ? ?/sec 1.09 5.0±0.24ms ? ?/sec
> physical_plan_clickbench_q3 1.00 2.9±0.03ms ? ?/sec 1.05 3.0±0.09ms ? ?/sec
> physical_plan_clickbench_q30 1.00 12.9±0.16ms ? ?/sec 1.06 13.7±0.33ms ? ?/sec
> physical_plan_clickbench_q31 1.00 3.9±0.04ms ? ?/sec 1.08 4.2±0.21ms ? ?/sec
> physical_plan_clickbench_q32 1.00 3.9±0.04ms ? ?/sec 1.08 4.2±0.21ms ? ?/sec
> physical_plan_clickbench_q33 1.00 3.4±0.04ms ? ?/sec 1.11 3.8±0.14ms ? ?/sec
> physical_plan_clickbench_q34 1.00 3.1±0.03ms ? ?/sec 1.08 3.4±0.17ms ? ?/sec
> physical_plan_clickbench_q35 1.00 3.2±0.02ms ? ?/sec 1.06 3.4±0.11ms ? ?/sec
> physical_plan_clickbench_q36 1.00 4.0±0.10ms ? ?/sec 1.03 4.1±0.18ms ? ?/sec
> physical_plan_clickbench_q37 1.00 3.9±0.05ms ? ?/sec 1.06 4.2±0.20ms ? ?/sec
> physical_plan_clickbench_q38 1.00 3.9±0.04ms ? ?/sec 1.06 4.2±0.21ms ? ?/sec
> physical_plan_clickbench_q39 1.00 3.8±0.05ms ? ?/sec 1.07 4.0±0.17ms ? ?/sec
> physical_plan_clickbench_q4 1.00 2.6±0.03ms ? ?/sec 1.03 2.7±0.11ms ? ?/sec
> physical_plan_clickbench_q40 1.00 4.4±0.08ms ? ?/sec 1.06 4.7±0.22ms ? ?/sec
> physical_plan_clickbench_q41 1.00 3.9±0.04ms ? ?/sec 1.05 4.1±0.17ms ? ?/sec
> physical_plan_clickbench_q42 1.00 3.9±0.04ms ? ?/sec 1.04 4.0±0.15ms ? ?/sec
> physical_plan_clickbench_q43 1.00 4.2±0.05ms ? ?/sec 1.08 4.6±0.22ms ? ?/sec
> physical_plan_clickbench_q44 1.00 2.8±0.03ms ? ?/sec 1.04 2.9±0.11ms ? ?/sec
> physical_plan_clickbench_q45 1.00 2.8±0.03ms ? ?/sec 1.04 2.9±0.12ms ? ?/sec
> physical_plan_clickbench_q46 1.00 3.2±0.03ms ? ?/sec 1.06 3.4±0.13ms ? ?/sec
> physical_plan_clickbench_q47 1.00 3.8±0.05ms ? ?/sec 1.09 4.1±0.20ms ? ?/sec
> physical_plan_clickbench_q48 1.00 4.5±0.05ms ? ?/sec 1.06 4.7±0.23ms ? ?/sec
> physical_plan_clickbench_q49 1.00 4.7±0.05ms ? ?/sec 1.08 5.1±0.28ms ? ?/sec
> physical_plan_clickbench_q5 1.00 2.9±0.03ms ? ?/sec 1.04 3.0±0.11ms ? ?/sec
> physical_plan_clickbench_q50 1.00 4.3±0.07ms ? ?/sec 1.06 4.5±0.15ms ? ?/sec
> physical_plan_clickbench_q51 1.00 3.3±0.03ms ? ?/sec 1.06 3.5±0.15ms ? ?/sec
> physical_plan_clickbench_q6 1.00 2.9±0.02ms ? ?/sec 1.01 2.9±0.08ms ? ?/sec
> physical_plan_clickbench_q7 1.00 2.6±0.02ms ? ?/sec 1.01 2.6±0.06ms ? ?/sec
> physical_plan_clickbench_q8 1.00 3.4±0.03ms ? ?/sec 1.01 3.4±0.10ms ? ?/sec
> physical_plan_clickbench_q9 1.00 3.3±0.03ms ? ?/sec 1.01 3.3±0.10ms ? ?/sec
> physical_plan_tpcds_all 1.00 1037.0±4.58ms ? ?/sec 1.02 1054.4±9.52ms ? ?/sec
> physical_plan_tpch_all 1.00 62.9±0.39ms ? ?/sec 1.00 63.0±1.20ms ? ?/sec
> physical_plan_tpch_q1 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
> physical_plan_tpch_q10 1.00 3.8±0.02ms ? ?/sec 1.01 3.8±0.07ms ? ?/sec
> physical_plan_tpch_q11 1.00 3.3±0.02ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
> physical_plan_tpch_q12 1.00 1810.0±9.40µs ? ?/sec 1.00 1801.7±26.08µs ? ?/sec
> physical_plan_tpch_q13 1.00 1466.8±8.13µs ? ?/sec 1.00 1464.8±10.59µs ? ?/sec
> physical_plan_tpch_q14 1.01 1936.8±9.22µs ? ?/sec 1.00 1926.2±12.19µs ? ?/sec
> physical_plan_tpch_q16 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
> physical_plan_tpch_q17 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
> physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.05ms ? ?/sec
> physical_plan_tpch_q19 1.02 3.2±0.01ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
> physical_plan_tpch_q2 1.00 5.5±0.07ms ? ?/sec 1.00 5.5±0.04ms ? ?/sec
> physical_plan_tpch_q20 1.01 3.1±0.02ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
> physical_plan_tpch_q21 1.00 4.1±0.02ms ? ?/sec 1.01 4.1±0.05ms ? ?/sec
> physical_plan_tpch_q22 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.04ms ? ?/sec
> physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
> physical_plan_tpch_q4 1.01 1533.8±10.45µs ? ?/sec 1.00 1525.9±5.09µs ? ?/sec
> physical_plan_tpch_q5 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
> physical_plan_tpch_q6 1.02 865.4±6.90µs ? ?/sec 1.00 848.0±5.88µs ? ?/sec
> physical_plan_tpch_q7 1.00 4.3±0.02ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
> physical_plan_tpch_q8 1.00 5.2±0.03ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
> physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.02 4.2±0.09ms ? ?/sec
> physical_select_aggregates_from_200 1.01 18.3±0.10ms ? ?/sec 1.00 18.1±0.09ms ? ?/sec
> physical_select_all_from_1000 1.00 24.6±0.10ms ? ?/sec 1.02 25.1±0.17ms ? ?/sec
> physical_select_one_from_700 1.00 1034.0±4.69µs ? ?/sec 1.07 1110.0±7.55µs ? ?/sec
> physical_sorted_union_orderby 1.00 41.8±0.29ms ? ?/sec 1.03 43.0±0.41ms ? ?/sec
> physical_theta_join_consider_sort 1.00 1728.4±8.41µs ? ?/sec 1.03 1777.1±5.94µs ? ?/sec
> physical_unnest_to_join 1.00 1281.0±5.26µs ? ?/sec 1.02 1309.8±7.33µs ? ?/sec
> with_param_values_many_columns 1.02 129.0±1.01µs ? ?/sec 1.00 126.3±0.67µs ? ?/sec
> ```
Than you @alamb , strange, it always bad performance now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3085636073
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (af82383c4da06da3ce8885da9d78a43c8867eb21) to 63dd4e27db1d176f094e29c7c14ead6128573fca [diff](https://github.com/apache/datafusion/compare/63dd4e27db1d176f094e29c7c14ead6128573fca..af82383c4da06da3ce8885da9d78a43c8867eb21)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3077988985
After opening the DF50.0.0 release issue, you can add it to the list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3078458739
> Strange, i can't reproduce this in my local:
I will rerun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2209914828
##########
datafusion/expr/src/expr_rewriter/mod.rs:
##########
@@ -250,7 +257,9 @@ fn coerce_exprs_for_schema(
let new_type = dst_schema.field(idx).data_type();
if new_type != &expr.get_type(src_schema)? {
match expr {
- Expr::Alias(Alias { expr, name, .. }) => {
+ Expr::Alias(alias_box) => {
+ // alias_box: Box<Alias>
+ let Alias { expr, name, .. } = *alias_box;
Review Comment:
Thank you @kosiew , addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3883377810
Seems still no improvement in latest benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3072240662
Updated:
Successfully changed the size from 128 to 80 in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3882866508
I revive this PR by claude help. cc @alamb
And let's see if benchmark will show improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3885018834
Thanks @alamb, i see, there are already 2 in queue.
> 🤖 Hi @alamb, you asked to view the benchmark queue ([#16771 (comment)](https://github.com/apache/datafusion/pull/16771#issuecomment-3884977302)).
>
> Job User Benchmarks Comment
> `arrow-9393-3884310682.sh` rluvaton builder cast_kernels comparison_kernels concatenate_kernel filter_kernels interleave_kernels sort_kernel take_kernels zip_kernels `https://github.com/apache/arrow-rs/pull/9393#issuecomment-3884310682`
> `16771_3884663325.sh` zhuqi-lucas sql_planner `https://github.com/apache/datafusion/pull/16771#issuecomment-3884663325`
> `20182_3884681965.sh` adriangb with_hashes `https://github.com/apache/datafusion/pull/20182#issuecomment-3884681965`
> `16771_3884933381.sh` zhuqi-lucas sql_planner `https://github.com/apache/datafusion/pull/16771#issuecomment-3884933381`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3884663325
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3884977302
show benchmark queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2206469957
##########
datafusion/sqllogictest/test_files/joins.slt:
##########
@@ -4009,12 +4009,12 @@ logical_plan
09)------------Unnest: lists[__unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))|depth=1] structs[]
10)--------------Projection: generate_series(Int64(1), CAST(outer_ref(t1.t1_int) AS Int64)) AS __unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))
11)----------------EmptyRelation
-physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn(UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" })
+physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn((UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" }))
Review Comment:
I changed it to OuterReference in latest PR, so it will use OuterReference instead of this change now, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3131466594
Thank you @alamb , i do some investigate in do smaller changes in latest PR, could you please trigger the benchmark again, let's see the result now, thanks!
And the size now is from 128 => 96.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3070783072
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.04 641.1±3.04µs ? ?/sec 1.00 615.9±3.14µs ? ?/sec
logical_select_all_from_1000 1.00 11.4±0.06ms ? ?/sec 1.01 11.4±0.04ms ? ?/sec
logical_select_one_from_700 1.06 428.1±3.01µs ? ?/sec 1.00 405.3±2.68µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.05 383.1±1.72µs ? ?/sec 1.00 364.9±2.48µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.05 367.9±1.93µs ? ?/sec 1.00 350.9±1.49µs ? ?/sec
physical_intersection 1.04 850.4±5.13µs ? ?/sec 1.00 818.8±5.07µs ? ?/sec
physical_join_consider_sort 1.05 1418.7±7.80µs ? ?/sec 1.00 1348.9±8.64µs ? ?/sec
physical_join_distinct 1.05 358.3±0.94µs ? ?/sec 1.00 341.1±1.58µs ? ?/sec
physical_many_self_joins 1.08 10.7±0.07ms ? ?/sec 1.00 9.9±0.05ms ? ?/sec
physical_plan_clickbench_all 1.01 187.2±1.44ms ? ?/sec 1.00 185.2±1.54ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.6±0.03ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.01 3.4±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.01 3.6±0.03ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.01 3.4±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q16 1.01 3.3±0.02ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.01 3.8±0.05ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.9±0.02ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.7±0.03ms ? ?/sec 1.00 2.7±0.06ms ? ?/sec
physical_plan_clickbench_q21 1.01 3.0±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.6±0.04ms ? ?/sec 1.00 3.5±0.09ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.8±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q24 1.01 4.3±0.05ms ? ?/sec 1.00 4.3±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.1±0.03ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_clickbench_q26 1.01 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q28 1.01 3.9±0.04ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q29 1.01 4.5±0.06ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.9±0.03ms ? ?/sec 1.01 2.9±0.13ms ? ?/sec
physical_plan_clickbench_q30 1.02 12.9±0.15ms ? ?/sec 1.00 12.7±0.14ms ? ?/sec
physical_plan_clickbench_q31 1.01 3.8±0.04ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q32 1.01 3.9±0.05ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.01 3.4±0.04ms ? ?/sec 1.00 3.4±0.05ms ? ?/sec
physical_plan_clickbench_q34 1.01 3.1±0.03ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.01 3.9±0.11ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.9±0.04ms ? ?/sec 1.00 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.9±0.05ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.6±0.03ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.3±0.05ms ? ?/sec 1.00 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.9±0.04ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.8±0.04ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.2±0.05ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.8±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.01 2.8±0.02ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.01 3.2±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.8±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q48 1.01 4.5±0.04ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q49 1.01 4.7±0.06ms ? ?/sec 1.00 4.7±0.05ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.8±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.01 4.2±0.06ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q51 1.01 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.01 2.8±0.02ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.01 2.6±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.3±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_tpcds_all 1.01 1033.2±2.65ms ? ?/sec 1.00 1026.0±3.38ms ? ?/sec
physical_plan_tpch_all 1.00 62.1±0.38ms ? ?/sec 1.00 61.9±0.34ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.01 3.8±0.02ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.01 3.3±0.01ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_tpch_q12 1.01 1781.3±12.28µs ? ?/sec 1.00 1764.3±6.79µs ? ?/sec
physical_plan_tpch_q13 1.00 1442.1±8.82µs ? ?/sec 1.01 1454.5±10.93µs ? ?/sec
physical_plan_tpch_q14 1.00 1919.5±20.30µs ? ?/sec 1.00 1918.9±14.12µs ? ?/sec
physical_plan_tpch_q16 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q17 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q19 1.02 3.2±0.01ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_tpch_q2 1.01 5.5±0.02ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_tpch_q20 1.01 3.1±0.01ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q21 1.01 4.1±0.05ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec
physical_plan_tpch_q22 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1509.9±9.88µs ? ?/sec 1.00 1511.5±8.97µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q6 1.01 850.3±4.71µs ? ?/sec 1.00 846.0±8.10µs ? ?/sec
physical_plan_tpch_q7 1.01 4.2±0.01ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.01 5.2±0.06ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.04ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.01 17.7±0.08ms ? ?/sec 1.00 17.6±0.08ms ? ?/sec
physical_select_all_from_1000 1.01 25.1±0.08ms ? ?/sec 1.00 24.8±0.11ms ? ?/sec
physical_select_one_from_700 1.08 1119.8±8.70µs ? ?/sec 1.00 1034.4±5.66µs ? ?/sec
physical_sorted_union_orderby 1.00 42.1±0.27ms ? ?/sec 1.01 42.4±0.28ms ? ?/sec
physical_theta_join_consider_sort 1.03 1779.4±8.36µs ? ?/sec 1.00 1719.7±7.17µs ? ?/sec
physical_unnest_to_join 1.03 1321.3±8.77µs ? ?/sec 1.00 1278.5±12.47µs ? ?/sec
with_param_values_many_columns 1.05 128.9±0.76µs ? ?/sec 1.00 123.1±0.68µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3137165165
Thank you for your diligence @zhuqi-lucas
I wonder if we can reproduce the results in https://github.com/apache/datafusion/pull/16771#issuecomment-3070783072
(maybe revert to that set of changes)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3134790124
Thank you @alamb , it looks performance not good, i will investigate further and submit new changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3109177350
I will try only to make Alias changing to Box soon, this should be the best choice from performance aspect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3075698800
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.03 638.2±3.12µs ? ?/sec 1.00 620.3±5.03µs ? ?/sec
logical_select_all_from_1000 1.00 11.3±0.07ms ? ?/sec 1.01 11.4±0.03ms ? ?/sec
logical_select_one_from_700 1.03 419.8±2.92µs ? ?/sec 1.00 407.3±1.32µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.03 378.9±1.38µs ? ?/sec 1.00 367.4±5.82µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.03 364.5±1.35µs ? ?/sec 1.00 353.6±2.57µs ? ?/sec
physical_intersection 1.03 842.8±9.70µs ? ?/sec 1.00 817.5±4.84µs ? ?/sec
physical_join_consider_sort 1.02 1393.1±6.25µs ? ?/sec 1.00 1359.8±33.11µs ? ?/sec
physical_join_distinct 1.04 358.1±4.82µs ? ?/sec 1.00 344.1±1.60µs ? ?/sec
physical_many_self_joins 1.04 10.4±0.05ms ? ?/sec 1.00 10.0±0.07ms ? ?/sec
physical_plan_clickbench_all 1.00 185.4±1.52ms ? ?/sec 1.00 185.6±1.53ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.3±0.04ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q11 1.02 3.6±0.14ms ? ?/sec 1.00 3.6±0.07ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.7±0.03ms ? ?/sec 1.02 3.7±0.08ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.3±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.5±0.03ms ? ?/sec 1.00 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.4±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.2±0.07ms ? ?/sec 1.01 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.3±0.03ms ? ?/sec 1.02 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.8±0.05ms ? ?/sec 1.01 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.9±0.02ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.9±0.02ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.01 3.5±0.07ms ? ?/sec 1.00 3.5±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.8±0.09ms ? ?/sec 1.00 3.8±0.09ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.06ms ? ?/sec 1.01 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.0±0.03ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.05ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.8±0.04ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.5±0.06ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q30 1.01 12.8±0.17ms ? ?/sec 1.00 12.6±0.15ms ? ?/sec
physical_plan_clickbench_q31 1.01 3.9±0.11ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.8±0.04ms ? ?/sec 1.01 3.8±0.08ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.3±0.04ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.9±0.15ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.8±0.03ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.9±0.04ms ? ?/sec 1.00 3.8±0.09ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.7±0.05ms ? ?/sec 1.00 3.7±0.06ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.01 4.3±0.07ms ? ?/sec 1.00 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.01 3.9±0.06ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.8±0.06ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.2±0.08ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.1±0.03ms ? ?/sec 1.01 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.7±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.4±0.05ms ? ?/sec 1.00 4.4±0.05ms ? ?/sec
physical_plan_clickbench_q49 1.01 4.7±0.11ms ? ?/sec 1.00 4.7±0.05ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q50 1.01 4.1±0.08ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.03ms ? ?/sec 1.01 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.8±0.03ms ? ?/sec 1.01 2.8±0.04ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.5±0.02ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_tpcds_all 1.01 1033.8±4.95ms ? ?/sec 1.00 1027.9±4.35ms ? ?/sec
physical_plan_tpch_all 1.00 62.2±0.33ms ? ?/sec 1.00 62.1±0.53ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.8±0.02ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q12 1.01 1784.1±14.73µs ? ?/sec 1.00 1760.5±9.11µs ? ?/sec
physical_plan_tpch_q13 1.00 1456.7±13.57µs ? ?/sec 1.00 1459.2±11.40µs ? ?/sec
physical_plan_tpch_q14 1.00 1915.3±10.76µs ? ?/sec 1.00 1918.9±15.09µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q17 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_tpch_q19 1.04 3.2±0.01ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q2 1.00 5.5±0.03ms ? ?/sec 1.00 5.5±0.04ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q21 1.00 4.0±0.02ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_plan_tpch_q22 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_tpch_q4 1.00 1517.2±9.81µs ? ?/sec 1.00 1516.6±8.58µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q6 1.01 859.5±8.00µs ? ?/sec 1.00 848.9±11.27µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.2±0.03ms ? ?/sec 1.01 5.2±0.04ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.01 4.1±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.01 17.7±0.19ms ? ?/sec 1.00 17.6±0.10ms ? ?/sec
physical_select_all_from_1000 1.01 25.0±0.10ms ? ?/sec 1.00 24.8±0.11ms ? ?/sec
physical_select_one_from_700 1.07 1101.0±8.51µs ? ?/sec 1.00 1029.6±5.26µs ? ?/sec
physical_sorted_union_orderby 1.00 41.9±0.36ms ? ?/sec 1.01 42.2±0.34ms ? ?/sec
physical_theta_join_consider_sort 1.02 1760.6±6.81µs ? ?/sec 1.00 1726.7±12.66µs ? ?/sec
physical_unnest_to_join 1.03 1308.3±11.00µs ? ?/sec 1.00 1275.9±10.38µs ? ?/sec
with_param_values_many_columns 1.00 129.8±0.74µs ? ?/sec 1.06 137.2±0.79µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2209435726
##########
datafusion/expr/src/utils.rs:
##########
@@ -992,7 +996,7 @@ pub fn iter_conjunction_owned(expr: Expr) -> impl Iterator<Item = Expr> {
stack.push(*right);
stack.push(*left);
}
- Expr::Alias(Alias { expr, .. }) => stack.push(*expr),
+ Expr::Alias(alias_box) => stack.push(*alias_box.expr),
Review Comment:
boxed_alias
##########
datafusion/expr/src/expr_rewriter/mod.rs:
##########
@@ -250,7 +257,9 @@ fn coerce_exprs_for_schema(
let new_type = dst_schema.field(idx).data_type();
if new_type != &expr.get_type(src_schema)? {
match expr {
- Expr::Alias(Alias { expr, name, .. }) => {
+ Expr::Alias(alias_box) => {
+ // alias_box: Box<Alias>
+ let Alias { expr, name, .. } = *alias_box;
Review Comment:
nit - use boxed_alias instead, for consistency
##########
datafusion/expr/src/expr_rewriter/order_by.rs:
##########
@@ -137,8 +137,11 @@ fn rewrite_in_terms_of_projection(
/// so avg(c) as average will match avgc
fn expr_match(needle: &Expr, expr: &Expr) -> bool {
// check inside aliases
- if let Expr::Alias(Alias { expr, .. }) = &expr {
- expr.as_ref() == needle
+ if let Expr::Alias(alias_box) = expr {
+ // alias_box: &Box<Alias>, so alias_box.as_ref() is &Alias
+ let alias: &Alias = alias_box.as_ref();
Review Comment:
same as above - boxed_alias
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -242,9 +246,11 @@ impl ExprSchemable for Expr {
/// column that does not exist in the schema.
fn nullable(&self, input_schema: &dyn ExprSchema) -> Result<bool> {
match self {
- Expr::Alias(Alias { expr, .. }) | Expr::Not(expr) | Expr::Negative(expr) => {
+ Expr::Alias(alias_box) => {
+ let Alias { expr, .. } = alias_box.as_ref();
Review Comment:
boxed_alias
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3078761333
> 🤖: Benchmark completed
>
> Details
>
> ```
> group main reduce_expr_size
> ----- ---- ----------------
> logical_aggregate_with_join 1.04 644.1±2.48µs ? ?/sec 1.00 622.3±3.87µs ? ?/sec
> logical_select_all_from_1000 1.00 11.1±0.04ms ? ?/sec 1.02 11.3±0.03ms ? ?/sec
> logical_select_one_from_700 1.06 430.0±2.43µs ? ?/sec 1.00 406.6±1.75µs ? ?/sec
> logical_trivial_join_high_numbered_columns 1.04 381.2±1.36µs ? ?/sec 1.00 367.1±6.18µs ? ?/sec
> logical_trivial_join_low_numbered_columns 1.04 367.2±1.42µs ? ?/sec 1.00 351.6±1.38µs ? ?/sec
> physical_intersection 1.04 853.4±6.19µs ? ?/sec 1.00 823.8±8.64µs ? ?/sec
> physical_join_consider_sort 1.04 1419.2±10.12µs ? ?/sec 1.00 1358.2±6.04µs ? ?/sec
> physical_join_distinct 1.05 360.7±1.77µs ? ?/sec 1.00 343.0±2.09µs ? ?/sec
> physical_many_self_joins 1.07 10.7±0.07ms ? ?/sec 1.00 10.0±0.04ms ? ?/sec
> physical_plan_clickbench_all 1.01 185.5±2.01ms ? ?/sec 1.00 184.0±1.03ms ? ?/sec
> physical_plan_clickbench_q1 1.00 2.5±0.05ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
> physical_plan_clickbench_q10 1.00 3.4±0.05ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
> physical_plan_clickbench_q11 1.00 3.5±0.04ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
> physical_plan_clickbench_q12 1.01 3.7±0.06ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
> physical_plan_clickbench_q13 1.00 3.3±0.03ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
> physical_plan_clickbench_q14 1.00 3.5±0.03ms ? ?/sec 1.01 3.6±0.03ms ? ?/sec
> physical_plan_clickbench_q15 1.04 3.5±0.05ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
> physical_plan_clickbench_q16 1.01 3.3±0.04ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
> physical_plan_clickbench_q17 1.00 3.3±0.03ms ? ?/sec 1.01 3.4±0.02ms ? ?/sec
> physical_plan_clickbench_q18 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
> physical_plan_clickbench_q19 1.02 3.9±0.06ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_clickbench_q2 1.00 2.9±0.05ms ? ?/sec 1.01 3.0±0.05ms ? ?/sec
> physical_plan_clickbench_q20 1.01 2.7±0.03ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
> physical_plan_clickbench_q21 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
> physical_plan_clickbench_q22 1.04 3.6±0.06ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
> physical_plan_clickbench_q23 1.02 3.8±0.05ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.01 4.3±0.03ms ? ?/sec
> physical_plan_clickbench_q25 1.02 3.1±0.10ms ? ?/sec 1.00 3.1±0.08ms ? ?/sec
> physical_plan_clickbench_q26 1.00 2.9±0.04ms ? ?/sec 1.00 2.9±0.08ms ? ?/sec
> physical_plan_clickbench_q27 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
> physical_plan_clickbench_q28 1.00 3.9±0.07ms ? ?/sec 1.02 3.9±0.15ms ? ?/sec
> physical_plan_clickbench_q29 1.00 4.5±0.07ms ? ?/sec 1.01 4.5±0.07ms ? ?/sec
> physical_plan_clickbench_q3 1.00 2.8±0.05ms ? ?/sec 1.01 2.9±0.02ms ? ?/sec
> physical_plan_clickbench_q30 1.03 12.8±0.16ms ? ?/sec 1.00 12.5±0.13ms ? ?/sec
> physical_plan_clickbench_q31 1.00 3.8±0.05ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
> physical_plan_clickbench_q32 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.06ms ? ?/sec
> physical_plan_clickbench_q33 1.00 3.3±0.03ms ? ?/sec 1.01 3.3±0.03ms ? ?/sec
> physical_plan_clickbench_q34 1.04 3.2±0.05ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
> physical_plan_clickbench_q35 1.02 3.2±0.07ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
> physical_plan_clickbench_q36 1.00 3.8±0.06ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
> physical_plan_clickbench_q37 1.01 3.9±0.05ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_clickbench_q38 1.01 3.8±0.04ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_clickbench_q39 1.01 3.7±0.03ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
> physical_plan_clickbench_q4 1.00 2.6±0.05ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
> physical_plan_clickbench_q40 1.04 4.4±0.08ms ? ?/sec 1.00 4.3±0.03ms ? ?/sec
> physical_plan_clickbench_q41 1.02 3.9±0.07ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_clickbench_q42 1.01 3.8±0.05ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
> physical_plan_clickbench_q43 1.00 4.2±0.07ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
> physical_plan_clickbench_q44 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
> physical_plan_clickbench_q45 1.04 2.8±0.06ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
> physical_plan_clickbench_q46 1.01 3.2±0.06ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
> physical_plan_clickbench_q47 1.01 3.8±0.05ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
> physical_plan_clickbench_q48 1.01 4.4±0.06ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
> physical_plan_clickbench_q49 1.00 4.6±0.04ms ? ?/sec 1.00 4.6±0.04ms ? ?/sec
> physical_plan_clickbench_q5 1.00 2.8±0.04ms ? ?/sec 1.01 2.8±0.08ms ? ?/sec
> physical_plan_clickbench_q50 1.05 4.3±0.09ms ? ?/sec 1.00 4.1±0.03ms ? ?/sec
> physical_plan_clickbench_q51 1.01 3.2±0.04ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
> physical_plan_clickbench_q6 1.00 2.8±0.04ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
> physical_plan_clickbench_q7 1.00 2.5±0.03ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
> physical_plan_clickbench_q8 1.00 3.4±0.05ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
> physical_plan_clickbench_q9 1.01 3.3±0.05ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
> physical_plan_tpcds_all 1.02 1043.2±8.58ms ? ?/sec 1.00 1020.5±3.68ms ? ?/sec
> physical_plan_tpch_all 1.02 62.9±0.68ms ? ?/sec 1.00 61.8±0.43ms ? ?/sec
> physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
> physical_plan_tpch_q10 1.01 3.8±0.02ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_tpch_q11 1.01 3.3±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
> physical_plan_tpch_q12 1.02 1799.4±14.71µs ? ?/sec 1.00 1768.5±12.12µs ? ?/sec
> physical_plan_tpch_q13 1.00 1456.9±12.44µs ? ?/sec 1.00 1463.4±5.33µs ? ?/sec
> physical_plan_tpch_q14 1.01 1924.3±21.56µs ? ?/sec 1.00 1912.4±10.19µs ? ?/sec
> physical_plan_tpch_q16 1.02 2.5±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
> physical_plan_tpch_q17 1.02 2.4±0.02ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
> physical_plan_tpch_q18 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
> physical_plan_tpch_q19 1.06 3.2±0.02ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
> physical_plan_tpch_q2 1.01 5.5±0.04ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
> physical_plan_tpch_q20 1.03 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
> physical_plan_tpch_q21 1.02 4.1±0.04ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
> physical_plan_tpch_q22 1.02 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
> physical_plan_tpch_q3 1.01 2.5±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
> physical_plan_tpch_q4 1.01 1527.6±10.50µs ? ?/sec 1.00 1517.5±9.80µs ? ?/sec
> physical_plan_tpch_q5 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
> physical_plan_tpch_q6 1.02 861.9±5.92µs ? ?/sec 1.00 842.1±5.20µs ? ?/sec
> physical_plan_tpch_q7 1.01 4.2±0.05ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
> physical_plan_tpch_q8 1.01 5.2±0.05ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
> physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
> physical_select_aggregates_from_200 1.01 17.8±0.09ms ? ?/sec 1.00 17.6±0.08ms ? ?/sec
> physical_select_all_from_1000 1.01 24.9±0.12ms ? ?/sec 1.00 24.6±0.09ms ? ?/sec
> physical_select_one_from_700 1.09 1125.1±6.41µs ? ?/sec 1.00 1036.1±11.76µs ? ?/sec
> physical_sorted_union_orderby 1.00 42.4±0.39ms ? ?/sec 1.00 42.3±0.66ms ? ?/sec
> physical_theta_join_consider_sort 1.04 1788.8±10.25µs ? ?/sec 1.00 1722.3±7.18µs ? ?/sec
> physical_unnest_to_join 1.04 1330.1±6.58µs ? ?/sec 1.00 1274.4±6.65µs ? ?/sec
> with_param_values_many_columns 1.00 130.8±0.72µs ? ?/sec 1.04 136.5±0.76µs ? ?/sec
> ```
Thank you @alamb , almost all improvement for most cases, but still the last one has %4 regression, and i can't reproduce in my local Mac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3079000603
My guess is that some of the new slowdown / less predictability is due to many more `Box`es (and thus allocations) -- I suggest we reconsider Boxing frequently used structures (like Column in alias, for example) -- that might make Expr larger but would result in less allocator pressure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2205599269
##########
datafusion/expr/src/expr.rs:
##########
@@ -3818,7 +3816,7 @@ mod test {
// If this test fails when you change `Expr`, please try
// `Box`ing the fields to make `Expr` smaller
// See https://github.com/apache/datafusion/issues/16199 for details
- assert_eq!(size_of::<Expr>(), 128);
+ assert_eq!(size_of::<Expr>(), 112);
Review Comment:
This change saves 16 bytes per Expr. Nice.
I will run some planning benchmarks and see if we can see any difference
##########
datafusion/expr/src/expr.rs:
##########
@@ -279,7 +279,7 @@ use sqlparser::ast::{
#[derive(Clone, PartialEq, PartialOrd, Eq, Debug, Hash)]
pub enum Expr {
/// An expression with a specific name.
- Alias(Alias),
+ Alias(Box<Alias>),
Review Comment:
Another change that might be a bit less impactful would be to box the fields of Alias instead
So Alias {
expr: Box<Expr>
..
}
```
It may be just as bad / worse though
##########
datafusion/sqllogictest/test_files/joins.slt:
##########
@@ -4009,12 +4009,12 @@ logical_plan
09)------------Unnest: lists[__unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))|depth=1] structs[]
10)--------------Projection: generate_series(Int64(1), CAST(outer_ref(t1.t1_int) AS Int64)) AS __unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))
11)----------------EmptyRelation
-physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn(UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" })
+physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn((UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" }))
Review Comment:
do you know why the plan changes? I don't think the extra `()` really adds much -- can we restore the original?
##########
datafusion/expr/src/expr.rs:
##########
@@ -362,7 +362,7 @@ pub enum Expr {
Placeholder(Placeholder),
/// A placeholder which holds a reference to a qualified field
/// in the outer query, used for correlated sub queries.
- OuterReferenceColumn(DataType, Column),
+ OuterReferenceColumn(Box<(DataType, Column)>),
Review Comment:
If we are going to make this change anyways, can we also pull this into a named struct rather than a unnamed tuple
like
```rust
struct OuterReference {
// fields here
}
enum Expr {
...
OuterReferenceColumn(OuterReference),
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3886271306
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (aba0a3d6bd4b8654f4025c45d805477b8cc5ee60) to f9d37cf769454fd1ce3f5af74ffd9323664a4c6d [diff](https://github.com/apache/datafusion/compare/f9d37cf769454fd1ce3f5af74ffd9323664a4c6d..aba0a3d6bd4b8654f4025c45d805477b8cc5ee60)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3133792914
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (4c1bfe292e4c6fde60ffe7dd5a6346c994c26eff) to aab44fdaaec28fd13b544ccff239c8b8fe6cee2f [diff](https://github.com/apache/datafusion/compare/aab44fdaaec28fd13b544ccff239c8b8fe6cee2f..4c1bfe292e4c6fde60ffe7dd5a6346c994c26eff)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16771: feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size
URL: https://github.com/apache/datafusion/pull/16771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3583934753
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3071768343
> Thanks @zhuqi-lucas -- this looks quite nice
>
> I think it is an API change so I will mark the PR as such and I think it is a good improvement
>
> However, given it is an API change I think we should
>
> 1. Wait until DataFusion 50.0.0 (let's not make any more changes in DataFusion 49 that we are starting to prepare for)
> 2. Update the upgrading guide in https://datafusion.apache.org/library-user-guide/upgrading.html to explain to people hw to change things
Thank you @alamb for review and i agree that we can do this for Datafusion 50.0.0, and since we have enough time, i want to do more improvement, try to reduce Expr more, from size 128 to 80, i am in progress now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2206218811
##########
datafusion/expr/src/expr.rs:
##########
@@ -3818,7 +3816,7 @@ mod test {
// If this test fails when you change `Expr`, please try
// `Box`ing the fields to make `Expr` smaller
// See https://github.com/apache/datafusion/issues/16199 for details
- assert_eq!(size_of::<Expr>(), 128);
+ assert_eq!(size_of::<Expr>(), 112);
Review Comment:
Thank you @alamb , i am doing more amazing experiment, try to reduce from 128 to 80, so we can save 48 bytes per Expr!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2206468646
##########
datafusion/expr/src/expr.rs:
##########
@@ -362,7 +362,7 @@ pub enum Expr {
Placeholder(Placeholder),
/// A placeholder which holds a reference to a qualified field
/// in the outer query, used for correlated sub queries.
- OuterReferenceColumn(DataType, Column),
+ OuterReferenceColumn(Box<(DataType, Column)>),
Review Comment:
Thank you @alamb for this good suggestion, addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3078463110
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (3cd8e4e5338b48fef2df5c7ab3893ef8143e7527) to c4b9995d9be601907556da2ca538a452e0e909c4 [diff](https://github.com/apache/datafusion/compare/c4b9995d9be601907556da2ca538a452e0e909c4..3cd8e4e5338b48fef2df5c7ab3893ef8143e7527)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3075424716
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (45ca84c2353a099cce7577bb10f353e7764f3554) to d1e6eb4b3fcf1e16391360c48bfe4bf9748ce557 [diff](https://github.com/apache/datafusion/compare/d1e6eb4b3fcf1e16391360c48bfe4bf9748ce557..45ca84c2353a099cce7577bb10f353e7764f3554)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3079128237
> My guess is that some of the new slowdown / less predictability is due to many more `Box`es (and thus allocations) -- I suggest we reconsider Boxing frequently used structures (like Column in alias, for example) -- that might make Expr larger but would result in less allocator pressure
Thank you @alamb , this is a very good point, let me try this and to see the benchmark result!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3086549129
> 🤖: Benchmark completed
>
> Details
>
> ```
> group main reduce_expr_size
> ----- ---- ----------------
> logical_aggregate_with_join 1.01 619.3±3.99µs ? ?/sec 1.00 615.6±2.26µs ? ?/sec
> logical_select_all_from_1000 1.00 11.2±0.05ms ? ?/sec 1.00 11.2±0.04ms ? ?/sec
> logical_select_one_from_700 1.00 405.8±1.41µs ? ?/sec 1.00 406.5±2.02µs ? ?/sec
> logical_trivial_join_high_numbered_columns 1.00 367.0±7.23µs ? ?/sec 1.00 367.9±2.19µs ? ?/sec
> logical_trivial_join_low_numbered_columns 1.00 351.9±1.27µs ? ?/sec 1.00 352.4±2.69µs ? ?/sec
> physical_intersection 1.00 823.7±13.80µs ? ?/sec 1.00 823.3±7.95µs ? ?/sec
> physical_join_consider_sort 1.00 1352.6±13.56µs ? ?/sec 1.00 1354.6±6.63µs ? ?/sec
> physical_join_distinct 1.00 342.1±1.02µs ? ?/sec 1.00 342.6±3.35µs ? ?/sec
> physical_many_self_joins 1.00 10.0±0.05ms ? ?/sec 1.00 10.0±0.04ms ? ?/sec
> physical_plan_clickbench_all 1.01 188.7±1.84ms ? ?/sec 1.00 187.7±3.23ms ? ?/sec
> physical_plan_clickbench_q1 1.00 2.5±0.02ms ? ?/sec 1.05 2.6±0.02ms ? ?/sec
> physical_plan_clickbench_q10 1.00 3.3±0.03ms ? ?/sec 1.04 3.5±0.04ms ? ?/sec
> physical_plan_clickbench_q11 1.00 3.5±0.03ms ? ?/sec 1.03 3.6±0.05ms ? ?/sec
> physical_plan_clickbench_q12 1.00 3.6±0.03ms ? ?/sec 1.04 3.8±0.04ms ? ?/sec
> physical_plan_clickbench_q13 1.00 3.3±0.03ms ? ?/sec 1.03 3.4±0.04ms ? ?/sec
> physical_plan_clickbench_q14 1.00 3.5±0.03ms ? ?/sec 1.03 3.6±0.07ms ? ?/sec
> physical_plan_clickbench_q15 1.00 3.4±0.04ms ? ?/sec 1.01 3.5±0.03ms ? ?/sec
> physical_plan_clickbench_q16 1.00 3.2±0.04ms ? ?/sec 1.02 3.3±0.02ms ? ?/sec
> physical_plan_clickbench_q17 1.00 3.3±0.03ms ? ?/sec 1.02 3.4±0.04ms ? ?/sec
> physical_plan_clickbench_q18 1.00 2.9±0.02ms ? ?/sec 1.03 2.9±0.02ms ? ?/sec
> physical_plan_clickbench_q19 1.00 3.7±0.03ms ? ?/sec 1.03 3.8±0.03ms ? ?/sec
> physical_plan_clickbench_q2 1.00 2.9±0.02ms ? ?/sec 1.04 3.0±0.07ms ? ?/sec
> physical_plan_clickbench_q20 1.00 2.6±0.02ms ? ?/sec 1.02 2.7±0.03ms ? ?/sec
> physical_plan_clickbench_q21 1.00 2.9±0.02ms ? ?/sec 1.03 3.0±0.03ms ? ?/sec
> physical_plan_clickbench_q22 1.00 3.5±0.03ms ? ?/sec 1.02 3.6±0.03ms ? ?/sec
> physical_plan_clickbench_q23 1.00 3.8±0.04ms ? ?/sec 1.02 3.8±0.03ms ? ?/sec
> physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.01 4.3±0.03ms ? ?/sec
> physical_plan_clickbench_q25 1.00 3.0±0.03ms ? ?/sec 1.03 3.1±0.03ms ? ?/sec
> physical_plan_clickbench_q26 1.00 2.9±0.03ms ? ?/sec 1.03 3.0±0.03ms ? ?/sec
> physical_plan_clickbench_q27 1.00 3.0±0.03ms ? ?/sec 1.02 3.1±0.03ms ? ?/sec
> physical_plan_clickbench_q28 1.00 3.8±0.12ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
> physical_plan_clickbench_q29 1.00 4.4±0.04ms ? ?/sec 1.02 4.5±0.04ms ? ?/sec
> physical_plan_clickbench_q3 1.00 2.8±0.03ms ? ?/sec 1.03 2.9±0.03ms ? ?/sec
> physical_plan_clickbench_q30 1.00 12.8±0.12ms ? ?/sec 1.00 12.7±0.15ms ? ?/sec
> physical_plan_clickbench_q31 1.00 3.8±0.03ms ? ?/sec 1.02 3.8±0.03ms ? ?/sec
> physical_plan_clickbench_q32 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.04ms ? ?/sec
> physical_plan_clickbench_q33 1.00 3.3±0.03ms ? ?/sec 1.02 3.4±0.03ms ? ?/sec
> physical_plan_clickbench_q34 1.00 3.0±0.03ms ? ?/sec 1.03 3.1±0.02ms ? ?/sec
> physical_plan_clickbench_q35 1.00 3.1±0.03ms ? ?/sec 1.02 3.2±0.03ms ? ?/sec
> physical_plan_clickbench_q36 1.00 3.8±0.03ms ? ?/sec 1.02 3.9±0.03ms ? ?/sec
> physical_plan_clickbench_q37 1.00 3.8±0.06ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
> physical_plan_clickbench_q38 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.03ms ? ?/sec
> physical_plan_clickbench_q39 1.00 3.6±0.03ms ? ?/sec 1.02 3.7±0.03ms ? ?/sec
> physical_plan_clickbench_q4 1.00 2.6±0.05ms ? ?/sec 1.03 2.6±0.02ms ? ?/sec
> physical_plan_clickbench_q40 1.00 4.2±0.05ms ? ?/sec 1.02 4.3±0.05ms ? ?/sec
> physical_plan_clickbench_q41 1.00 3.8±0.04ms ? ?/sec 1.02 3.9±0.03ms ? ?/sec
> physical_plan_clickbench_q42 1.00 3.7±0.03ms ? ?/sec 1.02 3.8±0.04ms ? ?/sec
> physical_plan_clickbench_q43 1.00 4.1±0.03ms ? ?/sec 1.03 4.2±0.03ms ? ?/sec
> physical_plan_clickbench_q44 1.00 2.7±0.02ms ? ?/sec 1.03 2.8±0.02ms ? ?/sec
> physical_plan_clickbench_q45 1.00 2.7±0.02ms ? ?/sec 1.03 2.8±0.03ms ? ?/sec
> physical_plan_clickbench_q46 1.00 3.1±0.03ms ? ?/sec 1.02 3.2±0.04ms ? ?/sec
> physical_plan_clickbench_q47 1.00 3.7±0.04ms ? ?/sec 1.02 3.8±0.05ms ? ?/sec
> physical_plan_clickbench_q48 1.00 4.3±0.03ms ? ?/sec 1.02 4.4±0.04ms ? ?/sec
> physical_plan_clickbench_q49 1.00 4.6±0.04ms ? ?/sec 1.02 4.7±0.05ms ? ?/sec
> physical_plan_clickbench_q5 1.00 2.8±0.02ms ? ?/sec 1.03 2.8±0.02ms ? ?/sec
> physical_plan_clickbench_q50 1.00 4.1±0.04ms ? ?/sec 1.01 4.1±0.03ms ? ?/sec
> physical_plan_clickbench_q51 1.00 3.2±0.03ms ? ?/sec 1.02 3.3±0.03ms ? ?/sec
> physical_plan_clickbench_q6 1.00 2.8±0.03ms ? ?/sec 1.02 2.9±0.03ms ? ?/sec
> physical_plan_clickbench_q7 1.00 2.5±0.02ms ? ?/sec 1.05 2.6±0.02ms ? ?/sec
> physical_plan_clickbench_q8 1.00 3.3±0.03ms ? ?/sec 1.05 3.5±0.05ms ? ?/sec
> physical_plan_clickbench_q9 1.00 3.2±0.04ms ? ?/sec 1.04 3.3±0.04ms ? ?/sec
> physical_plan_tpcds_all 1.01 1031.8±3.36ms ? ?/sec 1.00 1021.7±3.66ms ? ?/sec
> physical_plan_tpch_all 1.01 62.2±0.27ms ? ?/sec 1.00 61.7±0.30ms ? ?/sec
> physical_plan_tpch_q1 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
> physical_plan_tpch_q10 1.01 3.8±0.02ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_tpch_q11 1.01 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
> physical_plan_tpch_q12 1.01 1777.4±9.12µs ? ?/sec 1.00 1766.5±7.68µs ? ?/sec
> physical_plan_tpch_q13 1.00 1445.9±5.98µs ? ?/sec 1.00 1445.9±6.51µs ? ?/sec
> physical_plan_tpch_q14 1.00 1919.2±7.27µs ? ?/sec 1.00 1914.0±8.26µs ? ?/sec
> physical_plan_tpch_q16 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
> physical_plan_tpch_q17 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
> physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
> physical_plan_tpch_q19 1.02 3.2±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
> physical_plan_tpch_q2 1.01 5.5±0.03ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
> physical_plan_tpch_q20 1.01 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
> physical_plan_tpch_q21 1.00 4.1±0.02ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec
> physical_plan_tpch_q22 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
> physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
> physical_plan_tpch_q4 1.00 1512.3±7.03µs ? ?/sec 1.00 1511.6±8.29µs ? ?/sec
> physical_plan_tpch_q5 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
> physical_plan_tpch_q6 1.00 852.0±6.30µs ? ?/sec 1.00 851.5±5.11µs ? ?/sec
> physical_plan_tpch_q7 1.01 4.2±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
> physical_plan_tpch_q8 1.01 5.2±0.02ms ? ?/sec 1.00 5.1±0.03ms ? ?/sec
> physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
> physical_select_aggregates_from_200 1.01 17.7±0.08ms ? ?/sec 1.00 17.5±0.07ms ? ?/sec
> physical_select_all_from_1000 1.00 24.6±0.07ms ? ?/sec 1.00 24.6±0.14ms ? ?/sec
> physical_select_one_from_700 1.00 1034.5±4.94µs ? ?/sec 1.00 1036.4±5.79µs ? ?/sec
> physical_sorted_union_orderby 1.00 41.7±0.28ms ? ?/sec 1.00 41.6±0.31ms ? ?/sec
> physical_theta_join_consider_sort 1.00 1729.3±7.23µs ? ?/sec 1.00 1735.2±20.95µs ? ?/sec
> physical_unnest_to_join 1.00 1282.8±8.18µs ? ?/sec 1.00 1281.4±8.05µs ? ?/sec
> with_param_values_many_columns 1.03 129.1±0.88µs ? ?/sec 1.00 125.5±0.66µs ? ?/sec
> ```
The result is bad, no performance gain from only OuterReferenceColumn changing to Box, so i guess the improvement coming from Alias changing to Box? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3886218264
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.00 626.2±3.00µs ? ?/sec 1.01 631.5±3.84µs ? ?/sec
logical_plan_struct_join_agg_sort 1.01 288.6±2.56µs ? ?/sec 1.00 285.4±3.91µs ? ?/sec
logical_select_all_from_1000 1.00 10.5±0.06ms ? ?/sec 1.04 10.9±0.06ms ? ?/sec
logical_select_one_from_700 1.00 412.6±3.91µs ? ?/sec 1.01 415.5±7.29µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 369.5±3.11µs ? ?/sec 1.01 373.1±2.78µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 356.6±2.11µs ? ?/sec 1.01 359.1±2.58µs ? ?/sec
physical_intersection 1.00 1580.6±16.60µs ? ?/sec 1.00 1583.8±6.72µs ? ?/sec
physical_join_consider_sort 1.00 2.2±0.01ms ? ?/sec 1.01 2.3±0.04ms ? ?/sec
physical_join_distinct 1.00 349.4±3.50µs ? ?/sec 1.00 349.0±4.77µs ? ?/sec
physical_many_self_joins 1.00 12.4±0.05ms ? ?/sec 1.00 12.4±0.07ms ? ?/sec
physical_plan_clickbench_all 1.00 199.9±3.67ms ? ?/sec 1.03 205.6±2.51ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.1±0.02ms ? ?/sec 1.01 2.1±0.03ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.6±0.04ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.00 4.1±0.06ms ? ?/sec 1.02 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q12 1.00 4.2±0.05ms ? ?/sec 1.01 4.3±0.08ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.8±0.04ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.00 4.1±0.07ms ? ?/sec 1.01 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.8±0.03ms ? ?/sec 1.03 3.9±0.10ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.6±0.04ms ? ?/sec 1.02 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.7±0.04ms ? ?/sec 1.02 3.8±0.11ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.6±0.02ms ? ?/sec 1.02 2.7±0.09ms ? ?/sec
physical_plan_clickbench_q19 1.00 4.1±0.04ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.7±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.1±0.02ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.7±0.01ms ? ?/sec 1.02 2.8±0.13ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.9±0.03ms ? ?/sec 1.03 4.0±0.04ms ? ?/sec
physical_plan_clickbench_q23 1.00 4.2±0.04ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.8±0.03ms ? ?/sec 1.02 4.8±0.03ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.4±0.10ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.06ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.5±0.07ms ? ?/sec 1.01 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 4.4±0.07ms ? ?/sec 1.02 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.7±0.14ms ? ?/sec 1.00 4.7±0.03ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.5±0.04ms ? ?/sec 1.02 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.02 15.7±0.32ms ? ?/sec 1.00 15.4±0.17ms ? ?/sec
physical_plan_clickbench_q31 1.00 4.4±0.07ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
physical_plan_clickbench_q32 1.00 4.4±0.09ms ? ?/sec 1.02 4.5±0.09ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.6±0.07ms ? ?/sec 1.01 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.2±0.07ms ? ?/sec 1.01 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.3±0.08ms ? ?/sec 1.01 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.00 4.1±0.02ms ? ?/sec 1.02 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q37 1.00 4.6±0.08ms ? ?/sec 1.00 4.6±0.09ms ? ?/sec
physical_plan_clickbench_q38 1.01 4.6±0.15ms ? ?/sec 1.00 4.6±0.03ms ? ?/sec
physical_plan_clickbench_q39 1.00 4.0±0.07ms ? ?/sec 1.01 4.0±0.03ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.2±0.04ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.9±0.09ms ? ?/sec 1.01 4.9±0.02ms ? ?/sec
physical_plan_clickbench_q41 1.00 4.2±0.05ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q42 1.00 4.2±0.10ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.5±0.12ms ? ?/sec 1.02 4.6±0.06ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.3±0.03ms ? ?/sec 1.03 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.3±0.05ms ? ?/sec 1.01 2.3±0.05ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.2±0.04ms ? ?/sec 1.04 3.3±0.09ms ? ?/sec
physical_plan_clickbench_q47 1.02 4.9±0.09ms ? ?/sec 1.00 4.8±0.09ms ? ?/sec
physical_plan_clickbench_q48 1.00 5.2±0.09ms ? ?/sec 1.00 5.2±0.12ms ? ?/sec
physical_plan_clickbench_q49 1.00 5.4±0.15ms ? ?/sec 1.01 5.5±0.09ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.5±0.05ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.01 4.2±0.07ms ? ?/sec 1.00 4.1±0.06ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.5±0.06ms ? ?/sec 1.03 3.6±0.09ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.5±0.04ms ? ?/sec 1.00 2.5±0.04ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.1±0.07ms ? ?/sec 1.00 2.1±0.07ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.04ms ? ?/sec 1.00 3.4±0.06ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.6±0.03ms ? ?/sec 1.01 3.6±0.06ms ? ?/sec
physical_plan_struct_join_agg_sort 1.00 2.6±0.02ms ? ?/sec 1.01 2.7±0.02ms ? ?/sec
physical_plan_tpcds_all 1.00 1899.5±14.08ms ? ?/sec 1.02 1936.6±13.89ms ? ?/sec
physical_plan_tpch_all 1.00 125.1±1.29ms ? ?/sec 1.01 126.7±1.56ms ? ?/sec
physical_plan_tpch_q1 1.00 3.0±0.05ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q10 1.00 7.1±0.09ms ? ?/sec 1.01 7.2±0.08ms ? ?/sec
physical_plan_tpch_q11 1.00 8.4±0.07ms ? ?/sec 1.02 8.5±0.07ms ? ?/sec
physical_plan_tpch_q12 1.00 3.0±0.06ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_tpch_q13 1.00 3.0±0.02ms ? ?/sec 1.02 3.0±0.01ms ? ?/sec
physical_plan_tpch_q14 1.00 3.0±0.02ms ? ?/sec 1.01 3.0±0.05ms ? ?/sec
physical_plan_tpch_q16 1.00 5.1±0.07ms ? ?/sec 1.01 5.2±0.06ms ? ?/sec
physical_plan_tpch_q17 1.00 5.5±0.04ms ? ?/sec 1.02 5.6±0.06ms ? ?/sec
physical_plan_tpch_q18 1.00 5.9±0.08ms ? ?/sec 1.01 6.0±0.06ms ? ?/sec
physical_plan_tpch_q19 1.00 5.0±0.06ms ? ?/sec 1.01 5.0±0.14ms ? ?/sec
physical_plan_tpch_q2 1.00 12.2±0.06ms ? ?/sec 1.03 12.5±0.08ms ? ?/sec
physical_plan_tpch_q20 1.00 8.0±0.22ms ? ?/sec 1.02 8.1±0.25ms ? ?/sec
physical_plan_tpch_q21 1.00 10.0±0.36ms ? ?/sec 1.02 10.3±0.14ms ? ?/sec
physical_plan_tpch_q22 1.00 6.4±0.16ms ? ?/sec 1.02 6.5±0.18ms ? ?/sec
physical_plan_tpch_q3 1.00 5.6±0.04ms ? ?/sec 1.02 5.7±0.06ms ? ?/sec
physical_plan_tpch_q4 1.00 2.9±0.03ms ? ?/sec 1.01 3.0±0.05ms ? ?/sec
physical_plan_tpch_q5 1.00 5.9±0.09ms ? ?/sec 1.02 6.1±0.07ms ? ?/sec
physical_plan_tpch_q6 1.00 1602.7±19.23µs ? ?/sec 1.00 1608.7±21.18µs ? ?/sec
physical_plan_tpch_q7 1.00 7.0±0.04ms ? ?/sec 1.04 7.3±0.13ms ? ?/sec
physical_plan_tpch_q8 1.00 9.1±0.06ms ? ?/sec 1.03 9.4±0.12ms ? ?/sec
physical_plan_tpch_q9 1.00 6.5±0.12ms ? ?/sec 1.02 6.7±0.16ms ? ?/sec
physical_select_aggregates_from_200 1.03 17.3±0.11ms ? ?/sec 1.00 16.8±0.08ms ? ?/sec
physical_select_all_from_1000 1.00 23.5±0.14ms ? ?/sec 1.02 24.0±0.12ms ? ?/sec
physical_select_one_from_700 1.00 1320.2±11.15µs ? ?/sec 1.00 1315.8±5.82µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 10.9±0.04ms ? ?/sec 1.02 11.1±0.09ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 29.6±0.61ms ? ?/sec 1.00 29.7±0.21ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 196.2±2.34ms ? ?/sec 1.02 199.5±1.55ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.00 1074.6±8.45ms ? ?/sec 1.02 1100.3±9.51ms ? ?/sec
physical_theta_join_consider_sort 1.00 2.6±0.03ms ? ?/sec 1.03 2.7±0.02ms ? ?/sec
physical_unnest_to_join 1.00 3.1±0.05ms ? ?/sec 1.01 3.1±0.05ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1559.7±10.09µs ? ?/sec 1.01 1576.9±18.21µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 2.9±0.02ms ? ?/sec 1.01 2.9±0.02ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 1072.4±6.83µs ? ?/sec 1.00 1077.7±7.09µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 1249.9±13.54µs ? ?/sec 1.01 1258.7±40.61µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 1322.2±8.92µs ? ?/sec 1.00 1323.6±10.38µs ? ?/sec
with_param_values_many_columns 1.00 588.1±7.07µs ? ?/sec 1.02 599.3±9.81µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2207076263
##########
datafusion/sqllogictest/test_files/joins.slt:
##########
@@ -4009,12 +4009,12 @@ logical_plan
09)------------Unnest: lists[__unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))|depth=1] structs[]
10)--------------Projection: generate_series(Int64(1), CAST(outer_ref(t1.t1_int) AS Int64)) AS __unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))
11)----------------EmptyRelation
-physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn(UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" })
+physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn((UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" }))
Review Comment:
Next step, we may can add debug/fmt for OuterReference to restore the original behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#discussion_r2207426064
##########
datafusion/sqllogictest/test_files/joins.slt:
##########
@@ -4009,12 +4009,12 @@ logical_plan
09)------------Unnest: lists[__unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))|depth=1] structs[]
10)--------------Projection: generate_series(Int64(1), CAST(outer_ref(t1.t1_int) AS Int64)) AS __unnest_placeholder(generate_series(Int64(1),outer_ref(t1.t1_int)))
11)----------------EmptyRelation
-physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn(UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" })
+physical_plan_error This feature is not implemented: Physical plan does not support logical expression OuterReferenceColumn((UInt32, Column { relation: Some(Bare { table: "t1" }), name: "t1_int" }))
Review Comment:
Restore all slt in latest PR, no diff from main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3133571617
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (4c1bfe292e4c6fde60ffe7dd5a6346c994c26eff) to aab44fdaaec28fd13b544ccff239c8b8fe6cee2f [diff](https://github.com/apache/datafusion/compare/aab44fdaaec28fd13b544ccff239c8b8fe6cee2f..4c1bfe292e4c6fde60ffe7dd5a6346c994c26eff)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3133972910
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.00 612.9±1.97µs ? ?/sec 1.01 616.2±4.31µs ? ?/sec
logical_select_all_from_1000 1.00 11.0±0.03ms ? ?/sec 1.05 11.6±0.06ms ? ?/sec
logical_select_one_from_700 1.00 405.0±2.31µs ? ?/sec 1.00 405.0±1.37µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 362.8±1.44µs ? ?/sec 1.01 365.5±2.00µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 349.7±1.99µs ? ?/sec 1.01 352.7±2.33µs ? ?/sec
physical_intersection 1.00 818.7±5.39µs ? ?/sec 1.01 827.2±2.78µs ? ?/sec
physical_join_consider_sort 1.00 1351.7±7.02µs ? ?/sec 1.01 1359.1±5.80µs ? ?/sec
physical_join_distinct 1.00 342.7±2.79µs ? ?/sec 1.01 344.5±1.55µs ? ?/sec
physical_many_self_joins 1.00 9.9±0.04ms ? ?/sec 1.01 10.1±0.11ms ? ?/sec
physical_plan_clickbench_all 1.00 185.7±1.80ms ? ?/sec 1.01 187.7±1.43ms ? ?/sec
physical_plan_clickbench_q1 1.01 2.5±0.03ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.01 3.4±0.03ms ? ?/sec 1.00 3.4±0.05ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.6±0.05ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q12 1.01 3.7±0.05ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.4±0.04ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.6±0.04ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.5±0.05ms ? ?/sec 1.00 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q16 1.01 3.3±0.03ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.8±0.04ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.9±0.02ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.01 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.6±0.03ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.8±0.05ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.04ms ? ?/sec 1.01 4.4±0.05ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.9±0.04ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.6±0.08ms ? ?/sec 1.00 4.6±0.06ms ? ?/sec
physical_plan_clickbench_q3 1.01 2.9±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.00 13.0±0.18ms ? ?/sec 1.01 13.2±0.15ms ? ?/sec
physical_plan_clickbench_q31 1.00 3.8±0.04ms ? ?/sec 1.01 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q32 1.00 3.8±0.04ms ? ?/sec 1.00 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.4±0.03ms ? ?/sec 1.01 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.2±0.04ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.9±0.09ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q37 1.00 3.9±0.04ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.8±0.03ms ? ?/sec 1.01 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q39 1.00 3.7±0.04ms ? ?/sec 1.01 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q4 1.01 2.6±0.03ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.00 4.3±0.05ms ? ?/sec 1.00 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.00 3.9±0.05ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q42 1.00 3.8±0.06ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.00 4.2±0.04ms ? ?/sec 1.06 4.4±0.16ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.7±0.03ms ? ?/sec 1.01 2.8±0.05ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.7±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.7±0.04ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q48 1.01 4.5±0.13ms ? ?/sec 1.00 4.4±0.05ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.7±0.06ms ? ?/sec 1.00 4.7±0.06ms ? ?/sec
physical_plan_clickbench_q5 1.02 2.9±0.02ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.1±0.04ms ? ?/sec 1.01 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.2±0.04ms ? ?/sec 1.01 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.9±0.02ms ? ?/sec 1.00 2.9±0.07ms ? ?/sec
physical_plan_clickbench_q7 1.01 2.6±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.01 3.4±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.3±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_tpcds_all 1.00 1036.0±7.96ms ? ?/sec 1.02 1055.8±10.07ms ? ?/sec
physical_plan_tpch_all 1.00 62.2±0.31ms ? ?/sec 1.01 62.7±0.29ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.8±0.02ms ? ?/sec 1.01 3.8±0.03ms ? ?/sec
physical_plan_tpch_q11 1.00 3.3±0.01ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_tpch_q12 1.00 1784.5±6.48µs ? ?/sec 1.00 1781.5±7.82µs ? ?/sec
physical_plan_tpch_q13 1.00 1454.6±6.81µs ? ?/sec 1.01 1467.9±9.43µs ? ?/sec
physical_plan_tpch_q14 1.00 1923.1±9.65µs ? ?/sec 1.00 1932.4±8.71µs ? ?/sec
physical_plan_tpch_q16 1.00 2.4±0.01ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_tpch_q17 1.00 2.4±0.01ms ? ?/sec 1.01 2.4±0.02ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.02 3.2±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q2 1.00 5.5±0.02ms ? ?/sec 1.01 5.5±0.03ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q21 1.00 4.1±0.02ms ? ?/sec 1.02 4.1±0.02ms ? ?/sec
physical_plan_tpch_q22 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1522.9±13.34µs ? ?/sec 1.00 1526.1±8.33µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.01ms ? ?/sec 1.03 3.2±0.02ms ? ?/sec
physical_plan_tpch_q6 1.01 855.9±5.45µs ? ?/sec 1.00 848.7±3.78µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.02ms ? ?/sec 1.02 4.3±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.2±0.07ms ? ?/sec 1.02 5.3±0.03ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.02 4.2±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.00 18.1±0.08ms ? ?/sec 1.09 19.7±0.14ms ? ?/sec
physical_select_all_from_1000 1.00 24.3±0.10ms ? ?/sec 1.03 25.1±0.17ms ? ?/sec
physical_select_one_from_700 1.00 1036.3±7.33µs ? ?/sec 1.00 1037.3±19.26µs ? ?/sec
physical_sorted_union_orderby 1.00 42.0±0.42ms ? ?/sec 1.01 42.4±0.41ms ? ?/sec
physical_theta_join_consider_sort 1.00 1715.2±6.08µs ? ?/sec 1.09 1861.5±9.87µs ? ?/sec
physical_unnest_to_join 1.00 1278.3±5.77µs ? ?/sec 1.06 1358.2±26.94µs ? ?/sec
with_param_values_many_columns 1.00 128.5±0.67µs ? ?/sec 1.09 139.4±0.72µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3138370526
> Thank you for your diligence @zhuqi-lucas
>
> I wonder if we can reproduce the results in [#16771 (comment)](https://github.com/apache/datafusion/pull/16771#issuecomment-3070783072)
>
> (maybe revert to that set of changes)?
Thank you @alamb , i agree the result is good for that run! Let me do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-4057177399
I think my conclusion is that this PR doesn't really decrease planning time measurably, so marking as a draft for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3882864787
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3883139351
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main reduce_expr_size
----- ---- ----------------
logical_aggregate_with_join 1.00 631.6±2.74µs ? ?/sec 1.01 635.6±2.33µs ? ?/sec
logical_plan_struct_join_agg_sort 1.00 290.5±2.36µs ? ?/sec 1.00 291.5±2.55µs ? ?/sec
logical_select_all_from_1000 1.00 10.5±0.06ms ? ?/sec 1.01 10.6±0.12ms ? ?/sec
logical_select_one_from_700 1.00 411.8±2.86µs ? ?/sec 1.01 417.6±14.06µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 373.1±16.91µs ? ?/sec 1.00 374.8±1.39µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 355.9±5.53µs ? ?/sec 1.02 362.9±5.25µs ? ?/sec
physical_intersection 1.00 1587.6±7.74µs ? ?/sec 1.00 1593.9±24.77µs ? ?/sec
physical_join_consider_sort 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_join_distinct 1.00 348.8±8.09µs ? ?/sec 1.01 351.8±3.00µs ? ?/sec
physical_many_self_joins 1.00 12.4±0.06ms ? ?/sec 1.00 12.4±0.04ms ? ?/sec
physical_plan_clickbench_all 1.00 199.7±2.39ms ? ?/sec 1.00 200.4±4.22ms ? ?/sec
physical_plan_clickbench_q1 1.00 2.1±0.02ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.6±0.03ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q11 1.00 4.1±0.08ms ? ?/sec 1.01 4.1±0.06ms ? ?/sec
physical_plan_clickbench_q12 1.00 4.2±0.10ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.7±0.06ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.00 4.1±0.08ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.8±0.09ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.6±0.02ms ? ?/sec 1.01 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q17 1.01 3.8±0.10ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.05ms ? ?/sec
physical_plan_clickbench_q19 1.00 4.1±0.04ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q2 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.2±0.04ms ? ?/sec 1.00 2.2±0.04ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.7±0.02ms ? ?/sec 1.01 2.8±0.04ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.9±0.02ms ? ?/sec 1.01 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q23 1.00 4.1±0.03ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.8±0.04ms ? ?/sec 1.01 4.8±0.07ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.5±0.01ms ? ?/sec 1.01 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.01ms ? ?/sec 1.01 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.5±0.02ms ? ?/sec 1.01 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.00 4.4±0.04ms ? ?/sec 1.01 4.5±0.11ms ? ?/sec
physical_plan_clickbench_q29 1.00 4.8±0.04ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
physical_plan_clickbench_q3 1.00 2.5±0.02ms ? ?/sec 1.01 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.01 15.6±0.12ms ? ?/sec 1.00 15.5±0.24ms ? ?/sec
physical_plan_clickbench_q31 1.00 4.4±0.05ms ? ?/sec 1.01 4.5±0.10ms ? ?/sec
physical_plan_clickbench_q32 1.01 4.5±0.04ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.00 3.6±0.02ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q34 1.01 3.2±0.04ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.4±0.08ms ? ?/sec 1.00 3.4±0.09ms ? ?/sec
physical_plan_clickbench_q36 1.00 4.2±0.03ms ? ?/sec 1.00 4.2±0.06ms ? ?/sec
physical_plan_clickbench_q37 1.02 4.6±0.11ms ? ?/sec 1.00 4.5±0.09ms ? ?/sec
physical_plan_clickbench_q38 1.01 4.6±0.07ms ? ?/sec 1.00 4.6±0.09ms ? ?/sec
physical_plan_clickbench_q39 1.01 4.0±0.04ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q40 1.02 4.8±0.11ms ? ?/sec 1.00 4.8±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.02 4.2±0.12ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_plan_clickbench_q42 1.01 4.2±0.03ms ? ?/sec 1.00 4.1±0.06ms ? ?/sec
physical_plan_clickbench_q43 1.01 4.5±0.06ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q44 1.00 2.3±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q45 1.00 2.3±0.04ms ? ?/sec 1.01 2.3±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.2±0.05ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.00 4.7±0.05ms ? ?/sec 1.01 4.7±0.04ms ? ?/sec
physical_plan_clickbench_q48 1.01 5.1±0.15ms ? ?/sec 1.00 5.1±0.03ms ? ?/sec
physical_plan_clickbench_q49 1.02 5.4±0.16ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.04 4.1±0.09ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec
physical_plan_clickbench_q51 1.01 3.5±0.10ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.05ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.1±0.03ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.02ms ? ?/sec 1.01 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q9 1.00 3.6±0.04ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_struct_join_agg_sort 1.00 2.7±0.06ms ? ?/sec 1.02 2.7±0.12ms ? ?/sec
physical_plan_tpcds_all 1.00 1904.5±17.21ms ? ?/sec 1.00 1912.4±16.28ms ? ?/sec
physical_plan_tpch_all 1.01 126.3±2.32ms ? ?/sec 1.00 125.2±1.21ms ? ?/sec
physical_plan_tpch_q1 1.00 3.0±0.02ms ? ?/sec 1.02 3.0±0.09ms ? ?/sec
physical_plan_tpch_q10 1.01 7.2±0.21ms ? ?/sec 1.00 7.2±0.11ms ? ?/sec
physical_plan_tpch_q11 1.01 8.5±0.18ms ? ?/sec 1.00 8.4±0.03ms ? ?/sec
physical_plan_tpch_q12 1.01 3.0±0.06ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_tpch_q13 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q14 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.05ms ? ?/sec
physical_plan_tpch_q16 1.00 5.1±0.07ms ? ?/sec 1.00 5.2±0.06ms ? ?/sec
physical_plan_tpch_q17 1.01 5.6±0.12ms ? ?/sec 1.00 5.6±0.03ms ? ?/sec
physical_plan_tpch_q18 1.02 6.0±0.14ms ? ?/sec 1.00 5.9±0.07ms ? ?/sec
physical_plan_tpch_q19 1.02 5.0±0.02ms ? ?/sec 1.00 5.0±0.05ms ? ?/sec
physical_plan_tpch_q2 1.01 12.2±0.12ms ? ?/sec 1.00 12.2±0.22ms ? ?/sec
physical_plan_tpch_q20 1.00 8.0±0.06ms ? ?/sec 1.01 8.0±0.26ms ? ?/sec
physical_plan_tpch_q21 1.00 10.1±0.06ms ? ?/sec 1.00 10.1±0.15ms ? ?/sec
physical_plan_tpch_q22 1.01 6.5±0.23ms ? ?/sec 1.00 6.4±0.11ms ? ?/sec
physical_plan_tpch_q3 1.01 5.6±0.17ms ? ?/sec 1.00 5.6±0.05ms ? ?/sec
physical_plan_tpch_q4 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q5 1.01 6.0±0.19ms ? ?/sec 1.00 5.9±0.03ms ? ?/sec
physical_plan_tpch_q6 1.01 1601.2±29.06µs ? ?/sec 1.00 1578.8±17.54µs ? ?/sec
physical_plan_tpch_q7 1.01 7.1±0.13ms ? ?/sec 1.00 7.1±0.04ms ? ?/sec
physical_plan_tpch_q8 1.00 9.2±0.12ms ? ?/sec 1.00 9.2±0.08ms ? ?/sec
physical_plan_tpch_q9 1.00 6.6±0.18ms ? ?/sec 1.00 6.6±0.09ms ? ?/sec
physical_select_aggregates_from_200 1.02 17.2±0.07ms ? ?/sec 1.00 16.9±0.04ms ? ?/sec
physical_select_all_from_1000 1.01 23.3±0.15ms ? ?/sec 1.00 23.1±0.13ms ? ?/sec
physical_select_one_from_700 1.00 1317.4±9.30µs ? ?/sec 1.01 1326.7±20.01µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 11.0±0.08ms ? ?/sec 1.01 11.0±0.26ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.00 29.5±0.19ms ? ?/sec 1.00 29.3±0.07ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.00 196.6±1.46ms ? ?/sec 1.01 198.1±2.82ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.00 1069.7±11.19ms ? ?/sec 1.01 1079.9±12.36ms ? ?/sec
physical_theta_join_consider_sort 1.01 2.7±0.06ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_unnest_to_join 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 1573.6±8.85µs ? ?/sec 1.00 1581.2±22.41µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 2.9±0.02ms ? ?/sec 1.02 3.0±0.08ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 1082.8±13.77µs ? ?/sec 1.00 1084.0±5.80µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 1247.6±5.47µs ? ?/sec 1.01 1256.8±12.42µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.00 1319.6±7.72µs ? ?/sec 1.01 1333.5±7.00µs ? ?/sec
with_param_values_many_columns 1.00 582.6±4.12µs ? ?/sec 1.02 593.7±4.55µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3884978541
🤖 Hi @alamb, you asked to view the benchmark queue (https://github.com/apache/datafusion/pull/16771#issuecomment-3884977302).
| Job | User | Benchmarks | Comment |
| --- | --- | --- | --- |
| `arrow-9393-3884310682.sh` | rluvaton | builder cast_kernels comparison_kernels concatenate_kernel filter_kernels interleave_kernels sort_kernel take_kernels zip_kernels | `https://github.com/apache/arrow-rs/pull/9393#issuecomment-3884310682` |
| `16771_3884663325.sh` | zhuqi-lucas | sql_planner | `https://github.com/apache/datafusion/pull/16771#issuecomment-3884663325` |
| `20182_3884681965.sh` | adriangb | with_hashes | `https://github.com/apache/datafusion/pull/20182#issuecomment-3884681965` |
| `16771_3884933381.sh` | zhuqi-lucas | sql_planner | `https://github.com/apache/datafusion/pull/16771#issuecomment-3884933381` |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3885179375
> Thanks @alamb, i see, there are already 2 in queue.
>
👍 -- I just wanted to double check that the runner hadn't gotten stuck again (it sometimes does break / get restarted and needs manual intervention(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias and OuterReferenceColumn to Box type for reducing expr struct size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3070641130
🤖 `./gh_compare_branch_bench.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch_bench.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing reduce_expr_size (d62d0604324eb22a570248356e705cb51fbd453d) to 3291e4ec26a0efe361d8e4ddde56f9a27f8817ee [diff](https://github.com/apache/datafusion/compare/3291e4ec26a0efe361d8e4ddde56f9a27f8817ee..d62d0604324eb22a570248356e705cb51fbd453d)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=reduce_expr_size
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: change Expr Alias ,OuterReferenceColumn, Column to Box type for reducing expr struct size [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16771:
URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3076583715
> 🤖: Benchmark completed
>
> Details
Thank you @alamb for benchmark, interesting, it seems the performance decrease from 112 size to 80, i will investigate it, if i can't find the root cause, i will revert to 112 bytes first!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org