You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jizezhang (via GitHub)" <gi...@apache.org> on 2025/12/09 15:49:06 UTC
[PR] remove repartition exec from coalesce batches optimizer [datafusion]
jizezhang opened a new pull request, #19239:
URL: https://github.com/apache/datafusion/pull/19239
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3657340366
I merged up to resolve some conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3660736875
(I don't think this PR makes any meaningful difference on performance, as expected)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3644118512
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3660734795
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3660738012
Thank you @jizezhang @Dandandan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3662130327
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2752.93 ms โ 2833.55 ms โ no change โ
โ QQuery 1 โ 1334.02 ms โ 1313.99 ms โ no change โ
โ QQuery 2 โ 2503.21 ms โ 2515.26 ms โ no change โ
โ QQuery 3 โ 1137.68 ms โ 1166.77 ms โ no change โ
โ QQuery 4 โ 2333.81 ms โ 2321.91 ms โ no change โ
โ QQuery 5 โ 29034.68 ms โ 29119.55 ms โ no change โ
โ QQuery 6 โ 4165.77 ms โ 4185.77 ms โ no change โ
โ QQuery 7 โ 3770.29 ms โ 3678.40 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 47032.39ms โ
โ Total Time (jizezhang_main) โ 47135.20ms โ
โ Average Time (HEAD) โ 5879.05ms โ
โ Average Time (jizezhang_main) โ 5891.90ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 8 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.46 ms โ 2.28 ms โ +1.08x faster โ
โ QQuery 1 โ 50.46 ms โ 49.18 ms โ no change โ
โ QQuery 2 โ 136.12 ms โ 141.33 ms โ no change โ
โ QQuery 3 โ 153.27 ms โ 156.12 ms โ no change โ
โ QQuery 4 โ 1151.18 ms โ 1091.50 ms โ +1.05x faster โ
โ QQuery 5 โ 1590.18 ms โ 1522.21 ms โ no change โ
โ QQuery 6 โ 2.31 ms โ 2.12 ms โ +1.09x faster โ
โ QQuery 7 โ 57.88 ms โ 57.94 ms โ no change โ
โ QQuery 8 โ 1556.44 ms โ 1482.48 ms โ no change โ
โ QQuery 9 โ 1947.87 ms โ 1932.25 ms โ no change โ
โ QQuery 10 โ 381.38 ms โ 394.58 ms โ no change โ
โ QQuery 11 โ 433.58 ms โ 438.70 ms โ no change โ
โ QQuery 12 โ 1418.75 ms โ 1390.06 ms โ no change โ
โ QQuery 13 โ 2069.43 ms โ 2003.36 ms โ no change โ
โ QQuery 14 โ 1332.84 ms โ 1292.79 ms โ no change โ
โ QQuery 15 โ 1342.43 ms โ 1264.79 ms โ +1.06x faster โ
โ QQuery 16 โ 2809.11 ms โ 2746.40 ms โ no change โ
โ QQuery 17 โ 2761.72 ms โ 2781.23 ms โ no change โ
โ QQuery 18 โ 5907.30 ms โ 5189.89 ms โ +1.14x faster โ
โ QQuery 19 โ 125.46 ms โ 126.28 ms โ no change โ
โ QQuery 20 โ 1972.57 ms โ 1886.73 ms โ no change โ
โ QQuery 21 โ 2277.15 ms โ 2240.41 ms โ no change โ
โ QQuery 22 โ 3916.07 ms โ 3869.49 ms โ no change โ
โ QQuery 23 โ 17593.38 ms โ 12708.17 ms โ +1.38x faster โ
โ QQuery 24 โ 232.17 ms โ 208.77 ms โ +1.11x faster โ
โ QQuery 25 โ 483.36 ms โ 467.84 ms โ no change โ
โ QQuery 26 โ 221.06 ms โ 225.65 ms โ no change โ
โ QQuery 27 โ 2820.67 ms โ 2725.06 ms โ no change โ
โ QQuery 28 โ 25232.60 ms โ 24332.50 ms โ no change โ
โ QQuery 29 โ 983.71 ms โ 983.07 ms โ no change โ
โ QQuery 30 โ 1391.12 ms โ 1345.42 ms โ no change โ
โ QQuery 31 โ 1418.65 ms โ 1344.50 ms โ +1.06x faster โ
โ QQuery 32 โ 5239.80 ms โ 4909.99 ms โ +1.07x faster โ
โ QQuery 33 โ 6444.06 ms โ 5842.59 ms โ +1.10x faster โ
โ QQuery 34 โ 6505.13 ms โ 6010.55 ms โ +1.08x faster โ
โ QQuery 35 โ 2059.10 ms โ 1928.39 ms โ +1.07x faster โ
โ QQuery 36 โ 69.64 ms โ 67.15 ms โ no change โ
โ QQuery 37 โ 46.41 ms โ 45.93 ms โ no change โ
โ QQuery 38 โ 68.21 ms โ 67.84 ms โ no change โ
โ QQuery 39 โ 101.59 ms โ 104.09 ms โ no change โ
โ QQuery 40 โ 26.71 ms โ 29.41 ms โ 1.10x slower โ
โ QQuery 41 โ 24.16 ms โ 23.76 ms โ no change โ
โ QQuery 42 โ 20.85 ms โ 23.18 ms โ 1.11x slower โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 104378.34ms โ
โ Total Time (jizezhang_main) โ 95456.00ms โ
โ Average Time (HEAD) โ 2427.40ms โ
โ Average Time (jizezhang_main) โ 2219.91ms โ
โ Queries Faster โ 12 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 29 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 133.92 ms โ 135.04 ms โ no change โ
โ QQuery 2 โ 30.58 ms โ 30.82 ms โ no change โ
โ QQuery 3 โ 41.51 ms โ 33.86 ms โ +1.23x faster โ
โ QQuery 4 โ 30.71 ms โ 30.13 ms โ no change โ
โ QQuery 5 โ 91.80 ms โ 90.41 ms โ no change โ
โ QQuery 6 โ 19.98 ms โ 19.90 ms โ no change โ
โ QQuery 7 โ 253.11 ms โ 250.10 ms โ no change โ
โ QQuery 8 โ 39.48 ms โ 37.46 ms โ +1.05x faster โ
โ QQuery 9 โ 114.49 ms โ 113.47 ms โ no change โ
โ QQuery 10 โ 67.41 ms โ 63.91 ms โ +1.05x faster โ
โ QQuery 11 โ 18.61 ms โ 18.11 ms โ no change โ
โ QQuery 12 โ 54.01 ms โ 53.18 ms โ no change โ
โ QQuery 13 โ 51.73 ms โ 52.95 ms โ no change โ
โ QQuery 14 โ 14.59 ms โ 14.64 ms โ no change โ
โ QQuery 15 โ 25.75 ms โ 24.91 ms โ no change โ
โ QQuery 16 โ 25.84 ms โ 25.45 ms โ no change โ
โ QQuery 17 โ 161.77 ms โ 158.14 ms โ no change โ
โ QQuery 18 โ 302.07 ms โ 303.76 ms โ no change โ
โ QQuery 19 โ 38.79 ms โ 39.23 ms โ no change โ
โ QQuery 20 โ 52.62 ms โ 50.85 ms โ no change โ
โ QQuery 21 โ 343.00 ms โ 313.00 ms โ +1.10x faster โ
โ QQuery 22 โ 17.93 ms โ 17.45 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 1929.70ms โ
โ Total Time (jizezhang_main) โ 1876.78ms โ
โ Average Time (HEAD) โ 87.71ms โ
โ Average Time (jizezhang_main) โ 85.31ms โ
โ Queries Faster โ 4 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 18 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3655457258
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (d68586644549ff5e0ef43ca3aabe796462b1d88e) to 033116e8a2c2849a45836e6d5952c366aff24ab1 [diff](https://github.com/apache/datafusion/compare/033116e8a2c2849a45836e6d5952c366aff24ab1..d68586644549ff5e0ef43ca3aabe796462b1d88e) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3644118825
run benchmark tpch_mem10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3644318212
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (446f6cf4e066fe43df888ff93258758550a34143) to 20870c18a418ec081d44ecf8a90a30a95aa53138 [diff](https://github.com/apache/datafusion/compare/20870c18a418ec081d44ecf8a90a30a95aa53138..446f6cf4e066fe43df888ff93258758550a34143) using: tpch_mem10
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3655456628
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #19239:
URL: https://github.com/apache/datafusion/pull/19239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3661991989
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (3d70d935cbcc02f18b448090bd7361784b123d37) to 199cf06d20d2a24adbf843eaf6fb2a43eb1ecc40 [diff](https://github.com/apache/datafusion/compare/199cf06d20d2a24adbf843eaf6fb2a43eb1ecc40..3d70d935cbcc02f18b448090bd7361784b123d37) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#discussion_r2612321700
##########
datafusion/physical-optimizer/src/coalesce_batches.rs:
##########
@@ -57,23 +55,7 @@ impl PhysicalOptimizerRule for CoalesceBatches {
let target_batch_size = config.execution.batch_size;
plan.transform_up(|plan| {
let plan_any = plan.as_any();
- let wrap_in_coalesce = plan_any
- // Don't need to add CoalesceBatchesExec after a round robin RepartitionExec
- .downcast_ref::<RepartitionExec>()
- .map(|repart_exec| {
- !matches!(
- repart_exec.partitioning().clone(),
- Partitioning::RoundRobinBatch(_)
- )
- })
- .unwrap_or(false);
-
- if wrap_in_coalesce {
- Ok(Transformed::yes(Arc::new(CoalesceBatchesExec::new(
- plan,
- target_batch_size,
- ))))
- } else if let Some(async_exec) = plan_any.downcast_ref::<AsyncFuncExec>() {
+ if let Some(async_exec) = plan_any.downcast_ref::<AsyncFuncExec>() {
Review Comment:
looks like the only thing left is AsyncFuncExec and we could entirely remove the CoalesceBatches pass (and maybe even structure)
##########
datafusion/sqllogictest/test_files/tpch/plans/q8.slt.part:
##########
@@ -94,55 +94,40 @@ physical_plan
02)--SortExec: expr=[o_year@0 ASC NULLS LAST], preserve_partitioning=[true]
03)----ProjectionExec: expr=[o_year@0 as o_year, CAST(CAST(sum(CASE WHEN all_nations.nation = Utf8("BRAZIL") THEN all_nations.volume ELSE Int64(0) END)@1 AS Decimal128(12, 2)) / CAST(sum(all_nations.volume)@2 AS Decimal128(12, 2)) AS Decimal128(15, 2)) as mkt_share]
04)------AggregateExec: mode=FinalPartitioned, gby=[o_year@0 as o_year], aggr=[sum(CASE WHEN all_nations.nation = Utf8("BRAZIL") THEN all_nations.volume ELSE Int64(0) END), sum(all_nations.volume)]
-05)--------CoalesceBatchesExec: target_batch_size=8192
-06)----------RepartitionExec: partitioning=Hash([o_year@0], 4), input_partitions=4
-07)------------AggregateExec: mode=Partial, gby=[o_year@0 as o_year], aggr=[sum(CASE WHEN all_nations.nation = Utf8("BRAZIL") THEN all_nations.volume ELSE Int64(0) END), sum(all_nations.volume)]
-08)--------------ProjectionExec: expr=[date_part(YEAR, o_orderdate@2) as o_year, l_extendedprice@0 * (Some(1),20,0 - l_discount@1) as volume, n_name@3 as nation]
-09)----------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(n_regionkey@3, r_regionkey@0)], projection=[l_extendedprice@0, l_discount@1, o_orderdate@2, n_name@4]
-10)------------------CoalesceBatchesExec: target_batch_size=8192
-11)--------------------RepartitionExec: partitioning=Hash([n_regionkey@3], 4), input_partitions=4
-12)----------------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(s_nationkey@2, n_nationkey@0)], projection=[l_extendedprice@0, l_discount@1, o_orderdate@3, n_regionkey@4, n_name@6]
-13)------------------------CoalesceBatchesExec: target_batch_size=8192
-14)--------------------------RepartitionExec: partitioning=Hash([s_nationkey@2], 4), input_partitions=4
-15)----------------------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(c_nationkey@4, n_nationkey@0)], projection=[l_extendedprice@0, l_discount@1, s_nationkey@2, o_orderdate@3, n_regionkey@6]
-16)------------------------------CoalesceBatchesExec: target_batch_size=8192
-17)--------------------------------RepartitionExec: partitioning=Hash([c_nationkey@4], 4), input_partitions=4
-18)----------------------------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_custkey@3, c_custkey@0)], projection=[l_extendedprice@0, l_discount@1, s_nationkey@2, o_orderdate@4, c_nationkey@6]
-19)------------------------------------CoalesceBatchesExec: target_batch_size=8192
-20)--------------------------------------RepartitionExec: partitioning=Hash([o_custkey@3], 4), input_partitions=4
-21)----------------------------------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(l_orderkey@0, o_orderkey@0)], projection=[l_extendedprice@1, l_discount@2, s_nationkey@3, o_custkey@5, o_orderdate@6]
-22)------------------------------------------CoalesceBatchesExec: target_batch_size=8192
-23)--------------------------------------------RepartitionExec: partitioning=Hash([l_orderkey@0], 4), input_partitions=4
-24)----------------------------------------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(l_suppkey@1, s_suppkey@0)], projection=[l_orderkey@0, l_extendedprice@2, l_discount@3, s_nationkey@5]
-25)------------------------------------------------CoalesceBatchesExec: target_batch_size=8192
-26)--------------------------------------------------RepartitionExec: partitioning=Hash([l_suppkey@1], 4), input_partitions=4
-27)----------------------------------------------------HashJoinExec: mode=Partitioned, join_type=Inner, on=[(p_partkey@0, l_partkey@1)], projection=[l_orderkey@1, l_suppkey@3, l_extendedprice@4, l_discount@5]
-28)------------------------------------------------------CoalesceBatchesExec: target_batch_size=8192
-29)--------------------------------------------------------RepartitionExec: partitioning=Hash([p_partkey@0], 4), input_partitions=4
-30)----------------------------------------------------------FilterExec: p_type@1 = ECONOMY ANODIZED STEEL, projection=[p_partkey@0]
-31)------------------------------------------------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-32)--------------------------------------------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/part.tbl]]}, projection=[p_partkey, p_type], file_type=csv, has_header=false
-33)------------------------------------------------------CoalesceBatchesExec: target_batch_size=8192
-34)--------------------------------------------------------RepartitionExec: partitioning=Hash([l_partkey@1], 4), input_partitions=4
-35)----------------------------------------------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/lineitem.tbl:0..18561749], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/lineitem.tbl:18561749..37123498], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/lineitem.tbl:37123498..55685247], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/lineitem.tbl:55685247..74246996]]}, projection=[l_orderkey, l_partkey, l_suppkey, l_extendedprice, l_discount], file_type=csv, has_header=false
-36)------------------------------------------------CoalesceBatchesExec: target_batch_size=8192
-37)--------------------------------------------------RepartitionExec: partitioning=Hash([s_suppkey@0], 4), input_partitions=1
-38)----------------------------------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/supplier.tbl]]}, projection=[s_suppkey, s_nationkey], file_type=csv, has_header=false
-39)------------------------------------------CoalesceBatchesExec: target_batch_size=8192
-40)--------------------------------------------RepartitionExec: partitioning=Hash([o_orderkey@0], 4), input_partitions=4
-41)----------------------------------------------FilterExec: o_orderdate@2 >= 1995-01-01 AND o_orderdate@2 <= 1996-12-31
-42)------------------------------------------------DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/orders.tbl:0..4223281], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/orders.tbl:4223281..8446562], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/orders.tbl:8446562..12669843], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/orders.tbl:12669843..16893122]]}, projection=[o_orderkey, o_custkey, o_orderdate], file_type=csv, has_header=false
-43)------------------------------------CoalesceBatchesExec: target_batch_size=8192
-44)--------------------------------------RepartitionExec: partitioning=Hash([c_custkey@0], 4), input_partitions=1
-45)----------------------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/customer.tbl]]}, projection=[c_custkey, c_nationkey], file_type=csv, has_header=false
-46)------------------------------CoalesceBatchesExec: target_batch_size=8192
-47)--------------------------------RepartitionExec: partitioning=Hash([n_nationkey@0], 4), input_partitions=1
-48)----------------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/nation.tbl]]}, projection=[n_nationkey, n_regionkey], file_type=csv, has_header=false
-49)------------------------CoalesceBatchesExec: target_batch_size=8192
-50)--------------------------RepartitionExec: partitioning=Hash([n_nationkey@0], 4), input_partitions=1
-51)----------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/nation.tbl]]}, projection=[n_nationkey, n_name], file_type=csv, has_header=false
-52)------------------CoalesceBatchesExec: target_batch_size=8192
-53)--------------------RepartitionExec: partitioning=Hash([r_regionkey@0], 4), input_partitions=4
-54)----------------------FilterExec: r_name@1 = AMERICA, projection=[r_regionkey@0]
-55)------------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-56)--------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/region.tbl]]}, projection=[r_regionkey, r_name], file_type=csv, has_header=false
+05)--------RepartitionExec: partitioning=Hash([o_year@0], 4), input_partitions=4
Review Comment:
these complex query plans are so much easier to read now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3644318089
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2733.19 ms โ 2720.03 ms โ no change โ
โ QQuery 1 โ 1281.78 ms โ 1252.52 ms โ no change โ
โ QQuery 2 โ 2377.84 ms โ 2424.84 ms โ no change โ
โ QQuery 3 โ 1160.92 ms โ 1168.87 ms โ no change โ
โ QQuery 4 โ 2324.43 ms โ 2290.43 ms โ no change โ
โ QQuery 5 โ 28592.61 ms โ 28760.63 ms โ no change โ
โ QQuery 6 โ 4037.19 ms โ 3970.23 ms โ no change โ
โ QQuery 7 โ 3695.67 ms โ 3737.12 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 46203.63ms โ
โ Total Time (jizezhang_main) โ 46324.67ms โ
โ Average Time (HEAD) โ 5775.45ms โ
โ Average Time (jizezhang_main) โ 5790.58ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 8 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.57 ms โ 2.25 ms โ +1.14x faster โ
โ QQuery 1 โ 52.05 ms โ 52.40 ms โ no change โ
โ QQuery 2 โ 134.07 ms โ 134.10 ms โ no change โ
โ QQuery 3 โ 157.59 ms โ 154.23 ms โ no change โ
โ QQuery 4 โ 1177.17 ms โ 1092.07 ms โ +1.08x faster โ
โ QQuery 5 โ 1555.61 ms โ 1502.54 ms โ no change โ
โ QQuery 6 โ 2.14 ms โ 2.19 ms โ no change โ
โ QQuery 7 โ 56.24 ms โ 57.00 ms โ no change โ
โ QQuery 8 โ 1508.89 ms โ 1430.78 ms โ +1.05x faster โ
โ QQuery 9 โ 1970.06 ms โ 1881.53 ms โ no change โ
โ QQuery 10 โ 371.09 ms โ 374.63 ms โ no change โ
โ QQuery 11 โ 416.93 ms โ 423.24 ms โ no change โ
โ QQuery 12 โ 1440.40 ms โ 1365.87 ms โ +1.05x faster โ
โ QQuery 13 โ 2049.91 ms โ 2021.63 ms โ no change โ
โ QQuery 14 โ 1300.92 ms โ 1248.05 ms โ no change โ
โ QQuery 15 โ 1307.90 ms โ 1243.04 ms โ no change โ
โ QQuery 16 โ 2665.57 ms โ 2700.25 ms โ no change โ
โ QQuery 17 โ 2677.16 ms โ 2689.07 ms โ no change โ
โ QQuery 18 โ 5058.27 ms โ 5181.21 ms โ no change โ
โ QQuery 19 โ 126.21 ms โ 124.54 ms โ no change โ
โ QQuery 20 โ 1896.68 ms โ 1911.49 ms โ no change โ
โ QQuery 21 โ 2174.96 ms โ 2189.22 ms โ no change โ
โ QQuery 22 โ 3676.22 ms โ 3730.96 ms โ no change โ
โ QQuery 23 โ 12348.34 ms โ 12302.89 ms โ no change โ
โ QQuery 24 โ 208.97 ms โ 211.21 ms โ no change โ
โ QQuery 25 โ 464.76 ms โ 471.47 ms โ no change โ
โ QQuery 26 โ 220.82 ms โ 208.49 ms โ +1.06x faster โ
โ QQuery 27 โ 2708.31 ms โ 2710.37 ms โ no change โ
โ QQuery 28 โ 23987.85 ms โ 24389.78 ms โ no change โ
โ QQuery 29 โ 974.58 ms โ 940.27 ms โ no change โ
โ QQuery 30 โ 1352.37 ms โ 1309.12 ms โ no change โ
โ QQuery 31 โ 1365.80 ms โ 1327.44 ms โ no change โ
โ QQuery 32 โ 5194.56 ms โ 4840.81 ms โ +1.07x faster โ
โ QQuery 33 โ 5948.34 ms โ 5925.71 ms โ no change โ
โ QQuery 34 โ 6060.74 ms โ 6119.94 ms โ no change โ
โ QQuery 35 โ 2005.86 ms โ 1924.80 ms โ no change โ
โ QQuery 36 โ 118.66 ms โ 117.42 ms โ no change โ
โ QQuery 37 โ 54.36 ms โ 55.94 ms โ no change โ
โ QQuery 38 โ 117.99 ms โ 116.94 ms โ no change โ
โ QQuery 39 โ 188.85 ms โ 196.87 ms โ no change โ
โ QQuery 40 โ 46.58 ms โ 43.80 ms โ +1.06x faster โ
โ QQuery 41 โ 40.04 ms โ 41.09 ms โ no change โ
โ QQuery 42 โ 34.13 ms โ 33.88 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 95220.52ms โ
โ Total Time (jizezhang_main) โ 94800.51ms โ
โ Average Time (HEAD) โ 2214.43ms โ
โ Average Time (jizezhang_main) โ 2204.66ms โ
โ Queries Faster โ 7 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 36 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 129.27 ms โ 132.98 ms โ no change โ
โ QQuery 2 โ 27.66 ms โ 27.10 ms โ no change โ
โ QQuery 3 โ 37.19 ms โ 37.92 ms โ no change โ
โ QQuery 4 โ 28.76 ms โ 28.28 ms โ no change โ
โ QQuery 5 โ 87.45 ms โ 86.95 ms โ no change โ
โ QQuery 6 โ 19.85 ms โ 19.87 ms โ no change โ
โ QQuery 7 โ 227.31 ms โ 227.85 ms โ no change โ
โ QQuery 8 โ 36.11 ms โ 35.44 ms โ no change โ
โ QQuery 9 โ 106.23 ms โ 101.00 ms โ no change โ
โ QQuery 10 โ 63.19 ms โ 63.64 ms โ no change โ
โ QQuery 11 โ 17.70 ms โ 18.54 ms โ no change โ
โ QQuery 12 โ 52.76 ms โ 52.15 ms โ no change โ
โ QQuery 13 โ 47.73 ms โ 48.17 ms โ no change โ
โ QQuery 14 โ 13.77 ms โ 14.03 ms โ no change โ
โ QQuery 15 โ 24.71 ms โ 24.06 ms โ no change โ
โ QQuery 16 โ 25.16 ms โ 24.86 ms โ no change โ
โ QQuery 17 โ 150.79 ms โ 150.25 ms โ no change โ
โ QQuery 18 โ 279.74 ms โ 278.43 ms โ no change โ
โ QQuery 19 โ 37.56 ms โ 37.52 ms โ no change โ
โ QQuery 20 โ 49.26 ms โ 48.52 ms โ no change โ
โ QQuery 21 โ 313.83 ms โ 322.16 ms โ no change โ
โ QQuery 22 โ 17.49 ms โ 17.33 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 1793.52ms โ
โ Total Time (jizezhang_main) โ 1797.05ms โ
โ Average Time (HEAD) โ 81.52ms โ
โ Average Time (jizezhang_main) โ 81.68ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 22 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3644318338
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and jizezhang_main
--------------------
Benchmark tpch_mem_sf10.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ FAIL โ FAIL โ incomparable โ
โ QQuery 2 โ FAIL โ FAIL โ incomparable โ
โ QQuery 3 โ FAIL โ FAIL โ incomparable โ
โ QQuery 4 โ FAIL โ FAIL โ incomparable โ
โ QQuery 5 โ FAIL โ FAIL โ incomparable โ
โ QQuery 6 โ FAIL โ FAIL โ incomparable โ
โ QQuery 7 โ FAIL โ FAIL โ incomparable โ
โ QQuery 8 โ FAIL โ FAIL โ incomparable โ
โ QQuery 9 โ FAIL โ FAIL โ incomparable โ
โ QQuery 10 โ FAIL โ FAIL โ incomparable โ
โ QQuery 11 โ FAIL โ FAIL โ incomparable โ
โ QQuery 12 โ FAIL โ FAIL โ incomparable โ
โ QQuery 13 โ FAIL โ FAIL โ incomparable โ
โ QQuery 14 โ FAIL โ FAIL โ incomparable โ
โ QQuery 15 โ FAIL โ FAIL โ incomparable โ
โ QQuery 16 โ FAIL โ FAIL โ incomparable โ
โ QQuery 17 โ FAIL โ FAIL โ incomparable โ
โ QQuery 18 โ FAIL โ FAIL โ incomparable โ
โ QQuery 19 โ FAIL โ FAIL โ incomparable โ
โ QQuery 20 โ FAIL โ FAIL โ incomparable โ
โ QQuery 21 โ FAIL โ FAIL โ incomparable โ
โ QQuery 22 โ FAIL โ FAIL โ incomparable โ
โโโโโโโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 0.00ms โ
โ Total Time (jizezhang_main) โ 0.00ms โ
โ Average Time (HEAD) โ 0.00ms โ
โ Average Time (jizezhang_main) โ 0.00ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 0 โ
โ Queries with Failure โ 22 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3644211040
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (446f6cf4e066fe43df888ff93258758550a34143) to 20870c18a418ec081d44ecf8a90a30a95aa53138 [diff](https://github.com/apache/datafusion/compare/20870c18a418ec081d44ecf8a90a30a95aa53138..446f6cf4e066fe43df888ff93258758550a34143) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] remove repartition exec from coalesce batches optimizer [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19239:
URL: https://github.com/apache/datafusion/pull/19239#issuecomment-3655611206
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ jizezhang_main โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.12 ms โ 2.20 ms โ no change โ
โ QQuery 1 โ 49.90 ms โ 50.17 ms โ no change โ
โ QQuery 2 โ 135.88 ms โ 135.00 ms โ no change โ
โ QQuery 3 โ 153.88 ms โ 156.47 ms โ no change โ
โ QQuery 4 โ 1090.93 ms โ 1086.08 ms โ no change โ
โ QQuery 5 โ 1433.46 ms โ 1467.81 ms โ no change โ
โ QQuery 6 โ 2.20 ms โ 2.22 ms โ no change โ
โ QQuery 7 โ 55.37 ms โ 54.06 ms โ no change โ
โ QQuery 8 โ 1426.54 ms โ 1421.77 ms โ no change โ
โ QQuery 9 โ 1824.69 ms โ 1827.51 ms โ no change โ
โ QQuery 10 โ 354.93 ms โ 360.16 ms โ no change โ
โ QQuery 11 โ 416.23 ms โ 414.75 ms โ no change โ
โ QQuery 12 โ 1315.38 ms โ 1327.46 ms โ no change โ
โ QQuery 13 โ 2030.94 ms โ 2032.11 ms โ no change โ
โ QQuery 14 โ 1236.37 ms โ 1244.11 ms โ no change โ
โ QQuery 15 โ 1248.80 ms โ 1227.81 ms โ no change โ
โ QQuery 16 โ 2639.01 ms โ 2661.50 ms โ no change โ
โ QQuery 17 โ 2623.62 ms โ 2648.13 ms โ no change โ
โ QQuery 18 โ 5802.31 ms โ 5078.57 ms โ +1.14x faster โ
โ QQuery 19 โ 122.89 ms โ 125.37 ms โ no change โ
โ QQuery 20 โ 1925.23 ms โ 1816.50 ms โ +1.06x faster โ
โ QQuery 21 โ 2183.29 ms โ 2184.08 ms โ no change โ
โ QQuery 22 โ 3762.73 ms โ 3730.48 ms โ no change โ
โ QQuery 23 โ 18873.98 ms โ 12326.09 ms โ +1.53x faster โ
โ QQuery 24 โ 219.05 ms โ 223.43 ms โ no change โ
โ QQuery 25 โ 461.98 ms โ 455.19 ms โ no change โ
โ QQuery 26 โ 223.39 ms โ 218.96 ms โ no change โ
โ QQuery 27 โ 2765.04 ms โ 2686.97 ms โ no change โ
โ QQuery 28 โ 23681.66 ms โ 23350.57 ms โ no change โ
โ QQuery 29 โ 987.66 ms โ 962.29 ms โ no change โ
โ QQuery 30 โ 1313.97 ms โ 1298.99 ms โ no change โ
โ QQuery 31 โ 1326.60 ms โ 1344.03 ms โ no change โ
โ QQuery 32 โ 4965.74 ms โ 5040.46 ms โ no change โ
โ QQuery 33 โ 5944.09 ms โ 5822.75 ms โ no change โ
โ QQuery 34 โ 5960.38 ms โ 5866.65 ms โ no change โ
โ QQuery 35 โ 1931.03 ms โ 1890.30 ms โ no change โ
โ QQuery 36 โ 116.81 ms โ 118.06 ms โ no change โ
โ QQuery 37 โ 54.45 ms โ 55.22 ms โ no change โ
โ QQuery 38 โ 115.15 ms โ 117.00 ms โ no change โ
โ QQuery 39 โ 186.79 ms โ 188.61 ms โ no change โ
โ QQuery 40 โ 46.25 ms โ 45.71 ms โ no change โ
โ QQuery 41 โ 41.04 ms โ 41.15 ms โ no change โ
โ QQuery 42 โ 35.18 ms โ 34.14 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 101086.95ms โ
โ Total Time (jizezhang_main) โ 93140.89ms โ
โ Average Time (HEAD) โ 2350.86ms โ
โ Average Time (jizezhang_main) โ 2166.07ms โ
โ Queries Faster โ 3 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 40 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org