You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/12/16 23:32:49 UTC
[PR] allow expressions to pass through RepartitionExec [datafusion]
adriangb opened a new pull request, #19365:
URL: https://github.com/apache/datafusion/pull/19365
I don't see any reason for these limitations.
This behavior was introduced in #14235, @buraksenn @berkaysynnada if you have any context please share.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed pull request #19365: allow expressions to pass through RepartitionExec
URL: https://github.com/apache/datafusion/pull/19365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3666266507
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3666498250
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and repartition-exec-projection-swap
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ repartition-exec-projection-swap โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2840.48 ms โ 2686.98 ms โ +1.06x faster โ
โ QQuery 1 โ 1239.76 ms โ 1180.19 ms โ no change โ
โ QQuery 2 โ 2337.11 ms โ 2235.83 ms โ no change โ
โ QQuery 3 โ 1157.14 ms โ 1141.77 ms โ no change โ
โ QQuery 4 โ 2321.77 ms โ 2278.91 ms โ no change โ
โ QQuery 5 โ 29088.66 ms โ 28690.61 ms โ no change โ
โ QQuery 6 โ 4028.89 ms โ 4025.35 ms โ no change โ
โ QQuery 7 โ 3622.20 ms โ 3550.95 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 46636.01ms โ
โ Total Time (repartition-exec-projection-swap) โ 45790.59ms โ
โ Average Time (HEAD) โ 5829.50ms โ
โ Average Time (repartition-exec-projection-swap) โ 5723.82ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 7 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ repartition-exec-projection-swap โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.61 ms โ 2.56 ms โ no change โ
โ QQuery 1 โ 51.81 ms โ 51.42 ms โ no change โ
โ QQuery 2 โ 135.08 ms โ 135.85 ms โ no change โ
โ QQuery 3 โ 156.96 ms โ 159.16 ms โ no change โ
โ QQuery 4 โ 1091.22 ms โ 1227.73 ms โ 1.13x slower โ
โ QQuery 5 โ 1491.44 ms โ 1583.53 ms โ 1.06x slower โ
โ QQuery 6 โ 2.03 ms โ 2.26 ms โ 1.11x slower โ
โ QQuery 7 โ 56.91 ms โ 61.22 ms โ 1.08x slower โ
โ QQuery 8 โ 1469.91 ms โ 1544.11 ms โ 1.05x slower โ
โ QQuery 9 โ 1870.62 ms โ 1973.73 ms โ 1.06x slower โ
โ QQuery 10 โ 372.86 ms โ 372.76 ms โ no change โ
โ QQuery 11 โ 417.83 ms โ 429.44 ms โ no change โ
โ QQuery 12 โ 1366.19 ms โ 1511.92 ms โ 1.11x slower โ
โ QQuery 13 โ 2039.67 ms โ 2131.77 ms โ no change โ
โ QQuery 14 โ 1270.21 ms โ 1337.92 ms โ 1.05x slower โ
โ QQuery 15 โ 1216.67 ms โ 1349.11 ms โ 1.11x slower โ
โ QQuery 16 โ 2645.72 ms โ 2785.32 ms โ 1.05x slower โ
โ QQuery 17 โ 2626.45 ms โ 2766.10 ms โ 1.05x slower โ
โ QQuery 18 โ 5567.30 ms โ 5054.93 ms โ +1.10x faster โ
โ QQuery 19 โ 129.28 ms โ 127.67 ms โ no change โ
โ QQuery 20 โ 1963.74 ms โ 1922.43 ms โ no change โ
โ QQuery 21 โ 2264.13 ms โ 2215.85 ms โ no change โ
โ QQuery 22 โ 4173.48 ms โ 3788.04 ms โ +1.10x faster โ
โ QQuery 23 โ 19389.31 ms โ 12417.19 ms โ +1.56x faster โ
โ QQuery 24 โ 230.44 ms โ 224.62 ms โ no change โ
โ QQuery 25 โ 494.03 ms โ 457.99 ms โ +1.08x faster โ
โ QQuery 26 โ 227.41 ms โ 214.61 ms โ +1.06x faster โ
โ QQuery 27 โ 2848.47 ms โ 2791.87 ms โ no change โ
โ QQuery 28 โ 23844.65 ms โ 23548.77 ms โ no change โ
โ QQuery 29 โ 969.27 ms โ 944.55 ms โ no change โ
โ QQuery 30 โ 1342.46 ms โ 1381.74 ms โ no change โ
โ QQuery 31 โ 1408.55 ms โ 1360.04 ms โ no change โ
โ QQuery 32 โ 5005.00 ms โ 4745.24 ms โ +1.05x faster โ
โ QQuery 33 โ 6055.20 ms โ 5930.96 ms โ no change โ
โ QQuery 34 โ 6123.53 ms โ 6267.73 ms โ no change โ
โ QQuery 35 โ 2009.27 ms โ 2008.01 ms โ no change โ
โ QQuery 36 โ 65.69 ms โ 69.25 ms โ 1.05x slower โ
โ QQuery 37 โ 46.99 ms โ 45.49 ms โ no change โ
โ QQuery 38 โ 67.15 ms โ 65.93 ms โ no change โ
โ QQuery 39 โ 103.02 ms โ 108.67 ms โ 1.05x slower โ
โ QQuery 40 โ 28.01 ms โ 28.54 ms โ no change โ
โ QQuery 41 โ 23.14 ms โ 25.48 ms โ 1.10x slower โ
โ QQuery 42 โ 20.25 ms โ 20.72 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 102683.97ms โ
โ Total Time (repartition-exec-projection-swap) โ 95192.23ms โ
โ Average Time (HEAD) โ 2388.00ms โ
โ Average Time (repartition-exec-projection-swap) โ 2213.77ms โ
โ Queries Faster โ 6 โ
โ Queries Slower โ 14 โ
โ Queries with No Change โ 23 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ repartition-exec-projection-swap โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 132.92 ms โ 131.11 ms โ no change โ
โ QQuery 2 โ 30.03 ms โ 30.48 ms โ no change โ
โ QQuery 3 โ 40.03 ms โ 36.96 ms โ +1.08x faster โ
โ QQuery 4 โ 29.50 ms โ 29.18 ms โ no change โ
โ QQuery 5 โ 89.10 ms โ 90.10 ms โ no change โ
โ QQuery 6 โ 20.13 ms โ 20.04 ms โ no change โ
โ QQuery 7 โ 236.03 ms โ 240.67 ms โ no change โ
โ QQuery 8 โ 38.93 ms โ 38.79 ms โ no change โ
โ QQuery 9 โ 100.04 ms โ 106.81 ms โ 1.07x slower โ
โ QQuery 10 โ 65.91 ms โ 64.84 ms โ no change โ
โ QQuery 11 โ 17.47 ms โ 19.38 ms โ 1.11x slower โ
โ QQuery 12 โ 53.68 ms โ 52.42 ms โ no change โ
โ QQuery 13 โ 49.84 ms โ 51.77 ms โ no change โ
โ QQuery 14 โ 15.28 ms โ 13.99 ms โ +1.09x faster โ
โ QQuery 15 โ 25.01 ms โ 23.77 ms โ no change โ
โ QQuery 16 โ 24.79 ms โ 24.82 ms โ no change โ
โ QQuery 17 โ 155.65 ms โ 159.10 ms โ no change โ
โ QQuery 18 โ 292.45 ms โ 288.87 ms โ no change โ
โ QQuery 19 โ 37.42 ms โ 38.37 ms โ no change โ
โ QQuery 20 โ 50.51 ms โ 49.97 ms โ no change โ
โ QQuery 21 โ 328.39 ms โ 326.80 ms โ no change โ
โ QQuery 22 โ 18.17 ms โ 17.68 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 1851.28ms โ
โ Total Time (repartition-exec-projection-swap) โ 1855.91ms โ
โ Average Time (HEAD) โ 84.15ms โ
โ Average Time (repartition-exec-projection-swap) โ 84.36ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 18 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3667451801
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3667595763
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing repartition-exec-projection-swap (7759d7b30c15f4758b6174f2d8bd1c6f1dbb702d) to 2b67ba81c60690d045ddd71910dd92810704f40e [diff](https://github.com/apache/datafusion/compare/2b67ba81c60690d045ddd71910dd92810704f40e..7759d7b30c15f4758b6174f2d8bd1c6f1dbb702d) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#discussion_r2625923713
##########
datafusion/physical-plan/src/repartition/mod.rs:
##########
@@ -1042,18 +1042,6 @@ impl ExecutionPlan for RepartitionExec {
&self,
projection: &ProjectionExec,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
- // If the projection does not narrow the schema, we should not try to push it down.
Review Comment:
I think this still shows up in the plans, in some cases the number of fields is bigger before repartitition because of this (and might hurt performance as `RepartitionExec` copies the input columns (currently twice)). Can we instead relax the conditition to `>` instead of `>=` so it will be pushed down if it also keeps the number of fields equal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3666301214
We discussed in the meeting today.
The conclusion was basically that the heuristic that is currently in place is trying to answer the question "is this expression more or less expensive to evaluate than it's constituent parts".
For example, if the projection *adds* columns `a,b,a+b` that is true, if it removes columns `a,b` instead of `a,b,c` it's not.
However this is broken for things like struct field access or parquet variant field access where the expression `large_struct.small_int_field` might be considerably cheaper to evaluate than it's constituent parts (i.e. reading a single field from a struct is cheaper than reading the whole thing and then slicing it).
The most promising approach we discussed was adding a method to PhysicalExpr to let the expression determine if it is cheaper or more expensive than it's constituent parts.
There are other options, I'll open an issue to discuss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3666268278
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing repartition-exec-projection-swap (7759d7b30c15f4758b6174f2d8bd1c6f1dbb702d) to 2b67ba81c60690d045ddd71910dd92810704f40e [diff](https://github.com/apache/datafusion/compare/2b67ba81c60690d045ddd71910dd92810704f40e..7759d7b30c15f4758b6174f2d8bd1c6f1dbb702d) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3806482894
Closing in favor of #19538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#issuecomment-3667674506
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and repartition-exec-projection-swap
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ repartition-exec-projection-swap โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2754.37 ms โ 2763.91 ms โ no change โ
โ QQuery 1 โ 1224.37 ms โ 1222.34 ms โ no change โ
โ QQuery 2 โ 2327.60 ms โ 2364.70 ms โ no change โ
โ QQuery 3 โ 1199.44 ms โ 1159.69 ms โ no change โ
โ QQuery 4 โ 2283.38 ms โ 2314.42 ms โ no change โ
โ QQuery 5 โ 29261.13 ms โ 29361.39 ms โ no change โ
โ QQuery 6 โ 4112.78 ms โ 4036.08 ms โ no change โ
โ QQuery 7 โ 3931.12 ms โ 3731.86 ms โ +1.05x faster โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 47094.18ms โ
โ Total Time (repartition-exec-projection-swap) โ 46954.39ms โ
โ Average Time (HEAD) โ 5886.77ms โ
โ Average Time (repartition-exec-projection-swap) โ 5869.30ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 7 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ repartition-exec-projection-swap โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.23 ms โ 2.38 ms โ 1.07x slower โ
โ QQuery 1 โ 50.94 ms โ 51.32 ms โ no change โ
โ QQuery 2 โ 138.30 ms โ 138.20 ms โ no change โ
โ QQuery 3 โ 155.84 ms โ 153.73 ms โ no change โ
โ QQuery 4 โ 1149.65 ms โ 1148.51 ms โ no change โ
โ QQuery 5 โ 1505.46 ms โ 1533.06 ms โ no change โ
โ QQuery 6 โ 2.10 ms โ 2.05 ms โ no change โ
โ QQuery 7 โ 57.24 ms โ 59.55 ms โ no change โ
โ QQuery 8 โ 1475.04 ms โ 1469.12 ms โ no change โ
โ QQuery 9 โ 1898.74 ms โ 1958.68 ms โ no change โ
โ QQuery 10 โ 372.36 ms โ 371.88 ms โ no change โ
โ QQuery 11 โ 428.42 ms โ 427.50 ms โ no change โ
โ QQuery 12 โ 1375.71 ms โ 1379.38 ms โ no change โ
โ QQuery 13 โ 2045.25 ms โ 2049.84 ms โ no change โ
โ QQuery 14 โ 1272.71 ms โ 1294.44 ms โ no change โ
โ QQuery 15 โ 1280.91 ms โ 1280.95 ms โ no change โ
โ QQuery 16 โ 2726.83 ms โ 2681.93 ms โ no change โ
โ QQuery 17 โ 2699.63 ms โ 2736.94 ms โ no change โ
โ QQuery 18 โ 5619.65 ms โ 4981.69 ms โ +1.13x faster โ
โ QQuery 19 โ 123.66 ms โ 122.55 ms โ no change โ
โ QQuery 20 โ 1913.69 ms โ 1918.62 ms โ no change โ
โ QQuery 21 โ 2230.53 ms โ 2210.76 ms โ no change โ
โ QQuery 22 โ 3803.98 ms โ 3777.10 ms โ no change โ
โ QQuery 23 โ 12557.92 ms โ 12193.07 ms โ no change โ
โ QQuery 24 โ 213.81 ms โ 208.52 ms โ no change โ
โ QQuery 25 โ 474.44 ms โ 480.03 ms โ no change โ
โ QQuery 26 โ 241.63 ms โ 227.73 ms โ +1.06x faster โ
โ QQuery 27 โ 2798.44 ms โ 2759.81 ms โ no change โ
โ QQuery 28 โ 23685.63 ms โ 23708.62 ms โ no change โ
โ QQuery 29 โ 972.25 ms โ 976.77 ms โ no change โ
โ QQuery 30 โ 1361.57 ms โ 1319.12 ms โ no change โ
โ QQuery 31 โ 1365.84 ms โ 1341.59 ms โ no change โ
โ QQuery 32 โ 5462.39 ms โ 5324.64 ms โ no change โ
โ QQuery 33 โ 6240.03 ms โ 5906.89 ms โ +1.06x faster โ
โ QQuery 34 โ 6318.35 ms โ 6292.16 ms โ no change โ
โ QQuery 35 โ 1987.13 ms โ 1978.48 ms โ no change โ
โ QQuery 36 โ 67.65 ms โ 68.71 ms โ no change โ
โ QQuery 37 โ 48.26 ms โ 46.95 ms โ no change โ
โ QQuery 38 โ 68.72 ms โ 67.46 ms โ no change โ
โ QQuery 39 โ 105.82 ms โ 109.17 ms โ no change โ
โ QQuery 40 โ 27.72 ms โ 26.79 ms โ no change โ
โ QQuery 41 โ 24.43 ms โ 24.82 ms โ no change โ
โ QQuery 42 โ 21.18 ms โ 21.05 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 96372.06ms โ
โ Total Time (repartition-exec-projection-swap) โ 94832.54ms โ
โ Average Time (HEAD) โ 2241.21ms โ
โ Average Time (repartition-exec-projection-swap) โ 2205.41ms โ
โ Queries Faster โ 3 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 39 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ repartition-exec-projection-swap โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 134.45 ms โ 133.43 ms โ no change โ
โ QQuery 2 โ 29.84 ms โ 29.64 ms โ no change โ
โ QQuery 3 โ 39.41 ms โ 40.53 ms โ no change โ
โ QQuery 4 โ 29.94 ms โ 30.21 ms โ no change โ
โ QQuery 5 โ 89.62 ms โ 91.65 ms โ no change โ
โ QQuery 6 โ 19.88 ms โ 19.95 ms โ no change โ
โ QQuery 7 โ 238.36 ms โ 256.37 ms โ 1.08x slower โ
โ QQuery 8 โ 37.29 ms โ 36.29 ms โ no change โ
โ QQuery 9 โ 108.42 ms โ 111.93 ms โ no change โ
โ QQuery 10 โ 68.09 ms โ 64.47 ms โ +1.06x faster โ
โ QQuery 11 โ 19.79 ms โ 17.26 ms โ +1.15x faster โ
โ QQuery 12 โ 52.99 ms โ 53.36 ms โ no change โ
โ QQuery 13 โ 49.33 ms โ 49.40 ms โ no change โ
โ QQuery 14 โ 14.22 ms โ 14.57 ms โ no change โ
โ QQuery 15 โ 24.81 ms โ 25.25 ms โ no change โ
โ QQuery 16 โ 25.28 ms โ 25.69 ms โ no change โ
โ QQuery 17 โ 156.12 ms โ 160.40 ms โ no change โ
โ QQuery 18 โ 291.56 ms โ 294.02 ms โ no change โ
โ QQuery 19 โ 38.85 ms โ 38.34 ms โ no change โ
โ QQuery 20 โ 52.13 ms โ 50.93 ms โ no change โ
โ QQuery 21 โ 341.07 ms โ 334.08 ms โ no change โ
โ QQuery 22 โ 17.39 ms โ 17.89 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 1878.83ms โ
โ Total Time (repartition-exec-projection-swap) โ 1895.67ms โ
โ Average Time (HEAD) โ 85.40ms โ
โ Average Time (repartition-exec-projection-swap) โ 86.17ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 19 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] allow expressions to pass through RepartitionExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #19365:
URL: https://github.com/apache/datafusion/pull/19365#discussion_r2625923713
##########
datafusion/physical-plan/src/repartition/mod.rs:
##########
@@ -1042,18 +1042,6 @@ impl ExecutionPlan for RepartitionExec {
&self,
projection: &ProjectionExec,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
- // If the projection does not narrow the schema, we should not try to push it down.
Review Comment:
I think this still shows up in the plans, in some cases the number of fields is bigger before repartitition because of this (and might hurt performance as repartition copies the input columns). Can we instead relax the conditition to `>` instead of `>=` so it will be pushed down if it also keeps the number of fields equal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org