You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/07/29 14:08:06 UTC
[I] Support RangePartitioning with native shuffle [datafusion-comet]
andygrove opened a new issue, #458:
URL: https://github.com/apache/datafusion-comet/issues/458
### What is the problem the feature request solves?
We do not currently support RangePartitioning with native shuffle.
Adding this support would allow us to use native shuffle for more queries, including queries where there is a global sort.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-2762956253
> Hi @andygrove
> May I ask why we decide not support RangePartitioning ? and will it be supported in the near future ?
> Thanks
We didn't decide not to support it.
It just hasn't been implemented yet because it is more complex than other partitioning schemes. It would be great if someone could submit a PR to add support for this. It could really help with benchmark results and performance overall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-2125116224
Since the early development of Comet shuffle, this is one item on to do list. After we finish columnar shuffle which has higher coverage and support main partitioning types, we turn our focus to other more urgent items. Because we want to increase query coverage as much as possible. Currently we can run TPC-H queries mostly native, seems we can go back to look at this as it can give us more performance gain on shuffle (because native shuffle is faster than columnar shuffle).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #458: Support RangePartitioning with native shuffle
URL: https://github.com/apache/datafusion-comet/issues/458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-3825887950
Implemented in #1862, with follow-up fixes in #1907, #2258, and #2716. Closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-2763002946
The implementation issue or difference for RangePartitioning other than other partitioning like HashPartitioning, is that it involves some sampling operations that perform with RDD in Spark. That's said Comet lib at each executor cannot determine the sampling results separately like we did for general query execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #458: Support RangePartitioning with native shuffle
URL: https://github.com/apache/datafusion-comet/issues/458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "jinwenjie123 (via GitHub)" <gi...@apache.org>.
jinwenjie123 commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-2762497216
Hi @andygrove
May I ask why we decide not support RangePartitioning ? and will it be supported in the near future ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-2762962087
I discussed this feature with @mbutrovich recently and he may have additional thoughts on this topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support RangePartitioning with native shuffle [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #458:
URL: https://github.com/apache/datafusion-comet/issues/458#issuecomment-2125120170
I will work on this after finishing HashJoin build right support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org