You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NGA-TRAN (via GitHub)" <gi...@apache.org> on 2024/04/26 20:03:41 UTC
[I] Add an option to avoid grouping hash-partitioning [datafusion]
NGA-TRAN opened a new issue, #10257:
URL: https://github.com/apache/datafusion/issues/10257
### Is your feature request related to a problem or challenge?
We have run into a an issue in IOx and described [here](https://github.com/influxdata/arrow-datafusion/pull/4) that the `Union` is converted to `Interleave` if their input data can interleave. The Interleave's job is to send their corresponding partitions to the right output. If I understand correctly its purposed is to keep data grouped in their same partitions which will be useful if the operators down stream want data in that shape. As the result, it is not useful to push sort down because we do not need/want data sorted in that case.
However, for our IOx use case, even though the input data to Union can be interleave, a `Projection` above that (See [the plan here ](https://github.com/influxdata/arrow-datafusion/pull/4#issue-2223275385) adds different constants ("m0" and "m1" in the example) and if we add that constant as a column into the output, data no longer interleave even though their `output_partitioning` says the do. Further more, we do want those data to get sorted and hence need the sort-push-down.
With the opposite needed described in 2 paragraphs above, converting Union to Interleave is not always needed even if data interleave. This ticket is a request to avoid that from happening.
### Describe the solution you'd like
After chatting with @alamb we propose to add an `option` into the config to tell `enforce_distribution` no to use Interleave [at this step](https://github.com/apache/datafusion/blob/f8c623fe045d70a87eac8dc8620b74ff73be56d5/datafusion/core/src/physical_optimizer/enforce_distribution.rs#L1195)
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add an option to avoid grouping hash-partitioning [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10257: Add an option to avoid grouping hash-partitioning
URL: https://github.com/apache/datafusion/issues/10257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add an option to avoid grouping hash-partitioning [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #10257:
URL: https://github.com/apache/datafusion/issues/10257#issuecomment-2080046600
@alamb I am working on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add an option to avoid grouping hash-partitioning [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10257:
URL: https://github.com/apache/datafusion/issues/10257#issuecomment-2085793500
Closed in https://github.com/apache/datafusion/pull/10259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org