You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/18 12:26:45 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #4968: Don't repartition ProjectionExec when it does not compute anything
alamb opened a new issue, #4968:
URL: https://github.com/apache/arrow-datafusion/issues/4968
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
`ProjectionExec` can either have computations like (`col1` + `col2`) or it can be used to reorder / rename the columns
The first use case benefits from repartitioning (as then the calculation can be done in multiple cores)
The second use case (ordering) does not benefit from partitioning as it is simply a bookkeeping arrangement
Basically we have a plan like
```text
ProjectionExec: expr=[f@0 as f]
DeduplicateExec: [tag@1 ASC,time@2 ASC]
SortPreservingMergeExec: [tag@1 ASC,time@2 ASC]
UnionExec
```
That is then optimized by https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/physical_optimizer/repartition.rs to repartition before the projection
```text
ProjectionExec: expr=[f@0 as f]
RepartitionExec: partitioning=RoundRobinBatch(4) <-- This repartition node is likely worthless
DeduplicateExec: [tag@1 ASC,time@2 ASC]
SortPreservingMergeExec: [tag@1 ASC,time@2 ASC]
UnionExec
```
**Describe the solution you'd like**
This I think ProjectionExec should only "benefit from partitioning" when its partition expressions actually have calculations (aka are not just columns / aliases)
This would like defining `benefits_from_input_partitioning`
https://github.com/apache/arrow-datafusion/blob/906896b7c59ff14d71b3056ec4349274cf6662af/datafusion/core/src/physical_plan/mod.rs#L176-L183
For `impl ExecutionPlan for ProjectionExec`: https://github.com/apache/arrow-datafusion/blob/906896b7c59ff14d71b3056ec4349274cf6662af/datafusion/core/src/physical_plan/projection.rs#L151
So that it returned true only if there were expressions that had non column references / aliases
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
I think this is a good first issue as the code and desire is fairly straightforward and this would largely be an exercise in updating tests I suspect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #4968: Don't repartition ProjectionExec when it does not compute anything
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #4968: Don't repartition ProjectionExec when it does not compute anything
URL: https://github.com/apache/arrow-datafusion/issues/4968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xiaoyong-z commented on issue #4968: Don't repartition ProjectionExec when it does not compute anything
Posted by GitBox <gi...@apache.org>.
xiaoyong-z commented on issue #4968:
URL: https://github.com/apache/arrow-datafusion/issues/4968#issuecomment-1398079543
I'm intersted on this issue, please assign it to me if no one else is already working on it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #4968: Don't repartition ProjectionExec when it does not compute anything
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #4968:
URL: https://github.com/apache/arrow-datafusion/issues/4968#issuecomment-1398216418
Thanks @xiaoyong-z -- I don't know anyone else working on it. I have assigned it to you
A good way to start might be to make the change to ProjectionExec and see what tests in `repartition` fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org