You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jonahgao (via GitHub)" <gi...@apache.org> on 2024/07/18 15:29:22 UTC
[PR] fix: fall back to `UnionExec` if can't interleave [datafusion]
jonahgao opened a new pull request, #11536:
URL: https://github.com/apache/datafusion/pull/11536
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #11409.
## Rationale for this change
After certain optimizations such as `ProjectionPushdown`, the children of `InterleaveExec` can no longer meet the interleave [condition](https://github.com/apache/datafusion/blob/b19744968770c4ab426d065dec3cc5147534e87a/datafusion/physical-plan/src/union.rs#L363) because their output_partitioning becomes inconsistent.
Perhaps in this situation, we could consider falling back to `UnionExec` from `InterleaveExec` instead of throwing an error.
For example, in the issue #11409
**Before** optimization :
```sh
# InterleaveExec's children[0]
ProjectionExec: expr=[v2@2 as v2, v0@1 as v0], OutPartitioning=Hash([UnKnownColumn { name: "v0@0" }], 2)
ProjectionExec: expr=[v0@0 as v0, v0@1 as v0, v2@2 as v2], OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(v0@0, CAST(t1.v0 AS Float64)@2)], OutPartitioning=Hash([Column { name: "CAST(t1.v0 AS Float64)", index: 3 }], 2)
RepartitionExec: partitioning=Hash([v0@0], 2), input_partitions=1, OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
MemoryExec: partitions=1, partition_sizes=[1], OutPartitioning=UnknownPartitioning(1)
RepartitionExec: partitioning=Hash([CAST(t1.v0 AS Float64)@2], 2), input_partitions=1, OutPartitioning=Hash([Column { name: "CAST(t1.v0 AS Float64)", index: 2 }], 2)
ProjectionExec: expr=[v0@0 as v0, v2@1 as v2, CAST(v0@0 AS Float64) as CAST(t1.v0 AS Float64)], OutPartitioning=UnknownPartitioning(1)
MemoryExec: partitions=1, partition_sizes=[4], OutPartitioning=UnknownPartitioning(1)
# InterleaveExec's children[1]
ProjectionExec: expr=[v2@2 as v2, v0@1 as v0], OutPartitioning=Hash([UnKnownColumn { name: "v0@0" }], 2)
ProjectionExec: expr=[v0@0 as v0, v0@1 as v0, v2@2 as v2], OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
ProjectionExec: expr=[v0@3 as v0, v0@0 as v0, v2@1 as v2, CAST(t1.v0 AS Float64)@2 as CAST(t1.v0 AS Float64)], OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(CAST(t1.v0 AS Float64)@2, v0@0)], OutPartitioning=Hash([Column { name: "v0", index: 3 }], 2)
RepartitionExec: partitioning=Hash([CAST(t1.v0 AS Float64)@2], 2), input_partitions=2, OutPartitioning=Hash([Column { name: "CAST(t1.v0 AS Float64)", index: 2 }], 2)
ProjectionExec: expr=[v0@0 as v0, v2@1 as v2, CAST(v0@0 AS Float64) as CAST(t1.v0 AS Float64)], OutPartitioning=RoundRobinBatch(2)
RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1, OutPartitioning=RoundRobinBatch(2)
FilterExec: v2@1 IS NULL, OutPartitioning=UnknownPartitioning(1)
MemoryExec: partitions=1, partition_sizes=[4], OutPartitioning=UnknownPartitioning(1)
RepartitionExec: partitioning=Hash([v0@0], 2), input_partitions=1, OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
MemoryExec: partitions=1, partition_sizes=[1], OutPartitioning=UnknownPartitioning(1)
```
**After:**
```sh
# InterleaveExec's children[0] OutputPartitioning: Hash([UnKnownColumn { name: "CAST(t1.v0 AS Float64)@3" }], 2)
ProjectionExec: expr=[v2@1 as v2, v0@0 as v0], OutPartitioning=Hash([UnKnownColumn { name: "CAST(t1.v0 AS Float64)@3" }], 2)
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(v0@0, CAST(t1.v0 AS Float64)@2)], projection=[v0@1, v2@2], OutPartitioning=Hash([UnKnownColumn { name: "CAST(t1.v0 AS Float64)@3" }], 2)
RepartitionExec: partitioning=Hash([v0@0], 2), input_partitions=1, OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
MemoryExec: partitions=1, partition_sizes=[1], OutPartitioning=UnknownPartitioning(1)
RepartitionExec: partitioning=Hash([CAST(t1.v0 AS Float64)@2], 2), input_partitions=1, OutPartitioning=Hash([Column { name: "CAST(t1.v0 AS Float64)", index: 2 }], 2)
ProjectionExec: expr=[v0@0 as v0, v2@1 as v2, CAST(v0@0 AS Float64) as CAST(t1.v0 AS Float64)], OutPartitioning=UnknownPartitioning(1)
MemoryExec: partitions=1, partition_sizes=[4], OutPartitioning=UnknownPartitioning(1)
# InterleaveExec children[1] OutputPartitioning: Hash([UnKnownColumn { name: "v0@3" }], 2)
ProjectionExec: expr=[v2@1 as v2, v0@0 as v0], OutPartitioning=Hash([UnKnownColumn { name: "v0@3" }], 2)
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(CAST(t1.v0 AS Float64)@2, v0@0)], projection=[v0@0, v2@1], OutPartitioning=Hash([UnKnownColumn { name: "v0@3" }], 2)
RepartitionExec: partitioning=Hash([CAST(t1.v0 AS Float64)@2], 2), input_partitions=2, OutPartitioning=Hash([Column { name: "CAST(t1.v0 AS Float64)", index: 2 }], 2)
ProjectionExec: expr=[v0@0 as v0, v2@1 as v2, CAST(v0@0 AS Float64) as CAST(t1.v0 AS Float64)], OutPartitioning=RoundRobinBatch(2)
RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1, OutPartitioning=RoundRobinBatch(2)
FilterExec: v2@1 IS NULL, OutPartitioning=UnknownPartitioning(1)
MemoryExec: partitions=1, partition_sizes=[4], OutPartitioning=UnknownPartitioning(1)
RepartitionExec: partitioning=Hash([v0@0], 2), input_partitions=1, OutPartitioning=Hash([Column { name: "v0", index: 0 }], 2)
MemoryExec: partitions=1, partition_sizes=[1], OutPartitioning=UnknownPartitioning(1)
```
## What changes are included in this PR?
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make `UnKnownColumn`s not equal to others physical exprs [datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on PR #11536:
URL: https://github.com/apache/datafusion/pull/11536#issuecomment-2240806007
Thank you @alamb for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make `UnKnownColumn`s not equal to each other [datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #11536:
URL: https://github.com/apache/datafusion/pull/11536#discussion_r1683975286
##########
datafusion/physical-plan/src/union.rs:
##########
@@ -431,7 +431,12 @@ impl ExecutionPlan for InterleaveExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- Ok(Arc::new(InterleaveExec::try_new(children)?))
+ // New children may not be able to interleave; in that case, we fall back to UnionExec.
Review Comment:
Applied. Thank you @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make `UnKnownColumn`s not equal to others physical exprs [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11536:
URL: https://github.com/apache/datafusion/pull/11536#discussion_r1684926587
##########
datafusion/physical-expr/src/expressions/column.rs:
##########
@@ -95,11 +94,10 @@ impl PhysicalExpr for UnKnownColumn {
}
impl PartialEq<dyn Any> for UnKnownColumn {
- fn eq(&self, other: &dyn Any) -> bool {
- down_cast_any_ref(other)
- .downcast_ref::<Self>()
- .map(|x| self == x)
- .unwrap_or(false)
+ fn eq(&self, _other: &dyn Any) -> bool {
+ // UnknownColumn is not a valid expression, so it should not be equal to any other expression.
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make `UnKnownColumn`s not equal to others physical exprs [datafusion]
Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on code in PR #11536:
URL: https://github.com/apache/datafusion/pull/11536#discussion_r1684015535
##########
datafusion/physical-plan/src/union.rs:
##########
@@ -431,7 +431,12 @@ impl ExecutionPlan for InterleaveExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- Ok(Arc::new(InterleaveExec::try_new(children)?))
+ // New children may not be able to interleave; in that case, we fall back to UnionExec.
Review Comment:
> I see how this particular change is beneficial as now the plan runs where it didn't before, so that is hard to argue with and I would be ok merging it in
>
> However, I feel like this might be silently masking some bug in the planner. My personal preference would be to make this an error
>
> ```rust
> if !can_interleave(children.iter()) {
> return internal_err!("Can not create InterleaveExec: new children can not be interleaved");
> }
> ```
>
> And then track down / fix whatever optimizer pass is causing the children to no longer be interleavable
>
> FYI @mustafasrepo
This makes sense, invariant should be handled outside the operator during planning. This checks the whether assumptions met. Thanks for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: fall back to `UnionExec` if can't interleave [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11536:
URL: https://github.com/apache/datafusion/pull/11536#discussion_r1683441390
##########
datafusion/physical-plan/src/union.rs:
##########
@@ -431,7 +431,12 @@ impl ExecutionPlan for InterleaveExec {
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
- Ok(Arc::new(InterleaveExec::try_new(children)?))
+ // New children may not be able to interleave; in that case, we fall back to UnionExec.
Review Comment:
I see how this particular change is beneficial as now the plan runs where it didn't before, so that is hard to argue with and I would be ok merging it in
However, I feel like this might be silently masking some bug in the planner. My personal preference would be to make this an error
```rust
if !can_interleave(children.iter()) {
return internal_err!("Can not create InterleaveExec: new children can not be interleaved");
}
```
And then track down / fix whatever optimizer pass is causing the children to no longer be interleavable
FYI @mustafasrepo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make `UnKnownColumn`s not equal to each other [datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on PR #11536:
URL: https://github.com/apache/datafusion/pull/11536#issuecomment-2238594917
> However, I think the proposed fix might be treating the symptom rather than the root cause and I think more investigation is warranted. We could merge this PR and file a follow on ticket to investigate for example
@alamb I think the root clause should be UnKnownColumn. I changed to an alternative fix and updated the description in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make `UnKnownColumn`s not equal to others physical exprs [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11536:
URL: https://github.com/apache/datafusion/pull/11536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org