You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2025/08/30 14:32:46 UTC
[I] `NestedLoopJoin::swap_inputs` should check right partition number before swapping [datafusion]
milenkovicm opened a new issue, #17358:
URL: https://github.com/apache/datafusion/issues/17358
### Describe the bug
`NestedLoopJoin` left side has to have single partition, there is a check for this:
```rust
fn execute(
&self,
partition: usize,
context: Arc<TaskContext>,
) -> Result<SendableRecordBatchStream> {
if self.left.output_partitioning().partition_count() != 1 {
return internal_err!(
"Invalid NestedLoopJoinExec, the output partition count of the left child must be 1,\
consider using CoalescePartitionsExec or the EnforceDistribution rule"
);
}
...
}
```
but no check has been performed when join is created with `NestedLoopJoinExec::try_new` nor this is checked in `NestedLoopJoin::swap_inputs`.
I'm not sure if it is expected that additional `PhysicalOptimizerRule` which will insert `CoalescePartitionsExec` automatically (like `EnforceDistribution`) or this is a bug in NLJ .
### To Reproduce
_No response_
### Expected behavior
I believe this check should be performed when NLJ is created. Delaying check until `execute(...)` may be a bit too late.
Also, `NLJ::swap_inputs` should fail to swap left and right input if it will break this rule
Lastly, `JoinSelection` should not perform join re-order if right input has more than single partition.
https://github.com/apache/datafusion/blob/fd7df66724f958a2d44ba1fda1b11dc6833f0296/datafusion/physical-optimizer/src/join_selection.rs#L305-L306
### Additional context
This issue is experienced in ballista running TPC-DS query 54, https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3239033243
Ballista performs join re-ordering based on shuffle file information, before stage runs.
At that point all stages has been created so partitioning can't really change. Swapping join inputs and inserting `CoalescePartitionsExec` wont work for ballista.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #17358:
URL: https://github.com/apache/datafusion/issues/17358#issuecomment-3240205277
After looking at this issue I believe, this is NPT a datafusion bug, `EnforceDistribution` will add `CoalescePartitionsExec` so, proposed checks can't be done before `execute`. This is ballista issue and how it uses `JoinSelection`
I apologise for the noise!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #17358:
URL: https://github.com/apache/datafusion/issues/17358#issuecomment-3239569201
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed issue #17358: `NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs
URL: https://github.com/apache/datafusion/issues/17358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #17358:
URL: https://github.com/apache/datafusion/issues/17358#issuecomment-3240160723
@jonathanc-n before you start working on this, we need to establish if this is a bug or not, I'm not sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org