[I] InterleaveExec::with_new_children panics when optimizer rewrites change children's partitioning [datafusion]

["zhuqi-lucas (via GitHub)" <gi...@apache.org>]

zhuqi-lucas opened a new issue, #21826:
URL: https://github.com/apache/datafusion/issues/21826

   ## Describe the bug
   
   `InterleaveExec::with_new_children` uses `assert_or_internal_err!(can_interleave(children.iter()))` which panics when children's partitioning diverges after optimizer rewrites. This can happen when multiple physical optimizer passes modify the plan between when `InterleaveExec` is created and when `with_new_children` is called.
   
   ## To Reproduce
   
   This occurs in practice when:
   1. `EnforceDistribution` creates `InterleaveExec` for a `UnionExec` whose children all have matching Hash partitioning
   2. Subsequent optimizer passes (e.g. `JoinSelection`, additional `EnforceDistribution`/`EnforceSorting` passes) modify children's plan structure, changing their output partitioning
   3. `with_new_children` is called on the `InterleaveExec` with the modified children → assertion panic
   
   We hit this with a complex UNION ALL query (5 subqueries with JOINs, each reading from different table types — partitioned, single-file, MVs). The materialization task panics every time:
   
   ```
   join_selection
   caused by
   Internal error: Assertion failed: can_interleave(children.iter()): Can not create InterleaveExec: new children can not be interleaved.
   ```
   
   ## Expected behavior
   
   `InterleaveExec::with_new_children` should gracefully fall back to `UnionExec` when children are no longer interleavable, instead of panicking. The existing comment in the code already acknowledges this possibility:
   
   ```rust
   // New children are no longer interleavable, which might be a bug of optimization rewrite.
   ```
   
   Correctness is preserved when falling back to `UnionExec` — `EnforceDistribution` will add `RepartitionExec` as needed. Only the interleave optimization is lost.
   
   ## Proposed fix
   
   ```rust
   fn with_new_children(
       self: Arc<Self>,
       children: Vec<Arc<dyn ExecutionPlan>>,
   ) -> Result<Arc<dyn ExecutionPlan>> {
       if can_interleave(children.iter()) {
           Ok(Arc::new(InterleaveExec::try_new(children)?))
       } else {
           UnionExec::try_new(children)
       }
   }
   ```
   
   Will submit a PR with the fix and unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] InterleaveExec::with_new_children panics when optimizer rewrites change children's partitioning [datafusion]

["jayzhan211 (via GitHub)" <gi...@apache.org>]

jayzhan211 closed issue #21826: InterleaveExec::with_new_children panics when optimizer rewrites change children's partitioning
URL: https://github.com/apache/datafusion/issues/21826


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org