You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xudong963 (via GitHub)" <gi...@apache.org> on 2026/07/07 13:21:54 UTC

[PR] Fix union equivalence schema rewrite with stale constants [datafusion]

xudong963 opened a new pull request, #23375:
URL: https://github.com/apache/datafusion/pull/23375

   ## Which issue does this PR close?
   
   - Closes #23374.
   
   ## Rationale for this change
   
   `UnionExec::try_new` can panic while computing equivalence properties if stale constant metadata is carried across a projection and then rewritten to the union output schema.
   
   In the observed shape, a filter such as `ticker = 'ESU6'` can leave a uniform string constant in equivalence properties. After a parent projection drops `ticker`, union property schema rewriting can see the remaining column slot as a timestamp column and attempt to cast `'ESU6'` to `Timestamp`, which fails during planning.
   
   Equivalence constants are optimizer metadata, so an unrepresentable constant after schema rewrite should be discarded rather than failing query planning.
   
   ## What changes are included in this PR?
   
   - Drops a uniform constant during `EquivalenceProperties::with_new_schema` if its value cannot be cast to the rewritten expression type.
   - Removes trivial equivalence classes after dropping such constants.
   - Propagates `UnionExec::compute_properties` errors from `UnionExec::try_new` instead of unwrapping.
   - Adds a regression test for union equivalence schema rewrite with an unrepresentable stale constant value.
   
   ## Are these changes tested?
   
   Yes:
   
   - `cargo fmt --all -- --check`
   - `cargo test -p datafusion-physical-expr test_union_drops_unrepresentable_constant_value_after_schema_rewrite -- --nocapture`
   - `cargo test -p datafusion-physical-plan union::tests::test_union -- --nocapture`
   - `cargo clippy -p datafusion-physical-expr -p datafusion-physical-plan --all-targets -- -D warnings`
   
   ## Are there any user-facing changes?
   
   No API change. This prevents a planner panic for affected `UNION ALL` + filter + projection query shapes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix union equivalence schema rewrite with stale constants [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 merged PR #23375:
URL: https://github.com/apache/datafusion/pull/23375


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix union equivalence schema rewrite with stale constants [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23375:
URL: https://github.com/apache/datafusion/pull/23375#discussion_r3549331584


##########
datafusion/physical-expr/src/equivalence/properties/union.rs:
##########
@@ -899,6 +899,36 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn test_union_drops_unrepresentable_constant_value_after_schema_rewrite() -> Result<()>

Review Comment:
   Nice targeted test for the stale metadata state. As a follow-up, it may be worth adding a higher-level regression test that builds the filter/projection/union plan shape from the original issue and checks that `UnionExec::try_new` or physical planning succeeds. That would help protect the full path that produced the stale constant, not just the `calculate_union` boundary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix union equivalence schema rewrite with stale constants [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #23375:
URL: https://github.com/apache/datafusion/pull/23375#discussion_r3549390997


##########
datafusion/physical-expr/src/equivalence/properties/union.rs:
##########
@@ -899,6 +899,36 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn test_union_drops_unrepresentable_constant_value_after_schema_rewrite() -> Result<()>

Review Comment:
   Thanks, that makes sense. The current test intentionally targets the exact stale-metadata boundary in `calculate_union`. I agree a higher-level regression would be useful, especially if we can capture the original filter/projection/union shape reliably. Since the reduced issue does not include a concrete SQL repro and the normal projection path should drop removed-column constants, I’ll keep this PR focused and consider adding a `UnionExec::try_new`/planner-level regression as a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org