You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "emecii (via GitHub)" <gi...@apache.org> on 2026/09/10 07:01:53 UTC
[PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
emecii opened a new pull request, #25143:
URL: https://github.com/apache/datafusion/pull/25143
## Which issue does this PR close?
Closes #25136.
## Rationale for this change
`CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END` currently becomes an `AND` predicate. On input containing `'abc'`, predicate evaluation can reach the cast even though the `WHEN` excluded that row.
## What changes are included in this PR?
Restrict the searched Boolean CASE-to-AND/OR rewrite to columns or literals in `THEN`, `ELSE`, and later `WHEN` expressions. The first `WHEN` already runs on every row, but must be nonvolatile because expansion may repeat it. Apply the same restriction to the companion inversion rule.
This is deliberately conservative: it does not attempt general expression fallibility analysis or change predicate ordering. More complex conditional expressions retain their CASE structure. Existing column/literal rewrites remain available, and two optimizer tests record the intentionally reduced simplification.
Three EXPLAIN snapshots in `null_aware_mark_join.slt` and `subquery.slt` also change. CASE expressions generated for ANY comparisons now survive until EXISTS decorrelation, allowing the first mark column to be reused instead of duplicating its join. Their query-result expectations are unchanged.
## What is the testing strategy for this PR?
The SQL regressions in `case.slt` cover guarded casts in filters and projections, true/false/NULL/implicit ELSE, a fallible later WHEN with literal outputs, guarded division, and a selected invalid branch that must still error. Five queries failed against current main before the optimizer change. The unit regression also covers a volatile first WHEN.
Validation with pinned Rust 1.97.0:
- `cargo fmt --all` and `git diff --check` passed.
- `cargo clippy --all-targets --all-features -- -D warnings` passed.
- The complete `./dev/rust_lint.sh` suite passed.
- The contributor-guide extended workspace command passed: 11,341 Rust tests, 8 ignored, and all 513 SQL test files. No additional test exclusions were needed.
The nearby `simplify_function_over_case` microbenchmark was run against current main with separately rebuilt binaries (Rust 1.98.0, dev profile, short Criterion samples). The quieter comparison reported no change or changes within the noise threshold. It exercises string-output CASE, not the retained Boolean CASE workloads, so this is not a general performance claim.
## Are there any user-facing changes?
Guarded Boolean CASE expressions preserve branch-local evaluation. Some optimized plans retain CASE expressions where Boolean lowering was previously applied; there is no public API change.
Generated with OpenAI Codex assistance, including implementation, tests, and this description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25143:
URL: https://github.com/apache/datafusion/pull/25143#issuecomment-5633684322
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25143?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `96.38554%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.83%. Comparing base ([`423df6f`](https://app.codecov.io/gh/apache/datafusion/commit/423df6f4fc6a6eac6d0d1457166a166dafe8030d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`49c02ec`](https://app.codecov.io/gh/apache/datafusion/commit/49c02ec7c4b423c819d3be08f541630d21e64b52?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 36 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25143?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...imizer/src/simplify\_expressions/expr\_simplifier.rs](https://app.codecov.io/gh/apache/datafusion/pull/25143?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fsimplify_expressions%2Fexpr_simplifier.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3NpbXBsaWZ5X2V4cHJlc3Npb25zL2V4cHJfc2ltcGxpZmllci5ycw==) | 96.38% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25143 +/- ##
==========================================
- Coverage 81.83% 81.83% -0.01%
==========================================
Files 1130 1130
Lines 418785 418819 +34
Branches 418785 418819 +34
==========================================
+ Hits 342723 342727 +4
- Misses 55857 55883 +26
- Partials 20205 20209 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25143?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25143:
URL: https://github.com/apache/datafusion/pull/25143#issuecomment-5710130709
🚀
@emecii
Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
Posted by "emecii (via GitHub)" <gi...@apache.org>.
emecii commented on code in PR #25143:
URL: https://github.com/apache/datafusion/pull/25143#discussion_r3989414124
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -2432,6 +2436,26 @@ fn simplify_inlist_set_operation(
}))
}
+/// Conservatively checks the inputs whose evaluation can change when lowering
+/// CASE to AND/OR. [`Expr`] has no general fallibility analysis: only columns and
+/// literals are admitted from conditional branches, including later WHEN conditions.
+/// The first WHEN already runs on every row, but must not be volatile because
+/// the rewrite can evaluate it more than once.
+fn can_lower_case_to_boolean(
+ when_then_expr: &[(Box<Expr>, Box<Expr>)],
+ else_expr: Option<&Expr>,
+) -> bool {
+ let is_leaf = |expr: &Expr| matches!(expr, Expr::Column(_) | Expr::Literal(..));
+ when_then_expr.iter().enumerate().all(|(i, (when, then))| {
+ is_leaf(then)
+ && if i == 0 {
+ !when.is_volatile()
Review Comment:
Fixed in 3d52a60d0. The guard now requires the first WHEN, every later WHEN, and all branch outputs to be columns or literals. I added the CAST(s AS INT) regression with identical false outputs and a unit case proving the CASE is preserved.
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -2432,6 +2436,26 @@ fn simplify_inlist_set_operation(
}))
}
+/// Conservatively checks the inputs whose evaluation can change when lowering
+/// CASE to AND/OR. [`Expr`] has no general fallibility analysis: only columns and
+/// literals are admitted from conditional branches, including later WHEN conditions.
Review Comment:
Updated in 3d52a60d0. The invariant comment now explicitly states that all WHEN conditions and branch outputs must be columns or literals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
Posted by "emecii (via GitHub)" <gi...@apache.org>.
emecii commented on PR #25143:
URL: https://github.com/apache/datafusion/pull/25143#issuecomment-5662606317
Pushed 2effd1d4b to update the correlated `max(...) IS NULL` EXPLAIN expectation for the stricter first-WHEN guard. The normal, extended, SQLite, and benchmark-verification failures all reported this same single plan mismatch. Added result coverage for non-NULL maxima, all-NULL/empty groups, and NULL correlation keys; the stricter guard is unchanged.
Validation passed: 787 optimizer tests, all 512 SLT files supported by the existing review-fix binary (including the six focused files), all-target/all-feature Clippy, and the full lint suite. The separate CLI MinIO 404 is tracked in #25215 and fixed upstream by #25216. Hosted CI for this new head remains pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #25143:
URL: https://github.com/apache/datafusion/pull/25143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve short-circuit evaluation in Boolean CASE simplification [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #25143:
URL: https://github.com/apache/datafusion/pull/25143#discussion_r3988603167
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -2432,6 +2436,26 @@ fn simplify_inlist_set_operation(
}))
}
+/// Conservatively checks the inputs whose evaluation can change when lowering
+/// CASE to AND/OR. [`Expr`] has no general fallibility analysis: only columns and
+/// literals are admitted from conditional branches, including later WHEN conditions.
+/// The first WHEN already runs on every row, but must not be volatile because
+/// the rewrite can evaluate it more than once.
+fn can_lower_case_to_boolean(
+ when_then_expr: &[(Box<Expr>, Box<Expr>)],
+ else_expr: Option<&Expr>,
+) -> bool {
+ let is_leaf = |expr: &Expr| matches!(expr, Expr::Column(_) | Expr::Literal(..));
+ when_then_expr.iter().enumerate().all(|(i, (when, then))| {
+ is_leaf(then)
+ && if i == 0 {
+ !when.is_volatile()
Review Comment:
I think there is still a correctness issue with allowing the first `WHEN` as long as it is non-volatile. It can still fail, and the Boolean rewrite can eliminate its evaluation entirely.
For example, `SELECT CASE WHEN CAST(s AS INT) > 0 THEN false ELSE false END FROM (VALUES ('abc')) t(s)` should raise the cast error because CASE evaluates its first WHEN for every row. With this rewrite, both branches can simplify to `false`, so the condition is never evaluated and the query returns `false` instead.
Could we require the first `WHEN` to be a column or literal as well, unless we add a sound fallibility analysis? It would also be good to add this example as a regression test, particularly with identical literal results so we cover the case where simplification removes the condition.
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -2432,6 +2436,26 @@ fn simplify_inlist_set_operation(
}))
}
+/// Conservatively checks the inputs whose evaluation can change when lowering
+/// CASE to AND/OR. [`Expr`] has no general fallibility analysis: only columns and
+/// literals are admitted from conditional branches, including later WHEN conditions.
Review Comment:
Once the first-WHEN guard is tightened, could we update this comment to match the invariant and say that all `WHEN` conditions and branch outputs must be columns or literals? I think spelling that out here will make it less likely that a future special case accidentally reintroduces this error-preservation gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org