You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/17 06:38:53 UTC
(datafusion) 01/01: fix: preserve short-circuit evaluation in Boolean CASE simplification (#25143)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25143-16ecdc9bdff1c3ed11f6f2462ef9e908831d69a3
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 587e74c8442f95a8e7204dd5c71d2b8f0c02b194
Author: Yifan Chen <30...@users.noreply.github.com>
AuthorDate: Thu Sep 17 06:38:29 2026 +0000
fix: preserve short-circuit evaluation in Boolean CASE simplification (#25143)
## Which issue does this PR close?
Closes #25136.
## Rationale for this change
`CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END`
currently becomes an `AND` predicate. On input containing `'abc'`,
predicate evaluation can reach the cast even though the `WHEN` excluded
that row.
## What changes are included in this PR?
Restrict the searched Boolean CASE-to-AND/OR rewrite to columns or
literals in every `WHEN` condition and every `THEN`/`ELSE` output. Apply
the same restriction to the companion inversion rule. Even a nonvolatile
first condition can fail, so it must remain evaluated when both outputs
are identical constants.
This is deliberately conservative: it does not attempt general
expression fallibility analysis or change predicate ordering. More
complex conditional expressions retain their CASE structure. Existing
column/literal rewrites remain available, and two optimizer tests record
the intentionally reduced simplification.
Four EXPLAIN snapshots in `null_aware_mark_join.slt` and `subquery.slt`
also change. CASE expressions generated for ANY comparisons survive
until EXISTS decorrelation, allowing the first mark column to be reused
instead of duplicating its join. A correlated `max(...) IS NULL`
projection retains CASE under the stricter first-WHEN guard. Existing
query-result expectations are unchanged; an additional result test
covers non-NULL maxima, all-NULL groups, empty groups, and NULL
correlation keys.
## What is the testing strategy for this PR?
The SQL regressions in `case.slt` cover guarded casts in filters and
projections, true/false/NULL/implicit ELSE, a fallible later WHEN with
literal outputs, guarded division, and a selected invalid branch that
must still error. Five queries failed against current main before the
optimizer change. The regressions also cover a volatile first WHEN and a
fallible first WHEN with identical false outputs, which must still raise
the cast error.
Initial validation at `49c02ec7` with pinned Rust 1.97.0 (before the
first-WHEN review correction):
- `cargo fmt --all` and `git diff --check` passed.
- `cargo clippy --all-targets --all-features -- -D warnings` passed.
- The complete `./dev/rust_lint.sh` suite passed.
- The contributor-guide extended workspace command passed: 11,341 Rust
tests, 8 ignored, and all 513 SQL test files. No additional test
exclusions were needed.
Latest follow-up validation: all 787 optimizer tests and all 512 SQL
files supported by the existing review-fix binary passed, including six
focused CASE/subquery files. `cargo fmt --all`, `git diff --check`, and
`cargo clippy --all-targets --all-features -- -D warnings` passed with
pinned Rust 1.97.0. The full `./dev/rust_lint.sh` suite also passed.
Hosted CI for the follow-up remains pending.
The nearby `simplify_function_over_case` microbenchmark was run against
current main with separately rebuilt binaries (Rust 1.98.0, dev profile,
short Criterion samples). The quieter comparison reported no change or
changes within the noise threshold. It exercises string-output CASE, not
the retained Boolean CASE workloads, so this is not a general
performance claim.
## Are there any user-facing changes?
Guarded Boolean CASE expressions preserve branch-local evaluation. Some
optimized plans retain CASE expressions where Boolean lowering was
previously applied; there is no public API change.
Generated with OpenAI Codex assistance, including implementation, tests,
and this description.
---
.../src/simplify_expressions/expr_simplifier.rs | 229 +++++++++++----------
datafusion/sqllogictest/test_files/case.slt | 63 ++++++
.../test_files/null_aware_mark_join.slt | 70 +++----
datafusion/sqllogictest/test_files/subquery.slt | 49 +++--
4 files changed, 241 insertions(+), 170 deletions(-)
diff --git a/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs b/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs
index bb4a04bb7b..ed083996f9 100644
--- a/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs
+++ b/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs
@@ -1575,7 +1575,9 @@ impl TreeNodeRewriter for Simplifier<'_> {
// ---> (X AND A) OR (Y AND B AND NOT X) OR ... (NOT (X OR Y) AND Q)
//
// Note: the rationale for this rewrite is that the expr can then be further
- // simplified using the existing rules for AND/OR
+ // simplified using the existing rules for AND/OR. Unlike CASE, AND/OR
+ // do not guarantee branch-local evaluation, so only expose columns
+ // and literals from conditional branches.
Expr::Case(Case {
expr: None,
when_then_expr,
@@ -1586,7 +1588,8 @@ impl TreeNodeRewriter for Simplifier<'_> {
// or all thens are literal bools and a small number of them are true
|| (when_then_expr.iter().all(|(_, then)| is_bool_lit(then))
&& when_then_expr.iter().filter(|(_, then)| is_true(then)).count() < 3))
- && info.is_boolean_type(&when_then_expr[0].1)? =>
+ && info.is_boolean_type(&when_then_expr[0].1)?
+ && can_lower_case_to_boolean(&when_then_expr, else_expr.as_deref()) =>
{
// String disjunction of all the when predicates encountered so far. Not nullable.
let mut filter_expr = lit(false);
@@ -1643,7 +1646,8 @@ impl TreeNodeRewriter for Simplifier<'_> {
.filter(|(_, then)| is_false(then))
.count()
< 3
- && else_expr.as_deref().is_none_or(is_bool_lit) =>
+ && else_expr.as_deref().is_none_or(is_bool_lit)
+ && can_lower_case_to_boolean(&when_then_expr, else_expr.as_deref()) =>
{
Transformed::yes(
Expr::Case(Case {
@@ -2463,6 +2467,20 @@ fn simplify_inlist_set_operation(
}))
}
+/// Conservatively checks the inputs whose evaluation can change when lowering
+/// CASE to AND/OR. [`Expr`] has no general fallibility analysis: only columns and
+/// literals are admitted for all WHEN conditions and branch outputs.
+fn can_lower_case_to_boolean(
+ when_then_expr: &[(Box<Expr>, Box<Expr>)],
+ else_expr: Option<&Expr>,
+) -> bool {
+ let is_leaf = |expr: &Expr| matches!(expr, Expr::Column(_) | Expr::Literal(..));
+ when_then_expr
+ .iter()
+ .all(|(when, then)| is_leaf(when) && is_leaf(then))
+ && else_expr.is_none_or(is_leaf)
+}
+
/// Returns expression testing a boolean `expr` for being exactly `true` (not `false` or NULL).
fn is_exactly_true(expr: Expr, info: &SimplifyContext) -> Result<Expr> {
if !info.nullable(&expr)? {
@@ -2608,12 +2626,15 @@ mod tests {
// ELSE false
// END
//
- // Can be simplified to `i < 5`
+ // Fold the constant conditions, but preserve CASE because the THEN
+ // expression is outside the conservative column/literal subset.
let expr = when(col("i").gt(lit(5)).and(lit(false)), col("i").gt(lit(5)))
.when(col("i").lt(lit(5)).and(lit(true)), col("i").lt(lit(5)))
.otherwise(lit(false))
.unwrap();
- let expected = col("i").lt(lit(5));
+ let expected = when(col("i").lt(lit(5)), col("i").lt(lit(5)))
+ .otherwise(lit(false))
+ .unwrap();
assert_eq!(expected, simplifier.simplify(expr).unwrap());
}
@@ -4241,76 +4262,89 @@ mod tests {
Some(Box::new(lit("ready"))),
));
- assert_eq!(
- simplify(binary_expr(
- complex_case.clone(),
+ // Comparisons still fold into literal outputs, but later WHEN conditions must
+ // remain conditional: they are not columns or literals.
+ for (op, value, outputs) in [
+ (
Operator::Eq,
- lit("completed"),
- )),
- not_distinct_from(col("c1").eq(lit("completed")), lit(true)).and(
- distinct_from(col("c1").eq(lit("inboxed")), lit(true))
- .and(distinct_from(col("c1").eq(lit("scheduled")), lit(true)))
- )
- );
-
- assert_eq!(
- simplify(binary_expr(
- complex_case.clone(),
+ "completed",
+ [false, false, true, false, false, false, false],
+ ),
+ (
Operator::NotEq,
- lit("completed"),
- )),
- distinct_from(col("c1").eq(lit("completed")), lit(true))
- .or(not_distinct_from(col("c1").eq(lit("inboxed")), lit(true))
- .or(not_distinct_from(col("c1").eq(lit("scheduled")), lit(true))))
- );
-
- assert_eq!(
- simplify(binary_expr(
- complex_case.clone(),
+ "completed",
+ [true, true, false, true, true, true, true],
+ ),
+ (
Operator::Eq,
- lit("running"),
- )),
- not_distinct_from(col("c2"), lit(true)).and(
- distinct_from(col("c1").eq(lit("inboxed")), lit(true))
- .and(distinct_from(col("c1").eq(lit("scheduled")), lit(true)))
- .and(distinct_from(col("c1").eq(lit("completed")), lit(true)))
- .and(distinct_from(col("c1").eq(lit("paused")), lit(true)))
- )
- );
-
- assert_eq!(
- simplify(binary_expr(
- complex_case.clone(),
+ "running",
+ [false, false, false, false, true, false, false],
+ ),
+ (
Operator::Eq,
- lit("ready"),
- )),
- distinct_from(col("c1").eq(lit("inboxed")), lit(true))
- .and(distinct_from(col("c1").eq(lit("scheduled")), lit(true)))
- .and(distinct_from(col("c1").eq(lit("completed")), lit(true)))
- .and(distinct_from(col("c1").eq(lit("paused")), lit(true)))
- .and(distinct_from(col("c2"), lit(true)))
- .and(distinct_from(
- col("c1").eq(lit("invoked")).and(col("c3").gt(lit(0))),
- lit(true)
- ))
- );
-
- assert_eq!(
- simplify(binary_expr(
- complex_case.clone(),
+ "ready",
+ [false, false, false, false, false, false, true],
+ ),
+ (
Operator::NotEq,
- lit("ready"),
+ "ready",
+ [true, true, true, true, true, true, false],
+ ),
+ ] {
+ let Expr::Case(mut expected) = complex_case.clone() else {
+ unreachable!()
+ };
+ for ((_, then), value) in expected.when_then_expr.iter_mut().zip(outputs) {
+ **then = lit(value);
+ }
+ expected.else_expr = Some(Box::new(lit(outputs[6])));
+ assert_eq!(
+ simplify(binary_expr(complex_case.clone(), op, lit(value))),
+ Expr::Case(expected)
+ );
+ }
+ }
+
+ #[test]
+ fn simplify_case_preserves_conditional_expressions() {
+ let fallible =
+ Expr::Cast(Cast::new(Box::new(col("c1")), DataType::Int32)).gt(lit(0_i32));
+ let volatile = Expr::ScalarFunction(ScalarFunction::new_udf(
+ Arc::new(ScalarUDF::new_from_impl(VolatileUdf::new())),
+ vec![],
+ ));
+ for expr in [
+ Expr::Case(Case::new(
+ None,
+ vec![(Box::new(fallible.clone()), Box::new(lit(false)))],
+ Some(Box::new(lit(false))),
)),
- not_distinct_from(col("c1").eq(lit("inboxed")), lit(true))
- .or(not_distinct_from(col("c1").eq(lit("scheduled")), lit(true)))
- .or(not_distinct_from(col("c1").eq(lit("completed")), lit(true)))
- .or(not_distinct_from(col("c1").eq(lit("paused")), lit(true)))
- .or(not_distinct_from(col("c2"), lit(true)))
- .or(not_distinct_from(
- col("c1").eq(lit("invoked")).and(col("c3").gt(lit(0))),
- lit(true)
- ))
- );
+ Expr::Case(Case::new(
+ None,
+ vec![(Box::new(col("c2")), Box::new(fallible.clone()))],
+ Some(Box::new(lit(false))),
+ )),
+ Expr::Case(Case::new(
+ None,
+ vec![(Box::new(col("c2")), Box::new(lit(true)))],
+ Some(Box::new(fallible.clone())),
+ )),
+ Expr::Case(Case::new(
+ None,
+ vec![
+ (Box::new(col("c2")), Box::new(lit(false))),
+ (Box::new(fallible), Box::new(lit(true))),
+ ],
+ Some(Box::new(lit(false))),
+ )),
+ Expr::Case(Case::new(
+ None,
+ vec![(Box::new(volatile.gt(lit(0_i16))), Box::new(lit(true)))],
+ Some(Box::new(col("c2"))),
+ )),
+ ] {
+ assert_eq!(simplify(expr.clone()), expr);
+ }
}
#[test]
@@ -4354,19 +4388,20 @@ mod tests {
// CASE WHEN ISNULL(c2) THEN true ELSE c2
// -->
- // ISNULL(c2) OR c2
- //
- // Need to call simplify 2x due to
- // https://github.com/apache/datafusion/issues/1160
+ // Preserve CASE because the WHEN expression is outside the conservative
+ // column/literal subset.
+ let expected = Expr::Case(Case::new(
+ None,
+ vec![(Box::new(col("c2").is_null()), Box::new(lit(true)))],
+ Some(Box::new(col("c2"))),
+ ));
assert_eq!(
simplify(simplify(Expr::Case(Case::new(
None,
vec![(Box::new(col("c2").is_null()), Box::new(lit(true)),)],
Some(Box::new(col("c2"))),
)))),
- col("c2")
- .is_null()
- .or(col("c2").is_not_null().and(col("c2")))
+ expected
);
// CASE WHEN c1 then true WHEN c2 then false ELSE true
@@ -4407,29 +4442,21 @@ mod tests {
col("c1_non_null").or(col("c1_non_null").not().and(col("c2_non_null").not()))
);
- // CASE WHEN c > 0 THEN true END AS c1
- assert_eq!(
- simplify(simplify(Expr::Case(Case::new(
+ // Preserve CASE with a non-leaf WHEN condition, with or without ELSE.
+ for expr in [
+ Expr::Case(Case::new(
None,
vec![(Box::new(col("c3").gt(lit(0_i64))), Box::new(lit(true)))],
None,
- )))),
- not_distinct_from(col("c3").gt(lit(0_i64)), lit(true)).or(distinct_from(
- col("c3").gt(lit(0_i64)),
- lit(true)
- )
- .and(lit_bool_null()))
- );
-
- // CASE WHEN c > 0 THEN true ELSE false END AS c1
- assert_eq!(
- simplify(simplify(Expr::Case(Case::new(
+ )),
+ Expr::Case(Case::new(
None,
vec![(Box::new(col("c3").gt(lit(0_i64))), Box::new(lit(true)))],
Some(Box::new(lit(false))),
- )))),
- not_distinct_from(col("c3").gt(lit(0_i64)), lit(true))
- );
+ )),
+ ] {
+ assert_eq!(simplify(simplify(expr.clone())), expr);
+ }
}
#[test]
@@ -4626,22 +4653,6 @@ mod tests {
assert_eq!(simplify(expr.clone()), expr);
}
- fn distinct_from(left: impl Into<Expr>, right: impl Into<Expr>) -> Expr {
- Expr::BinaryExpr(BinaryExpr {
- left: Box::new(left.into()),
- op: Operator::IsDistinctFrom,
- right: Box::new(right.into()),
- })
- }
-
- fn not_distinct_from(left: impl Into<Expr>, right: impl Into<Expr>) -> Expr {
- Expr::BinaryExpr(BinaryExpr {
- left: Box::new(left.into()),
- op: Operator::IsNotDistinctFrom,
- right: Box::new(right.into()),
- })
- }
-
#[test]
fn simplify_expr_bool_or() {
// col || true is always true
diff --git a/datafusion/sqllogictest/test_files/case.slt b/datafusion/sqllogictest/test_files/case.slt
index f692d1d039..3f9058bce8 100644
--- a/datafusion/sqllogictest/test_files/case.slt
+++ b/datafusion/sqllogictest/test_files/case.slt
@@ -906,3 +906,66 @@ drop table floats;
#####
# End of lookup table CASE tests
#####
+
+statement ok
+CREATE VIEW guarded_cast AS SELECT * FROM (VALUES ('1'), ('abc'), ('2'), (NULL)) t(s);
+
+# A Boolean CASE must not become an AND that exposes the cast to 'abc'.
+# Regression for https://github.com/apache/datafusion/issues/25136.
+query T rowsort
+SELECT s FROM guarded_cast
+WHERE CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END;
+----
+1
+2
+
+# Preserve branch evaluation in projections, including both forms of ELSE NULL.
+query TBBB rowsort
+SELECT s,
+ CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE true END,
+ CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE NULL END,
+ CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 END
+FROM guarded_cast;
+----
+1 true true true
+2 true true true
+NULL true NULL NULL
+abc true NULL NULL
+
+# The ELSE expression also runs only on rows that reach it.
+query T rowsort
+SELECT s FROM guarded_cast
+WHERE CASE WHEN s ~ '^[a-z]+$' THEN false ELSE CAST(s AS INT) > 0 END;
+----
+1
+2
+
+# Literal outputs alone are insufficient: later WHEN expressions are conditional.
+query T rowsort
+SELECT s FROM guarded_cast
+WHERE CASE WHEN s ~ '^[a-z]+$' THEN false
+ WHEN CAST(s AS INT) > 0 THEN true ELSE false END;
+----
+1
+2
+
+# Division is another fallible branch expression, independent of cast handling.
+query I rowsort
+SELECT n FROM (VALUES (0), (1), (2), (NULL)) t(n)
+WHERE CASE WHEN n <> 0 THEN 10 / n > 1 ELSE false END;
+----
+1
+2
+
+# A selected invalid branch must still raise its original error.
+query error Cannot cast string 'abc' to value of Int32 type
+SELECT CASE WHEN s ~ '^[a-z]+$' THEN CAST(s AS INT) > 0 ELSE false END
+FROM guarded_cast;
+
+# The first WHEN must be evaluated even when both branch outputs are identical.
+query error Cannot cast string 'abc' to value of Int32 type
+SELECT CASE WHEN CAST(s AS INT) > 0 THEN false ELSE false END
+FROM (VALUES ('abc')) t(s);
+
+statement ok
+DROP VIEW guarded_cast;
diff --git a/datafusion/sqllogictest/test_files/null_aware_mark_join.slt b/datafusion/sqllogictest/test_files/null_aware_mark_join.slt
index 62c0dd3192..9d606bbe9b 100644
--- a/datafusion/sqllogictest/test_files/null_aware_mark_join.slt
+++ b/datafusion/sqllogictest/test_files/null_aware_mark_join.slt
@@ -298,28 +298,23 @@ WHERE (id = ANY (SELECT id FROM inner_table_with_null)) IS NULL;
----
logical_plan
01)Projection: outer_table.value
-02)--Filter: __correlated_sq_1.mark OR __correlated_sq_2.mark AND NOT __correlated_sq_3.mark AND Boolean(NULL) IS NULL
-03)----Projection: outer_table.value, __correlated_sq_1.mark, __correlated_sq_2.mark, __correlated_sq_3.mark
-04)------LeftMark Join: Filter: outer_table.id = __correlated_sq_3.id IS TRUE
-05)--------LeftMark Join: Filter: outer_table.id = __correlated_sq_2.id IS NULL
-06)----------LeftMark Join: Filter: outer_table.id = __correlated_sq_1.id IS TRUE
-07)------------TableScan: outer_table projection=[id, value]
-08)------------SubqueryAlias: __correlated_sq_1
-09)--------------TableScan: inner_table_with_null projection=[id]
-10)----------SubqueryAlias: __correlated_sq_2
-11)------------TableScan: inner_table_with_null projection=[id]
-12)--------SubqueryAlias: __correlated_sq_3
-13)----------TableScan: inner_table_with_null projection=[id]
+02)--Filter: __correlated_sq_1.mark IS NOT DISTINCT FROM Boolean(true) OR __correlated_sq_2.mark IS NOT DISTINCT FROM Boolean(true) AND __correlated_sq_1.mark IS DISTINCT FROM Boolean(true) AND Boolean(NULL) IS NULL
+03)----Projection: outer_table.value, __correlated_sq_1.mark, __correlated_sq_2.mark
+04)------LeftMark Join: Filter: outer_table.id = __correlated_sq_2.id IS NULL
+05)--------LeftMark Join: Filter: outer_table.id = __correlated_sq_1.id IS TRUE
+06)----------TableScan: outer_table projection=[id, value]
+07)----------SubqueryAlias: __correlated_sq_1
+08)------------TableScan: inner_table_with_null projection=[id]
+09)--------SubqueryAlias: __correlated_sq_2
+10)----------TableScan: inner_table_with_null projection=[id]
physical_plan
-01)FilterExec: mark@1 OR mark@2 AND NOT mark@3 AND NULL IS NULL, projection=[value@0]
-02)--NestedLoopJoinExec: join_type=RightMark, filter=(id@0 = id@1) IS NOT DISTINCT FROM true, projection=[value@1, mark@2, mark@3, mark@4]
+01)FilterExec: mark@1 IS NOT DISTINCT FROM true OR mark@2 IS NOT DISTINCT FROM true AND mark@1 IS DISTINCT FROM true AND NULL IS NULL, projection=[value@0]
+02)--NestedLoopJoinExec: join_type=RightMark, filter=id@0 = id@1 IS NULL, projection=[value@1, mark@2, mark@3]
03)----DataSourceExec: partitions=1, partition_sizes=[1]
-04)----NestedLoopJoinExec: join_type=RightMark, filter=id@0 = id@1 IS NULL
-05)------DataSourceExec: partitions=1, partition_sizes=[1]
-06)------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
-07)--------NestedLoopJoinExec: join_type=RightMark, filter=(id@0 = id@1) IS NOT DISTINCT FROM true
-08)----------DataSourceExec: partitions=1, partition_sizes=[1]
-09)----------DataSourceExec: partitions=1, partition_sizes=[1]
+04)----RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
+05)------NestedLoopJoinExec: join_type=RightMark, filter=(id@0 = id@1) IS NOT DISTINCT FROM true
+06)--------DataSourceExec: partitions=1, partition_sizes=[1]
+07)--------DataSourceExec: partitions=1, partition_sizes=[1]
query T rowsort
SELECT value
@@ -338,28 +333,23 @@ WHERE (id = ANY (SELECT id FROM inner_table_no_null)) IS TRUE;
----
logical_plan
01)Projection: outer_table.value
-02)--Filter: __correlated_sq_1.mark OR __correlated_sq_2.mark AND NOT __correlated_sq_3.mark AND Boolean(NULL) IS TRUE
-03)----Projection: outer_table.value, __correlated_sq_1.mark, __correlated_sq_2.mark, __correlated_sq_3.mark
-04)------LeftMark Join: Filter: outer_table.id = __correlated_sq_3.id IS TRUE
-05)--------LeftMark Join: Filter: outer_table.id = __correlated_sq_2.id IS NULL
-06)----------LeftMark Join: Filter: outer_table.id = __correlated_sq_1.id IS TRUE
-07)------------TableScan: outer_table projection=[id, value]
-08)------------SubqueryAlias: __correlated_sq_1
-09)--------------TableScan: inner_table_no_null projection=[id]
-10)----------SubqueryAlias: __correlated_sq_2
-11)------------TableScan: inner_table_no_null projection=[id]
-12)--------SubqueryAlias: __correlated_sq_3
-13)----------TableScan: inner_table_no_null projection=[id]
+02)--Filter: __correlated_sq_1.mark IS NOT DISTINCT FROM Boolean(true) OR __correlated_sq_2.mark IS NOT DISTINCT FROM Boolean(true) AND __correlated_sq_1.mark IS DISTINCT FROM Boolean(true) AND Boolean(NULL) IS TRUE
+03)----Projection: outer_table.value, __correlated_sq_1.mark, __correlated_sq_2.mark
+04)------LeftMark Join: Filter: outer_table.id = __correlated_sq_2.id IS NULL
+05)--------LeftMark Join: Filter: outer_table.id = __correlated_sq_1.id IS TRUE
+06)----------TableScan: outer_table projection=[id, value]
+07)----------SubqueryAlias: __correlated_sq_1
+08)------------TableScan: inner_table_no_null projection=[id]
+09)--------SubqueryAlias: __correlated_sq_2
+10)----------TableScan: inner_table_no_null projection=[id]
physical_plan
-01)FilterExec: (mark@1 OR mark@2 AND NOT mark@3 AND NULL) IS NOT DISTINCT FROM true, projection=[value@0]
-02)--NestedLoopJoinExec: join_type=RightMark, filter=(id@0 = id@1) IS NOT DISTINCT FROM true, projection=[value@1, mark@2, mark@3, mark@4]
+01)FilterExec: (mark@1 IS NOT DISTINCT FROM true OR mark@2 IS NOT DISTINCT FROM true AND mark@1 IS DISTINCT FROM true AND NULL) IS NOT DISTINCT FROM true, projection=[value@0]
+02)--NestedLoopJoinExec: join_type=RightMark, filter=id@0 = id@1 IS NULL, projection=[value@1, mark@2, mark@3]
03)----DataSourceExec: partitions=1, partition_sizes=[1]
-04)----NestedLoopJoinExec: join_type=RightMark, filter=id@0 = id@1 IS NULL
-05)------DataSourceExec: partitions=1, partition_sizes=[1]
-06)------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
-07)--------NestedLoopJoinExec: join_type=RightMark, filter=(id@0 = id@1) IS NOT DISTINCT FROM true
-08)----------DataSourceExec: partitions=1, partition_sizes=[1]
-09)----------DataSourceExec: partitions=1, partition_sizes=[1]
+04)----RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
+05)------NestedLoopJoinExec: join_type=RightMark, filter=(id@0 = id@1) IS NOT DISTINCT FROM true
+06)--------DataSourceExec: partitions=1, partition_sizes=[1]
+07)--------DataSourceExec: partitions=1, partition_sizes=[1]
query T rowsort
SELECT value
diff --git a/datafusion/sqllogictest/test_files/subquery.slt b/datafusion/sqllogictest/test_files/subquery.slt
index 7aa24d4c7f..626ef60762 100644
--- a/datafusion/sqllogictest/test_files/subquery.slt
+++ b/datafusion/sqllogictest/test_files/subquery.slt
@@ -1191,7 +1191,7 @@ query TT
explain SELECT t1_id, (SELECT max(t2.t2_id) is null FROM t2 WHERE t2.t2_int = t1.t1_int) x from t1
----
logical_plan
-01)Projection: t1.t1_id, __scalar_sq_1.__always_true IS NULL OR __scalar_sq_1.__always_true IS NOT NULL AND __scalar_sq_1.max(t2.t2_id) IS NULL AS x
+01)Projection: t1.t1_id, CASE WHEN __scalar_sq_1.__always_true IS NULL THEN Boolean(true) ELSE __scalar_sq_1.max(t2.t2_id) IS NULL END AS x
02)--Left Join: t1.t1_int = __scalar_sq_1.t2_int
03)----TableScan: t1 projection=[t1_id, t1_int]
04)----SubqueryAlias: __scalar_sq_1
@@ -1199,6 +1199,19 @@ logical_plan
06)--------Aggregate: groupBy=[[t2.t2_int]], aggr=[[max(t2.t2_id)]]
07)----------TableScan: t2 projection=[t2_id, t2_int]
+# A correlated aggregate must distinguish a non-NULL maximum from both an
+# all-NULL group and an empty group (including a NULL correlation key).
+query IB rowsort
+WITH outer_rows(k) AS (VALUES (1), (2), (3), (NULL)),
+ inner_rows(k, v) AS (VALUES (1, 10), (1, NULL), (2, NULL), (NULL, 20))
+SELECT k, (SELECT max(v) IS NULL FROM inner_rows WHERE inner_rows.k = outer_rows.k)
+FROM outer_rows
+----
+1 false
+2 true
+3 true
+NULL true
+
query TT
explain SELECT t1_id, (SELECT max(t2.t2_id) FROM t2 WHERE t2.t2_int = t1.t1_int) x from t1
----
@@ -1781,27 +1794,21 @@ explain select v from (values (1), (6), (10)) set_cmp_t(v) where v > any(select
----
logical_plan
01)Projection: set_cmp_t.v
-02)--LeftMark Join: Filter: set_cmp_t.v > __correlated_sq_3.v IS TRUE
+02)--LeftMark Join: Filter: set_cmp_t.v > __correlated_sq_2.v IS NULL
03)----Projection: set_cmp_t.v
-04)------LeftMark Join: Filter: set_cmp_t.v > __correlated_sq_2.v IS NULL
-05)--------Projection: set_cmp_t.v
-06)----------Filter: __correlated_sq_1.mark
-07)------------LeftMark Join: Filter: set_cmp_t.v > __correlated_sq_1.v IS TRUE
-08)--------------SubqueryAlias: set_cmp_t
-09)----------------Projection: column1 AS v
-10)------------------Values: (Int64(1)), (Int64(6)), (Int64(10))
-11)--------------SubqueryAlias: __correlated_sq_1
-12)----------------SubqueryAlias: set_cmp_s
-13)------------------Projection: column1 AS v
-14)--------------------Values: (Int64(5)), (Int64(NULL))
-15)--------SubqueryAlias: __correlated_sq_2
-16)----------SubqueryAlias: set_cmp_s
-17)------------Projection: column1 AS v
-18)--------------Values: (Int64(5)), (Int64(NULL))
-19)----SubqueryAlias: __correlated_sq_3
-20)------SubqueryAlias: set_cmp_s
-21)--------Projection: column1 AS v
-22)----------Values: (Int64(5)), (Int64(NULL))
+04)------Filter: __correlated_sq_1.mark IS NOT DISTINCT FROM Boolean(true)
+05)--------LeftMark Join: Filter: set_cmp_t.v > __correlated_sq_1.v IS TRUE
+06)----------SubqueryAlias: set_cmp_t
+07)------------Projection: column1 AS v
+08)--------------Values: (Int64(1)), (Int64(6)), (Int64(10))
+09)----------SubqueryAlias: __correlated_sq_1
+10)------------SubqueryAlias: set_cmp_s
+11)--------------Projection: column1 AS v
+12)----------------Values: (Int64(5)), (Int64(NULL))
+13)----SubqueryAlias: __correlated_sq_2
+14)------SubqueryAlias: set_cmp_s
+15)--------Projection: column1 AS v
+16)----------Values: (Int64(5)), (Int64(NULL))
# same-table `= ANY` / `<> ALL` must plan without
# "duplicate unqualified field name mark".
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org