You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/21 16:38:48 UTC

(spark) branch branch-4.3 updated: [SPARK-59579][4.3][SQL] Keep the conditional-branch guards after the subexpression elimination shortcut peel

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 7fb067473a60 [SPARK-59579][4.3][SQL] Keep the conditional-branch guards after the subexpression elimination shortcut peel
7fb067473a60 is described below

commit 7fb067473a60f32837315390b70a3db3ea086689
Author: YangJie <ya...@baidu.com>
AuthorDate: Mon Sep 21 09:38:37 2026 -0700

    [SPARK-59579][4.3][SQL] Keep the conditional-branch guards after the subexpression elimination shortcut peel
    
    ### What changes were proposed in this pull request?
    
    This backports #58868 to `branch-4.3`. One thing differs from the master change: `childrenToRecurse` here has no `With` case, because SPARK-58818 is not on this branch, so there is no after-peel re-check to remove and the `With` half of the master change does not apply. The rest is the same.
    
    `EquivalentExpressions.childrenToRecurse` refuses to descend into children that must not be evaluated ahead of time: a `CodegenFallback` generates no code for them, a `ConditionalExpression` offers only its `alwaysEvaluatedInputs`, and a `HigherOrderFunction` only its always-evaluated arguments.
    
    With `spark.sql.subexpressionElimination.skipForShortcutExpr` on, `skipForShortcut` peels the leading `And`/`Or` operands to reach the one operand of the chain that is always evaluated. `And`/`Or` are not `ConditionalExpression`s, so that peel walked straight past every case above and the result was used as `peeled.children`, recursing into all of the peeled expression's children, conditional branches included.
    
    The always-evaluated operand is now computed once in `updateExprTree`, and both `childrenToRecurse` and `commonChildrenToRecurse` are asked about that operand. Asking `commonChildrenToRecurse` recovers what the peel used to drop: the branch groups of a conditional the peel lands on, whose shared subexpression is safe to evaluate once because the conditional itself always runs.
    
    `childrenToRecurse` still peels the always-evaluated inputs of a `ConditionalExpression`. That peel no longer keeps the descent safe: `updateExprTree` peels before it descends, so `childrenToRecurse` is asked about the peeled node either way. It stays because dropping it would pass `updateExprTree` the input itself rather than the operand its peel lands on.
    
    One consequence beyond stopping the leak: this branch reached a peeled-to conditional's branches by recursing into all of its children, so a subexpression shared by every branch picked up a use count of 2 and was eliminated. It now goes through the branch-group intersection, which contributes 1, and is eliminated only if it also occurs in an always-evaluated position, which is how a conditional met directly has always behaved. Where each branch holds the subexpression once, that elimi [...]
    
    Nothing changes while the config is off, where `skipForShortcut` returns its argument unchanged and the recursion meets the conditional directly, guards and all.
    
    ### Why are the changes needed?
    
    The config exists to stop a short-circuited operand from being evaluated eagerly, and it made a conditional branch behind that operand eligible instead. With ANSI mode and both subexpression elimination configs on, `select (case when id = 0 then false else (1 / id + 1 / id) > 0 end) and id >= 0 from range(0, 1, 1, 1)` raised `[DIVIDE_BY_ZERO]`. `1 / id` is repeated inside a single branch body, is shared with no other branch, and should run only when that branch runs; for id = 0 the ot [...]
    
    The bypass reaches the `HigherOrderFunction` case too, where the cost is a failure rather than an early evaluation: a subexpression repeated inside a lambda body becomes a candidate, `supportedExpression` does not stop it because `NamedLambdaVariable` carries no `LAMBDA_VARIABLE` pattern, and generating that candidate at the top of the projection leaves `CodegenContext.getLambdaVar` with no variable to bind. No query was found where that changes an answer, so that half closes a hole r [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, for a query run with `spark.sql.subexpressionElimination.skipForShortcutExpr` enabled, which is off by default: the query above returns `false` where it used to raise `[DIVIDE_BY_ZERO]`. No change with the config off.
    
    ### How was this patch tested?
    
    `SubexpressionEliminationSuite` pins that no subexpression from a conditional branch, a `CodegenFallback`'s children or a lambda body is eliminated, with each shape met directly and behind an `And`, an `Or` chain and a mixed chain. Two positive controls sit beside it: the operand the peel lands on still contributes its own duplicates, and a duplicate shared by every branch of an `If` or `CaseWhen` group is still eliminated once, with the use count pinning that it came from the group r [...]
    
    `SQLQuerySuite` covers the query above, in both its `CASE WHEN` and `IF` spellings. ANSI mode is pinned on because that is what turns the extra evaluation into a failure; with it off the division returns null, the branch still yields false, and the case would pass either way in the scheduled non-ANSI build. The SPARK-58211 case beside it gains the same pin, which it was missing for the same reason.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #58935 from LuciferYang/SPARK-59579-4.3.
    
    Authored-by: YangJie <ya...@baidu.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../expressions/EquivalentExpressions.scala        | 20 ++++-
 .../SubexpressionEliminationSuite.scala            | 89 ++++++++++++++++++++++
 .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 28 ++++++-
 3 files changed, 132 insertions(+), 5 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
index 7722d5611670..5e82a00a6689 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
@@ -155,15 +155,21 @@ class EquivalentExpressions(
   //   3. HigherOrderFunction: lambda functions operate in the context of local lambdas and can't
   //        be called outside of that scope, only always-evaluated arguments can be evaluated
   //        ahead of time.
+  // The caller has already applied `skipForShortcut`, so these cases are asked about the operand
+  // that the peel lands on rather than about the `And`/`Or` chain above it.
   private def childrenToRecurse(expr: Expression): Seq[Expression] = expr match {
     case _: CodegenFallback => Nil
+    // Peeling here is redundant for safety: `updateExprTree` peels before it descends, so this
+    // method sees the peeled node either way. It stays because dropping it would pass
+    // `updateExprTree` the input itself rather than the operand its peel lands on.
     case c: ConditionalExpression => c.alwaysEvaluatedInputs.map(skipForShortcut)
     case h: HigherOrderFunction => h.alwaysEvaluatedArguments
-    case other => skipForShortcut(other).children
+    case other => other.children
   }
 
   // For some special expressions we cannot just recurse into all of its children, but we can
-  // recursively add the common expressions shared between all of its children.
+  // recursively add the common expressions shared between all of its children. As with
+  // `childrenToRecurse`, the caller has already applied `skipForShortcut`.
   private def commonChildrenToRecurse(expr: Expression): Seq[Seq[Expression]] = expr match {
     case _: CodegenFallback => Nil
     case c: ConditionalExpression => c.branchGroups
@@ -199,8 +205,14 @@ class EquivalentExpressions(
 
     if (!skip && !updateExprInMap(expr, map, useCount)) {
       val uc = useCount.sign
-      childrenToRecurse(expr).foreach(updateExprTree(_, map, uc))
-      commonChildrenToRecurse(expr).filter(_.nonEmpty).foreach(updateCommonExprs(_, map, uc))
+      // Peel first: which children may be recursed into is a question about the operand that is
+      // always evaluated, and `And`/`Or` are not `ConditionalExpression`s. Asking
+      // `childrenToRecurse` about the chain above that operand walks past its guards; asking
+      // `commonChildrenToRecurse` about the chain reaches no branch group at all.
+      val alwaysEvaluated = skipForShortcut(expr)
+      childrenToRecurse(alwaysEvaluated).foreach(updateExprTree(_, map, uc))
+      commonChildrenToRecurse(alwaysEvaluated).filter(_.nonEmpty)
+        .foreach(updateCommonExprs(_, map, uc))
     }
   }
 
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala
index 4f5525158af9..98ed67273174 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala
@@ -545,6 +545,95 @@ class SubexpressionEliminationSuite extends SparkFunSuite with ExpressionEvalHel
     equivalence1.addExprTree(caseWhenExpr1)
     assert(equivalence1.getCommonSubexpressions.size == 1)
   }
+
+  test("SPARK-59579: the short-circuit peel keeps the guards of the operand it lands on") {
+    // `skipForShortcut` peels the leading `And`/`Or` operands down to the one operand that is
+    // always evaluated. That operand can be an expression whose children must not be recursed
+    // into, and `And`/`Or` are not `ConditionalExpression`s, so the peel walks right past the
+    // cases that stop the descent. What may be recursed into has to be asked about the operand
+    // the peel lands on, not about the `And`/`Or` chain it started from.
+    val a = AttributeReference("a", IntegerType)()
+    val add = Add(a, Literal(1))
+    // `add` occurs twice in one branch body and nowhere else, so no group of branches shares it:
+    // `branchGroups` intersects the branches of a group, so a body that shares nothing with the
+    // other branches contributes nothing. That is what makes the assertions below hold for the
+    // right reason rather than because nothing was recorded at all.
+    val body = GreaterThan(Multiply(add, add), Literal(0))
+    val otherBody = GreaterThan(a, Literal(1))
+    val lambdaVar = NamedLambdaVariable("x", IntegerType, nullable = false)
+    val lambdaAdd = Add(lambdaVar, Literal(1))
+    val guarded = Seq[Expression](
+      If(GreaterThan(a, Literal(0)), body, otherBody),
+      CaseWhen(Seq((GreaterThan(a, Literal(0)), body)), otherBody),
+      // Two branches, so that the conditions group is not empty either.
+      CaseWhen(
+        Seq((GreaterThan(a, Literal(0)), body), (LessThan(a, Literal(0)), otherBody)),
+        otherBody),
+      CodegenFallbackExpression(body),
+      // A candidate from a lambda body is a worse outcome than one evaluated too eagerly:
+      // generating it outside the lambda leaves `getLambdaVar` with no variable to bind, which
+      // throws. Nothing else stops it -- `supportedExpression` looks for the `LAMBDA_VARIABLE`
+      // pattern, which `NamedLambdaVariable` does not carry.
+      ArrayExists(
+        CreateArray(Seq(a)),
+        LambdaFunction(
+          GreaterThan(Multiply(lambdaAdd, lambdaAdd), Literal(0)),
+          Seq(lambdaVar)),
+        followThreeValuedLogic = true))
+
+    def statesOf(expr: Expression): Seq[ExpressionStats] = {
+      val equivalence = new EquivalentExpressions(skipForShortcutEnable = true)
+      equivalence.addExprTree(expr)
+      equivalence.getAllExprStates()
+    }
+
+    def assertNoneEliminated(expr: Expression, hint: String): Unit = {
+      val states = statesOf(expr)
+      // Guards the next assert against passing vacuously: `supportedExpression` would refuse the
+      // whole tree if `NamedLambdaVariable` ever carried the `LAMBDA_VARIABLE` pattern, and then
+      // nothing at all would be recorded.
+      assert(states.nonEmpty, s"nothing was recorded, $hint: $expr")
+      assert(states.forall(_.useCount == 1),
+        s"eliminated ${states.filter(_.useCount > 1).map(_.expr)}, $hint: $expr")
+    }
+
+    guarded.foreach { expr =>
+      // Met directly, the guards are the ones that were already there: `skipForShortcut` returns a
+      // node that is not an `And`/`Or` unchanged, so this line behaves as it did before. It is here
+      // so that a future change to the guards has to keep both cases in step.
+      assertNoneEliminated(expr, "met directly")
+      // Behind one, behind a chain of, and behind a mix of short-circuit operands is where the peel
+      // used to walk past the guards.
+      assertNoneEliminated(And(expr, otherBody), "behind an And")
+      assertNoneEliminated(Or(Or(expr, otherBody), otherBody), "behind an Or chain")
+      assertNoneEliminated(And(Or(expr, otherBody), otherBody), "behind a mixed chain")
+    }
+
+    def eliminated(expr: Expression): Seq[ExpressionStats] =
+      statesOf(expr).filter(_.useCount > 1)
+
+    // The operand the peel lands on is not treated as opaque: it still contributes its own
+    // duplicates when nothing guards it.
+    assert(eliminated(And(body, otherBody)).map(_.expr) == Seq(add))
+    // A duplicate shared by every branch of a group is still eliminated, one evaluation standing
+    // for whichever branch runs, as long as it occurs once more outside the group -- here in the
+    // condition, which is always evaluated. The use count is what pins this down: asking
+    // `branchGroups` about the `And` instead of about the operand the peel lands on leaves 1 and
+    // no elimination, and recursing into every branch as if it always ran gives 3. The `If` shape
+    // is here because `branchGroups` groups its two values, where `CaseWhen` groups conditions and
+    // values separately.
+    val shared = CaseWhen(
+      Seq((GreaterThan(add, Literal(0)), GreaterThan(add, Literal(1)))),
+      GreaterThan(add, Literal(2)))
+    val sharedIf =
+      If(GreaterThan(add, Literal(0)), GreaterThan(add, Literal(1)), GreaterThan(add, Literal(2)))
+    Seq[Expression](shared, And(shared, otherBody), sharedIf, And(sharedIf, otherBody))
+        .foreach { expr =>
+      val states = eliminated(expr)
+      assert(states.map(_.expr) == Seq(add), s"$expr")
+      assert(states.head.useCount == 2, s"$expr")
+    }
+  }
 }
 
 case class CodegenFallbackExpression(child: Expression)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 713fa39a9243..25184f52b219 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -3340,8 +3340,11 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
     // is duplicated *inside* operand 2 (not split across operands 2 and 3) so the failure shape
     // does not depend on how many operands the one-level peel happened to drop, and operand 3 is
     // non-foldable to keep it in the plan. skipForShortcutExpr must peel every leading AND/OR
-    // operand, not just one, to make this hold for three or more operands.
+    // operand, not just one, to make this hold for three or more operands. ANSI mode is pinned
+    // because that is what turns the extra evaluation into a failure: with it off the division
+    // returns null and both answers hold either way.
     withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "true",
       SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true",
       SQLConf.SUBEXPRESSION_ELIMINATION_SKIP_FOR_SHORTCUT_EXPR.key -> "true") {
       val andQuery =
@@ -3354,6 +3357,29 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
     }
   }
 
+  test("SPARK-59579: subexpression elimination respects a conditional the shortcut peel reaches") {
+    // The peel walks the leading AND operands to reach the CASE WHEN / IF, which is the one operand
+    // that is always evaluated. `1 / id` is repeated inside a single branch body, so it is shared
+    // with no other branch and runs only when that branch does. For id = 0 the other branch runs,
+    // so `1 / id` should never be computed. Recursing into every child of the expression the peel
+    // lands on hoisted it out of the branch, computing it for id = 0 and raising [DIVIDE_BY_ZERO].
+    // ANSI mode is pinned on because that is what turns the extra evaluation into a failure: with
+    // it off the division returns null, the branch still yields false, and the test would pass
+    // either way in the scheduled non-ANSI build.
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "true",
+      SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true",
+      SQLConf.SUBEXPRESSION_ELIMINATION_SKIP_FOR_SHORTCUT_EXPR.key -> "true") {
+      val caseWhenQuery = "select (case when id = 0 then false " +
+        "else (1 / id + 1 / id) > 0 end) and id >= 0 from range(0, 1, 1, 1)"
+      checkAnswer(sql(caseWhenQuery), Row(false))
+
+      val ifQuery =
+        "select if(id = 0, false, (1 / id + 1 / id) > 0) and id >= 0 from range(0, 1, 1, 1)"
+      checkAnswer(sql(ifQuery), Row(false))
+    }
+  }
+
   test("SPARK-29213: FilterExec should not throw NPE") {
     // Under ANSI mode, casting string '' as numeric will cause runtime error
     if (!conf.ansiEnabled) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org