You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/21 16:38:48 UTC
(spark) branch branch-4.3 updated: [SPARK-59579][4.3][SQL] Keep the conditional-branch guards after the subexpression elimination shortcut peel
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 7fb067473a60 [SPARK-59579][4.3][SQL] Keep the conditional-branch guards after the subexpression elimination shortcut peel
7fb067473a60 is described below
commit 7fb067473a60f32837315390b70a3db3ea086689
Author: YangJie <ya...@baidu.com>
AuthorDate: Mon Sep 21 09:38:37 2026 -0700
[SPARK-59579][4.3][SQL] Keep the conditional-branch guards after the subexpression elimination shortcut peel
### What changes were proposed in this pull request?
This backports #58868 to `branch-4.3`. One thing differs from the master change: `childrenToRecurse` here has no `With` case, because SPARK-58818 is not on this branch, so there is no after-peel re-check to remove and the `With` half of the master change does not apply. The rest is the same.
`EquivalentExpressions.childrenToRecurse` refuses to descend into children that must not be evaluated ahead of time: a `CodegenFallback` generates no code for them, a `ConditionalExpression` offers only its `alwaysEvaluatedInputs`, and a `HigherOrderFunction` only its always-evaluated arguments.
With `spark.sql.subexpressionElimination.skipForShortcutExpr` on, `skipForShortcut` peels the leading `And`/`Or` operands to reach the one operand of the chain that is always evaluated. `And`/`Or` are not `ConditionalExpression`s, so that peel walked straight past every case above and the result was used as `peeled.children`, recursing into all of the peeled expression's children, conditional branches included.
The always-evaluated operand is now computed once in `updateExprTree`, and both `childrenToRecurse` and `commonChildrenToRecurse` are asked about that operand. Asking `commonChildrenToRecurse` recovers what the peel used to drop: the branch groups of a conditional the peel lands on, whose shared subexpression is safe to evaluate once because the conditional itself always runs.
`childrenToRecurse` still peels the always-evaluated inputs of a `ConditionalExpression`. That peel no longer keeps the descent safe: `updateExprTree` peels before it descends, so `childrenToRecurse` is asked about the peeled node either way. It stays because dropping it would pass `updateExprTree` the input itself rather than the operand its peel lands on.
One consequence beyond stopping the leak: this branch reached a peeled-to conditional's branches by recursing into all of its children, so a subexpression shared by every branch picked up a use count of 2 and was eliminated. It now goes through the branch-group intersection, which contributes 1, and is eliminated only if it also occurs in an always-evaluated position, which is how a conditional met directly has always behaved. Where each branch holds the subexpression once, that elimi [...]
Nothing changes while the config is off, where `skipForShortcut` returns its argument unchanged and the recursion meets the conditional directly, guards and all.
### Why are the changes needed?
The config exists to stop a short-circuited operand from being evaluated eagerly, and it made a conditional branch behind that operand eligible instead. With ANSI mode and both subexpression elimination configs on, `select (case when id = 0 then false else (1 / id + 1 / id) > 0 end) and id >= 0 from range(0, 1, 1, 1)` raised `[DIVIDE_BY_ZERO]`. `1 / id` is repeated inside a single branch body, is shared with no other branch, and should run only when that branch runs; for id = 0 the ot [...]
The bypass reaches the `HigherOrderFunction` case too, where the cost is a failure rather than an early evaluation: a subexpression repeated inside a lambda body becomes a candidate, `supportedExpression` does not stop it because `NamedLambdaVariable` carries no `LAMBDA_VARIABLE` pattern, and generating that candidate at the top of the projection leaves `CodegenContext.getLambdaVar` with no variable to bind. No query was found where that changes an answer, so that half closes a hole r [...]
### Does this PR introduce _any_ user-facing change?
Yes, for a query run with `spark.sql.subexpressionElimination.skipForShortcutExpr` enabled, which is off by default: the query above returns `false` where it used to raise `[DIVIDE_BY_ZERO]`. No change with the config off.
### How was this patch tested?
`SubexpressionEliminationSuite` pins that no subexpression from a conditional branch, a `CodegenFallback`'s children or a lambda body is eliminated, with each shape met directly and behind an `And`, an `Or` chain and a mixed chain. Two positive controls sit beside it: the operand the peel lands on still contributes its own duplicates, and a duplicate shared by every branch of an `If` or `CaseWhen` group is still eliminated once, with the use count pinning that it came from the group r [...]
`SQLQuerySuite` covers the query above, in both its `CASE WHEN` and `IF` spellings. ANSI mode is pinned on because that is what turns the extra evaluation into a failure; with it off the division returns null, the branch still yields false, and the case would pass either way in the scheduled non-ANSI build. The SPARK-58211 case beside it gains the same pin, which it was missing for the same reason.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #58935 from LuciferYang/SPARK-59579-4.3.
Authored-by: YangJie <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../expressions/EquivalentExpressions.scala | 20 ++++-
.../SubexpressionEliminationSuite.scala | 89 ++++++++++++++++++++++
.../scala/org/apache/spark/sql/SQLQuerySuite.scala | 28 ++++++-
3 files changed, 132 insertions(+), 5 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
index 7722d5611670..5e82a00a6689 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
@@ -155,15 +155,21 @@ class EquivalentExpressions(
// 3. HigherOrderFunction: lambda functions operate in the context of local lambdas and can't
// be called outside of that scope, only always-evaluated arguments can be evaluated
// ahead of time.
+ // The caller has already applied `skipForShortcut`, so these cases are asked about the operand
+ // that the peel lands on rather than about the `And`/`Or` chain above it.
private def childrenToRecurse(expr: Expression): Seq[Expression] = expr match {
case _: CodegenFallback => Nil
+ // Peeling here is redundant for safety: `updateExprTree` peels before it descends, so this
+ // method sees the peeled node either way. It stays because dropping it would pass
+ // `updateExprTree` the input itself rather than the operand its peel lands on.
case c: ConditionalExpression => c.alwaysEvaluatedInputs.map(skipForShortcut)
case h: HigherOrderFunction => h.alwaysEvaluatedArguments
- case other => skipForShortcut(other).children
+ case other => other.children
}
// For some special expressions we cannot just recurse into all of its children, but we can
- // recursively add the common expressions shared between all of its children.
+ // recursively add the common expressions shared between all of its children. As with
+ // `childrenToRecurse`, the caller has already applied `skipForShortcut`.
private def commonChildrenToRecurse(expr: Expression): Seq[Seq[Expression]] = expr match {
case _: CodegenFallback => Nil
case c: ConditionalExpression => c.branchGroups
@@ -199,8 +205,14 @@ class EquivalentExpressions(
if (!skip && !updateExprInMap(expr, map, useCount)) {
val uc = useCount.sign
- childrenToRecurse(expr).foreach(updateExprTree(_, map, uc))
- commonChildrenToRecurse(expr).filter(_.nonEmpty).foreach(updateCommonExprs(_, map, uc))
+ // Peel first: which children may be recursed into is a question about the operand that is
+ // always evaluated, and `And`/`Or` are not `ConditionalExpression`s. Asking
+ // `childrenToRecurse` about the chain above that operand walks past its guards; asking
+ // `commonChildrenToRecurse` about the chain reaches no branch group at all.
+ val alwaysEvaluated = skipForShortcut(expr)
+ childrenToRecurse(alwaysEvaluated).foreach(updateExprTree(_, map, uc))
+ commonChildrenToRecurse(alwaysEvaluated).filter(_.nonEmpty)
+ .foreach(updateCommonExprs(_, map, uc))
}
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala
index 4f5525158af9..98ed67273174 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubexpressionEliminationSuite.scala
@@ -545,6 +545,95 @@ class SubexpressionEliminationSuite extends SparkFunSuite with ExpressionEvalHel
equivalence1.addExprTree(caseWhenExpr1)
assert(equivalence1.getCommonSubexpressions.size == 1)
}
+
+ test("SPARK-59579: the short-circuit peel keeps the guards of the operand it lands on") {
+ // `skipForShortcut` peels the leading `And`/`Or` operands down to the one operand that is
+ // always evaluated. That operand can be an expression whose children must not be recursed
+ // into, and `And`/`Or` are not `ConditionalExpression`s, so the peel walks right past the
+ // cases that stop the descent. What may be recursed into has to be asked about the operand
+ // the peel lands on, not about the `And`/`Or` chain it started from.
+ val a = AttributeReference("a", IntegerType)()
+ val add = Add(a, Literal(1))
+ // `add` occurs twice in one branch body and nowhere else, so no group of branches shares it:
+ // `branchGroups` intersects the branches of a group, so a body that shares nothing with the
+ // other branches contributes nothing. That is what makes the assertions below hold for the
+ // right reason rather than because nothing was recorded at all.
+ val body = GreaterThan(Multiply(add, add), Literal(0))
+ val otherBody = GreaterThan(a, Literal(1))
+ val lambdaVar = NamedLambdaVariable("x", IntegerType, nullable = false)
+ val lambdaAdd = Add(lambdaVar, Literal(1))
+ val guarded = Seq[Expression](
+ If(GreaterThan(a, Literal(0)), body, otherBody),
+ CaseWhen(Seq((GreaterThan(a, Literal(0)), body)), otherBody),
+ // Two branches, so that the conditions group is not empty either.
+ CaseWhen(
+ Seq((GreaterThan(a, Literal(0)), body), (LessThan(a, Literal(0)), otherBody)),
+ otherBody),
+ CodegenFallbackExpression(body),
+ // A candidate from a lambda body is a worse outcome than one evaluated too eagerly:
+ // generating it outside the lambda leaves `getLambdaVar` with no variable to bind, which
+ // throws. Nothing else stops it -- `supportedExpression` looks for the `LAMBDA_VARIABLE`
+ // pattern, which `NamedLambdaVariable` does not carry.
+ ArrayExists(
+ CreateArray(Seq(a)),
+ LambdaFunction(
+ GreaterThan(Multiply(lambdaAdd, lambdaAdd), Literal(0)),
+ Seq(lambdaVar)),
+ followThreeValuedLogic = true))
+
+ def statesOf(expr: Expression): Seq[ExpressionStats] = {
+ val equivalence = new EquivalentExpressions(skipForShortcutEnable = true)
+ equivalence.addExprTree(expr)
+ equivalence.getAllExprStates()
+ }
+
+ def assertNoneEliminated(expr: Expression, hint: String): Unit = {
+ val states = statesOf(expr)
+ // Guards the next assert against passing vacuously: `supportedExpression` would refuse the
+ // whole tree if `NamedLambdaVariable` ever carried the `LAMBDA_VARIABLE` pattern, and then
+ // nothing at all would be recorded.
+ assert(states.nonEmpty, s"nothing was recorded, $hint: $expr")
+ assert(states.forall(_.useCount == 1),
+ s"eliminated ${states.filter(_.useCount > 1).map(_.expr)}, $hint: $expr")
+ }
+
+ guarded.foreach { expr =>
+ // Met directly, the guards are the ones that were already there: `skipForShortcut` returns a
+ // node that is not an `And`/`Or` unchanged, so this line behaves as it did before. It is here
+ // so that a future change to the guards has to keep both cases in step.
+ assertNoneEliminated(expr, "met directly")
+ // Behind one, behind a chain of, and behind a mix of short-circuit operands is where the peel
+ // used to walk past the guards.
+ assertNoneEliminated(And(expr, otherBody), "behind an And")
+ assertNoneEliminated(Or(Or(expr, otherBody), otherBody), "behind an Or chain")
+ assertNoneEliminated(And(Or(expr, otherBody), otherBody), "behind a mixed chain")
+ }
+
+ def eliminated(expr: Expression): Seq[ExpressionStats] =
+ statesOf(expr).filter(_.useCount > 1)
+
+ // The operand the peel lands on is not treated as opaque: it still contributes its own
+ // duplicates when nothing guards it.
+ assert(eliminated(And(body, otherBody)).map(_.expr) == Seq(add))
+ // A duplicate shared by every branch of a group is still eliminated, one evaluation standing
+ // for whichever branch runs, as long as it occurs once more outside the group -- here in the
+ // condition, which is always evaluated. The use count is what pins this down: asking
+ // `branchGroups` about the `And` instead of about the operand the peel lands on leaves 1 and
+ // no elimination, and recursing into every branch as if it always ran gives 3. The `If` shape
+ // is here because `branchGroups` groups its two values, where `CaseWhen` groups conditions and
+ // values separately.
+ val shared = CaseWhen(
+ Seq((GreaterThan(add, Literal(0)), GreaterThan(add, Literal(1)))),
+ GreaterThan(add, Literal(2)))
+ val sharedIf =
+ If(GreaterThan(add, Literal(0)), GreaterThan(add, Literal(1)), GreaterThan(add, Literal(2)))
+ Seq[Expression](shared, And(shared, otherBody), sharedIf, And(sharedIf, otherBody))
+ .foreach { expr =>
+ val states = eliminated(expr)
+ assert(states.map(_.expr) == Seq(add), s"$expr")
+ assert(states.head.useCount == 2, s"$expr")
+ }
+ }
}
case class CodegenFallbackExpression(child: Expression)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 713fa39a9243..25184f52b219 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -3340,8 +3340,11 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
// is duplicated *inside* operand 2 (not split across operands 2 and 3) so the failure shape
// does not depend on how many operands the one-level peel happened to drop, and operand 3 is
// non-foldable to keep it in the plan. skipForShortcutExpr must peel every leading AND/OR
- // operand, not just one, to make this hold for three or more operands.
+ // operand, not just one, to make this hold for three or more operands. ANSI mode is pinned
+ // because that is what turns the extra evaluation into a failure: with it off the division
+ // returns null and both answers hold either way.
withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true",
SQLConf.SUBEXPRESSION_ELIMINATION_SKIP_FOR_SHORTCUT_EXPR.key -> "true") {
val andQuery =
@@ -3354,6 +3357,29 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
}
}
+ test("SPARK-59579: subexpression elimination respects a conditional the shortcut peel reaches") {
+ // The peel walks the leading AND operands to reach the CASE WHEN / IF, which is the one operand
+ // that is always evaluated. `1 / id` is repeated inside a single branch body, so it is shared
+ // with no other branch and runs only when that branch does. For id = 0 the other branch runs,
+ // so `1 / id` should never be computed. Recursing into every child of the expression the peel
+ // lands on hoisted it out of the branch, computing it for id = 0 and raising [DIVIDE_BY_ZERO].
+ // ANSI mode is pinned on because that is what turns the extra evaluation into a failure: with
+ // it off the division returns null, the branch still yields false, and the test would pass
+ // either way in the scheduled non-ANSI build.
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true",
+ SQLConf.SUBEXPRESSION_ELIMINATION_SKIP_FOR_SHORTCUT_EXPR.key -> "true") {
+ val caseWhenQuery = "select (case when id = 0 then false " +
+ "else (1 / id + 1 / id) > 0 end) and id >= 0 from range(0, 1, 1, 1)"
+ checkAnswer(sql(caseWhenQuery), Row(false))
+
+ val ifQuery =
+ "select if(id = 0, false, (1 / id + 1 / id) > 0) and id >= 0 from range(0, 1, 1, 1)"
+ checkAnswer(sql(ifQuery), Row(false))
+ }
+ }
+
test("SPARK-29213: FilterExec should not throw NPE") {
// Under ANSI mode, casting string '' as numeric will cause runtime error
if (!conf.ansiEnabled) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org