You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2026/09/14 07:29:06 UTC

(spark) branch master updated: [SPARK-59295][SQL] Emit a common expression's definition once per scope under whole-stage codegen

This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 87d38416f266 [SPARK-59295][SQL] Emit a common expression's definition once per scope under whole-stage codegen
87d38416f266 is described below

commit 87d38416f266f5bd2e8a589d9afa7575438b9126
Author: YangJie <ya...@baidu.com>
AuthorDate: Mon Sep 14 15:28:54 2026 +0800

    [SPARK-59295][SQL] Emit a common expression's definition once per scope under whole-stage codegen
    
    ### What changes were proposed in this pull request?
    
    `CodegenContext.CommonExprSlots.fill` emits a `With` common expression's definition once per scope by putting its body in a private method, but it could only do that where the definition read the input row (`INPUT_ROW != null && currentVars == null`), the condition `Expression.reduceCodeSize` splits under. A whole-stage `Project` or `Filter` passes its input as local variables instead, so there the body was pasted at every `CommonExpressionRef`, doubling per nesting level. This is the [...]
    
    This PR builds the method's parameter list, so the body goes into a method under whole-stage codegen too. The parameters are what the body would otherwise read from the scope the call replaces it in: the input row, and an input variable the operator evaluated before generating this expression. Everything else the body reads is a field -- the slots, a lambda variable, an expression's own mutable state -- and needs nothing passed. Collecting them follows the same laziness the generated  [...]
    
    Four shapes are refused instead, and keep the inline body they have today:
    
    - A definition reading an input variable the operator has **not** evaluated yet, which is any attribute a `Project`'s list mentions only once. That variable's code was generated against the scope of the operator producing the row and names a local of it -- the column batch's row index, the input adapter's row -- so it cannot travel into the method. Hoisting it to before the call, the way `getLocalInputVariableValues` does for subexpression elimination, would evaluate it on rows that r [...]
    - A value whose name is not a Java identifier. `ExpandExec` allocates a varying output column as mutable state but hands it out as a local, and for a non-primitive type that name is a slot of the compacted array, `mutableStateArray_0[3]`.
    - A value that is an expression rather than a name, where there is no way to tell which locals the expression names: `GenerateExec` gives the position of `posexplode_outer` the nullness `index == -1`, and a `Byte` or `Short` literal's value is `(byte)1`.
    - A definition holding a node subexpression elimination has computed. `Expression.genCode` reads such a node off its state, while `Alias`, `Collate` and an identity `Cast` override `genCode` and generate their child again, so which values the body reads there cannot be told from the tree. Refusing also keeps the parameter list inside what `getLocalInputVariableValues` collects, which stops at a state and computes the parameters of the methods `ExpandExec` and the aggregates move this  [...]
    
    A parameter list too long for a valid method descriptor falls back the same way.
    
    ### Why are the changes needed?
    
    Generated code size. For a `With` nested d levels deep, each level reading its definition twice, the whole-stage source of the expression was:
    
    | depth | before | after |
    |---|---|---|
    | 2 | 2087 | 1315 |
    | 3 | 4619 | 1810 |
    | 4 | 9755 | 2314 |
    | 5 | 19979 | 2812 |
    
    Copies of the innermost body: 2, 4, 8, 16, 32 before, and 2 at every depth after. Exponential growth reaches the method size limit a few levels further on, and dropping the whole stage costs far more than the one call per row this adds.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The call replaces the body inside the same `if (!computed)` guard, so a definition is still evaluated at most once per row, at the first reference reached; only the generated Java differs.
    
    ### How was this patch tested?
    
    New cases in `WithExpressionEvalSuite` and `WholeStageCodegenSuite`. In the first, one test generates against the context a whole-stage operator sets up and asserts the innermost body is emitted twice at any depth, that an evaluated variable and its nullness arrive as parameters, and that a definition no reference reaches neither contributes a parameter nor refuses the method; a second covers the shapes a method is not possible for, one case each, asserting the inline body doubled per [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #58665 from LuciferYang/SPARK-59295.
    
    Authored-by: YangJie <ya...@baidu.com>
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 .../spark/sql/catalyst/expressions/With.scala      |   3 +-
 .../expressions/codegen/CodeGenerator.scala        | 146 ++++++++++++----
 .../expressions/WithExpressionEvalSuite.scala      | 193 +++++++++++++++++++--
 .../sql/execution/WholeStageCodegenSuite.scala     |  68 +++++++-
 4 files changed, 358 insertions(+), 52 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/With.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/With.scala
index 6f7467005369..85388c271885 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/With.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/With.scala
@@ -425,8 +425,7 @@ case class CommonExpressionRef(id: CommonExpressionId, dataType: DataType, nulla
    * body encloses another, so repeating it declares nothing twice in one scope: a definition
    * reading a sibling is generated inside that sibling's guard, not inside its own, and one that
    * reaches its own id -- directly or around a cycle -- re-enters `fill` and is refused before any
-   * Java exists. See `CommonExprSlots.fill`, which also says what bounds the code when a method is
-   * not possible.
+   * Java exists. See `CommonExprSlots.fill`.
    */
   override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
     val slots = ctx.getCommonExpr(id.id)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
index 4cacc6efc055..0cf8096081f6 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
@@ -236,13 +236,10 @@ class CodegenContext extends Logging {
      * `filling` catches a definition that references its own id, which would otherwise re-enter and
      * recurse, since `fillCode` is set only after `definition.genCode` returns.
      *
-     * The body goes into a method where it can and is worth it -- a definition that is or holds
-     * another `With`, or a body past the split threshold -- so it is emitted once per scope rather
-     * than once per reference, which for nested `With`s would double per level. A method is only
-     * possible where the definition reads the input row rather than local variables, the condition
-     * `reduceCodeSize` splits under. That is not the same as whole-stage codegen being off: a
-     * whole-stage `Project` or `Filter` passes local variables, while
-     * `SortMergeJoinExec.createJoinKey` and the aggregate output paths generate against a row.
+     * The body goes into a method where it is worth it -- a definition that is or holds another
+     * `With`, or a body past the split threshold -- so it is emitted once per scope rather than
+     * once per reference, which for nested `With`s would double per level. `methodArgs` says what
+     * the method takes, and which definitions cannot have one.
      */
     def fill: Block = {
       if (fillCode.isEmpty) {
@@ -278,38 +275,117 @@ class CodegenContext extends Logging {
          |${value.value} = ${defGen.value};
          |$computed = true;
        """.stripMargin
-      // TODO(SPARK-59295): cover the local-variable case too, by passing the `currentVars` values a
-      //   definition reads into the method as parameters, the way
-      //   `subexpressionEliminationForWholeStageCodegen` does. It needs a decision first:
-      //   `getLocalInputVariableValues` hoists an input variable that is not evaluated yet to
-      //   before the call, which for a reference behind a branch means evaluating it on rows that
-      //   never reach the reference.
-      val canPutInMethod = INPUT_ROW != null && currentVars == null
-      // A definition that is or holds another `With` is the shape whose code doubles per level,
-      // and what this is aimed at. It is not the only one -- a definition referencing a sibling
-      // definition of the same `With` doubles the same way, and codegen accepts that, since the
-      // sibling's slots are in scope while this definition is generated (`With.refsToBind` says
-      // why nothing builds that tree, and that evaluating one raises). What bounds those is not
-      // the length arm below: `body` is assembled after `definition.genCode` already ran
-      // `reduceCodeSize`, so the arm fires only in the band just under the threshold. It is
-      // `reduceCodeSize` itself, which hoists whichever node's code first passes the threshold as
-      // generation walks up, capping what one level contributes, so the code stays linear in the
-      // depth either way. The length arm just keeps the same body from being split once per
-      // reference, which leaves the methods small and the code as large.
+      // A definition that is or holds another `With` is the shape whose code doubles per level, and
+      // what this is aimed at. A definition reading a sibling definition of the same `With` doubles
+      // the same way, but not through this arm: a `CommonExpressionRef` carries `COMMON_EXPR_REF`,
+      // not `WITH_EXPRESSION`, so a short one reaches a method only through the length arm. Nothing
+      // builds that tree today (`With.refsToBind` says why). The length arm keeps one body from
+      // being split once per reference; it fires in the band just under the threshold where
+      // `reduceCodeSize` applies, since `body` is assembled after `definition.genCode` already ran
+      // it.
       val worthAMethod = definition.containsPattern(WITH_EXPRESSION) ||
         body.length > SQLConf.get.methodSplitThreshold
-      if (canPutInMethod && worthAMethod) {
-        val funcName = freshName("computeCommonExpr")
-        val funcFullName = addNewFunction(funcName,
-          s"""
-             |private void $funcName(InternalRow $INPUT_ROW) {
-             |  $body
-             |}
+      (if (worthAMethod) methodArgs else None) match {
+        case Some(args) =>
+          val funcName = freshName("computeCommonExpr")
+          val params = args.map(a => s"${typeName(a.javaType)} ${a.variableName}").mkString(", ")
+          val funcFullName = addNewFunction(funcName,
+            s"""
+               |private void $funcName($params) {
+               |  $body
+               |}
            """.stripMargin)
-        code"$funcFullName($INPUT_ROW);"
-      } else {
-        body
+          code"$funcFullName(${args.map(_.variableName).mkString(", ")});"
+        case None =>
+          body
+      }
+    }
+
+    /**
+     * The locals to pass the method, or None where a method is not possible. What it collects are
+     * the values the body would otherwise read from the scope the call replaces it in: the input
+     * row, and an input variable the operator evaluated before generating this expression. Those
+     * variables are declared by code the operator emits ahead of this expression, so they enclose
+     * every reference to this definition; the input row is taken on the operator's word that
+     * `INPUT_ROW` names something in scope where it has this expression generated, which is what
+     * `Expression.reduceCodeSize` takes it on as well.
+     *
+     * A definition that reads an input variable the operator has *not* evaluated yet gets no
+     * method: that variable's code cannot travel into one, since it was generated by the operator
+     * producing the row, against that operator's scope, so it names a local of that scope -- the
+     * column batch's row index, or the input adapter's row. Nor can it be hoisted to before the
+     * call, the way `getLocalInputVariableValues` does for subexpression elimination, since that
+     * evaluates it on rows that reach no reference.
+     *
+     * Nor does a definition that is or holds a node subexpression elimination has computed, for
+     * which see the walk below. The two remaining refusals are local: `canPass` on a value no
+     * parameter can name, and the descriptor length at the end.
+     */
+    private def methodArgs: Option[Seq[VariableValue]] = {
+      val args = mutable.LinkedHashMap.empty[String, VariableValue]
+      // False for a value no parameter can carry: `ExpandExec` hands out a `VariableValue` naming a
+      // slot of a compacted mutable state array, and a `SimpleExprValue` is an expression rather
+      // than a name -- `posexplode_outer` gives its position the nullness `index == -1`, and a
+      // `Byte` or `Short` literal's value is `(byte)1`. A field or a literal needs no parameter and
+      // is read as it stands.
+      def canPass(v: ExprValue): Boolean = v match {
+        case local: VariableValue =>
+          val name = local.variableName
+          val isName = name.nonEmpty && Character.isJavaIdentifierStart(name.head) &&
+            name.forall(Character.isJavaIdentifierPart)
+          if (isName) {
+            args.getOrElseUpdate(name, local)
+          }
+          isName
+        case _: GlobalValue | _: LiteralValue => true
+        case _ => false
+      }
+      var possible = INPUT_ROW == null ||
+        canPass(JavaCode.variable(INPUT_ROW, classOf[InternalRow]))
+      val visited = mutable.HashSet.empty[Long]
+      // The definitions of a `With` in the tree walked here, which reach `currentCommonExprs` only
+      // once that `With` is generated. Ids come from one counter, so one map serves every scope.
+      val nestedDefs = mutable.HashMap.empty[Long, Expression]
+      val toVisit = mutable.Stack[Expression](definition)
+      while (possible && toVisit.nonEmpty) {
+        val next = toVisit.pop()
+        // A node subexpression elimination has computed keeps the definition inline. Which values
+        // the body reads there cannot be told from the tree: `Expression.genCode` reads the state,
+        // for a `With` as much as anything else since `With` overrides only `doGenCode`, while
+        // `Alias`, `Collate` and an identity `Cast` override `genCode` and generate their child
+        // again -- two different parameter lists. Refusing also keeps what this collects within
+        // what `getLocalInputVariableValues` collects for the operator's whole expression, which
+        // stops at a state as well and computes the parameters of the methods `ExpandExec` and the
+        // aggregates move this call into.
+        if (subExprEliminationExprs.contains(ExpressionEquals(next))) {
+          possible = false
+        } else {
+          next match {
+            case ref: BoundReference if currentVars != null && currentVars(ref.ordinal) != null =>
+              val input = currentVars(ref.ordinal)
+              possible = input.code == EmptyBlock && canPass(input.value) && canPass(input.isNull)
+            case w: With =>
+              // Only `child` is generated: a definition is generated where a reference reaches it,
+              // so one no reference reaches is not in the body, and what it reads decides nothing.
+              w.defs.foreach(d => nestedDefs.put(d.id.id, d.child))
+              toVisit.push(w.child)
+            case ref: CommonExpressionRef =>
+              // The definition this reference fills, which it does inside this method, so what
+              // that definition reads has to come in as well. One belonging to a `With` in the
+              // tree walked here is in `nestedDefs`; a sibling of this definition, or one of an
+              // enclosing scope, is registered in `currentCommonExprs`.
+              if (visited.add(ref.id.id)) {
+                nestedDefs.get(ref.id.id)
+                  .orElse(currentCommonExprs.get(ref.id.id).map(_.definition))
+                  .foreach(toVisit.push)
+              }
+            case e => toVisit.pushAll(e.children)
+          }
+        }
       }
+      val params = args.values.toSeq
+      Option.when(
+        possible && isValidParamLength(calculateParamLengthFromExprValues(params)))(params)
     }
   }
 
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
index a19f6e987adb..fd9a42a14bcd 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
@@ -19,7 +19,8 @@ package org.apache.spark.sql.catalyst.expressions
 
 import org.apache.spark.{SparkException, SparkFunSuite}
 import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, CodegenFallback, ExprCode, GenerateMutableProjection}
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, CodegenFallback, EmptyBlock, ExprCode, FalseLiteral, GenerateMutableProjection, JavaCode, SubExprEliminationState}
+import org.apache.spark.sql.catalyst.expressions.codegen.Block._
 import org.apache.spark.sql.catalyst.plans.SQLHelper
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types.{BooleanType, DataType, IntegerType}
@@ -62,6 +63,48 @@ class WithExpressionEvalSuite extends SparkFunSuite with SQLHelper {
       copy(child = newChild)
   }
 
+  /** Carried by the innermost body of [[nested]], so counting it counts the copies of that body. */
+  private val marker = 1234567
+
+  private def markersIn(source: String): Int = marker.toString.r.findAllMatchIn(source).size
+
+  /**
+   * A `With` nested `depth` levels deep, each level reading its definition twice: pasting a body
+   * doubles it per level, and putting it in a method leaves the innermost one emitted twice.
+   */
+  private def nested(depth: Int, nullable: Boolean = false): Expression = {
+    val leaf: Expression = Add(BoundReference(0, IntegerType, nullable), Literal(marker))
+    (1 to depth).foldLeft(leaf) { (inner, _) =>
+      With(inner) { case Seq(ref) => Add(ref, ref) }
+    }
+  }
+
+  /**
+   * The context a whole-stage `Project` or `Filter` sets up: no input row, the input as local
+   * variables.
+   */
+  private def localInputContext(inputs: ExprCode*): CodegenContext = {
+    val ctx = new CodegenContext
+    ctx.INPUT_ROW = null
+    ctx.currentVars = inputs
+    ctx
+  }
+
+  /** Generates `e` against [[localInputContext]]: the code it emitted, and the methods added. */
+  private def generateWithLocalInput(e: Expression, inputs: ExprCode*): (String, String) = {
+    val ctx = localInputContext(inputs: _*)
+    (e.genCode(ctx).code.toString, ctx.declareAddedFunctions())
+  }
+
+  /**
+   * An input variable as `currentVars` carries it. `evaluated` is whether the operator emitted its
+   * code before generating this expression, which is what decides whether a method can take it.
+   */
+  private def intVar(name: String = "v0", evaluated: Boolean): ExprCode = {
+    val code = if (evaluated) EmptyBlock else code"int $name = i.getInt(0);"
+    ExprCode(code, FalseLiteral, JavaCode.variable(name, IntegerType))
+  }
+
   test("a definition is evaluated once per row however many references read it") {
     // `ref + ref` is the shape `BETWEEN` produces. Memoized, both references read one value, so the
     // sum is 2n on the nth row; inlined it would be n + (n + 1).
@@ -168,21 +211,12 @@ class WithExpressionEvalSuite extends SparkFunSuite with SQLHelper {
 
   test("SPARK-58902: nesting does not multiply a definition's body when it can go in a method") {
     // Each level reads its definition twice, so pasting the body doubles it per level;
-    // `CommonExprSlots.fill` putting it in a method makes it 2 at any depth. Not covered here: the
-    // shape where the input arrives as local variables, as a whole-stage `Project` or `Filter`
-    // passes it, where no method is possible and the count is 2, 4, 8, ... 256 at depths 1 to 8 --
-    // see SPARK-59295. A bare `CodegenContext` has `INPUT_ROW` set and `currentVars` null.
-    val marker = 1234567
-    def nested(depth: Int): Expression = {
-      val leaf: Expression = Add(BoundReference(0, IntegerType, nullable = false), Literal(marker))
-      (1 to depth).foldLeft(leaf) { (inner, _) =>
-        With(inner) { case Seq(ref) => Add(ref, ref) }
-      }
-    }
+    // `CommonExprSlots.fill` putting it in a method makes it 2 at any depth. A bare
+    // `CodegenContext` has `INPUT_ROW` set and `currentVars` null; the case below covers the input
+    // arriving as local variables instead.
     def markerCount(depth: Int): Int = {
       val ctx = new CodegenContext
-      val source = nested(depth).genCode(ctx).code.toString + ctx.declareAddedFunctions()
-      marker.toString.r.findAllMatchIn(source).size
+      markersIn(nested(depth).genCode(ctx).code.toString + ctx.declareAddedFunctions())
     }
     // Pin the threshold: below the innermost body's length the fill would go into a method of its
     // own and the count would be 1, for a reason unrelated to nesting.
@@ -203,6 +237,137 @@ class WithExpressionEvalSuite extends SparkFunSuite with SQLHelper {
     }
   }
 
+  test("SPARK-59295: a method for the local variables the operator has already evaluated") {
+    // What a whole-stage `Project` or `Filter` passes: no input row, the input as local variables.
+    // A method is possible for what the operator has already evaluated; the shapes it is not
+    // possible for are the case below.
+    withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
+      (1 to 6).foreach { depth =>
+        val (body, methods) = generateWithLocalInput(nested(depth), intVar(evaluated = true))
+        val count = markersIn(body + methods)
+        assert(count == 2, s"the innermost body was emitted $count times at depth $depth")
+      }
+      // The variable is declared in the scope the call sits in, so the method takes it as a
+      // parameter rather than reading it as a field. A nullable one carries its nullness in a
+      // second local, which the method needs just as much.
+      val (inline, methods) = generateWithLocalInput(nested(2), intVar(evaluated = true))
+      assert("private void computeCommonExpr_[0-9]+\\(int v0\\)".r
+        .findFirstIn(methods).isDefined, methods)
+      assert("computeCommonExpr_[0-9]+\\(v0\\);".r.findFirstIn(inline).isDefined, inline)
+      val (nullableInline, nullableMethods) = generateWithLocalInput(
+        nested(2, nullable = true),
+        ExprCode(EmptyBlock, JavaCode.isNullVariable("v0IsNull"),
+          JavaCode.variable("v0", IntegerType)))
+      assert("private void computeCommonExpr_[0-9]+\\(int v0, boolean v0IsNull\\)".r
+        .findFirstIn(nullableMethods).isDefined, nullableMethods)
+      assert("computeCommonExpr_[0-9]+\\(v0, v0IsNull\\);".r
+        .findFirstIn(nullableInline).isDefined, nullableInline)
+
+      // A definition reading an enclosing scope's reference needs what that definition reads, which
+      // is behind a `CommonExpressionRef` rather than in the tree being walked.
+      val outer = With(BoundReference(0, IntegerType, nullable = false)) { case Seq(outerRef) =>
+        With(With(outerRef) { case Seq(ref) => Add(ref, Literal(marker)) }) {
+          case Seq(ref) => Add(ref, ref)
+        }
+      }
+      val (_, enclosingMethods) = generateWithLocalInput(outer, intVar(evaluated = true))
+      assert("private void computeCommonExpr_[0-9]+\\(int v0\\)".r
+        .findFirstIn(enclosingMethods).isDefined, enclosingMethods)
+
+      // A definition no reference reaches is never generated, so what it reads decides nothing:
+      // only `v0` is read by a body here, and the one method takes just that. Walking every
+      // definition of the nested `With` instead would find the unevaluated `v1` and refuse it.
+      val unreferenced = With(
+        BoundReference(0, IntegerType, nullable = false),
+        Add(BoundReference(1, IntegerType, nullable = false), Literal(marker))) {
+        case Seq(live, _) => Add(live, live)
+      }
+      val (unreferencedBody, unreferencedMethods) = generateWithLocalInput(
+        With(unreferenced) { case Seq(ref) => Add(ref, ref) },
+        intVar(evaluated = true),
+        intVar("v1", evaluated = false))
+      assert("private void computeCommonExpr_[0-9]+\\(int v0\\)".r
+        .findAllMatchIn(unreferencedMethods).size == 1, unreferencedMethods)
+      assert(markersIn(unreferencedBody + unreferencedMethods) == 0,
+        "the unreferenced definition was generated")
+    }
+  }
+
+  test("SPARK-59295: the shapes a method is not possible for keep the inline body") {
+    withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
+      // An unevaluated variable: `int v0 = i.getInt(0);` names `i`, the producing operator's local,
+      // which is not in scope inside a method here. So the bodies stay inline and double per level,
+      // as they did before a method was possible at all.
+      (1 to 4).foreach { depth =>
+        val (body, methods) = generateWithLocalInput(nested(depth), intVar(evaluated = false))
+        val count = markersIn(body + methods)
+        assert(count == (1 << depth),
+          s"the innermost body was emitted $count times at depth $depth")
+        assert(!methods.contains("computeCommonExpr"), methods)
+      }
+      // A value whose name is not one a parameter can take: `ExpandExec` hands out a compacted
+      // mutable state array slot as if it were a local. The marker count says the fallback pasted
+      // the body at both references; that a method was attempted at all is the case above, which
+      // gets one from the same shape with a passable name.
+      val slot = ExprCode(EmptyBlock, FalseLiteral,
+        JavaCode.variable("mutableStateArray_0[3]", IntegerType))
+      val (slotBody, slotMethods) = generateWithLocalInput(nested(2), slot)
+      assert(!slotMethods.contains("computeCommonExpr"), slotMethods)
+      assert(markersIn(slotBody + slotMethods) == 4, slotBody + slotMethods)
+      // A nullness that is an expression rather than a name, the shape `GenerateExec` hands out
+      // for the position of `posexplode_outer`: whichever locals such an expression names, this
+      // has no way to pass them. A plain name in its place does get a method, in the case above.
+      val position = ExprCode(EmptyBlock, JavaCode.isNullExpression("index_0 == -1"),
+        JavaCode.variable("index_0", IntegerType))
+      val (positionBody, positionMethods) =
+        generateWithLocalInput(nested(2, nullable = true), position)
+      assert(!positionMethods.contains("computeCommonExpr"), positionMethods)
+      assert(markersIn(positionBody + positionMethods) == 4, positionBody + positionMethods)
+    }
+    // More parameters than a method descriptor can carry. The limit is 255 words, which takes an
+    // operator handing out that many variables, so the conf the other splitting paths test this
+    // with stands in for it.
+    withSQLConf(
+        SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024",
+        "spark.sql.CodeGenerator.validParamLength" -> "0") {
+      val (body, methods) = generateWithLocalInput(nested(2), intVar(evaluated = true))
+      assert(!methods.contains("computeCommonExpr"), methods)
+      val count = markersIn(body + methods)
+      assert(count == 4, s"the innermost body was emitted $count times")
+    }
+  }
+
+  test("SPARK-59295: a field needs no parameter, a subexpression elimination state no method") {
+    withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
+      // An input variable the operator allocated as mutable state, which is what the aggregate
+      // paths hand out for their buffer and result variables: a field is in scope in the method.
+      val field = ExprCode(EmptyBlock, FalseLiteral, JavaCode.global("aggValue_0", IntegerType))
+      val (_, fieldMethods) = generateWithLocalInput(nested(2), field)
+      assert("private void computeCommonExpr_[0-9]+\\(\\)".r
+        .findFirstIn(fieldMethods).isDefined, fieldMethods)
+
+      // A definition holding a node subexpression elimination has computed keeps the inline body.
+      // The node here is the definition itself, so `Expression.genCode` reads the state and the
+      // body assigns the slots from its two locals; one whose node were an `Alias` or a `Collate`
+      // would read the regenerated subtree instead, and the walk cannot tell which it would be. The
+      // state's code is unemitted here, as it is where `ProjectExec.doConsume` generates.
+      val definition = nested(1, nullable = true)
+      val state = SubExprEliminationState(
+        ExprCode(code"boolean subIsNull = false;\nint subValue = 1;",
+          JavaCode.isNullVariable("subIsNull"), JavaCode.variable("subValue", IntegerType)))
+      val ctx = localInputContext(intVar(evaluated = true))
+      val body = ctx.withSubExprEliminationExprs(Map(ExpressionEquals(definition) -> state)) {
+        Seq(With(definition) { case Seq(ref) => Add(ref, ref) }.genCode(ctx))
+      }.head.code.toString
+      val methods = ctx.declareAddedFunctions()
+      assert(!methods.contains("computeCommonExpr"), methods)
+      // The state was read rather than the definition generated: no marker anywhere below it, and
+      // the slot assigned from the state's value.
+      assert(markersIn(body) == 0, body)
+      assert(body.contains("= subValue;") && body.contains("= subIsNull;"), body)
+    }
+  }
+
   test("SPARK-58902: a nested With sharing a reference object restores the outer binding") {
     // Two `With`s over one reference object, the inner one nested in the outer one's child and
     // redefining the id the outer one defines. Only a caller building the case class directly
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
index 6af7d066fccd..4402b5558cc9 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
@@ -23,7 +23,7 @@ import java.time.Duration
 import org.apache.spark.SparkException
 import org.apache.spark.rdd.MapPartitionsWithEvaluatorRDD
 import org.apache.spark.sql.{Dataset, Row, SaveMode}
-import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Cast, CodegenObjectFactoryMode, Expression, IsNotNull}
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Cast, CodegenObjectFactoryMode, Expression, IsNotNull, With}
 import org.apache.spark.sql.catalyst.expressions.codegen.{ByteCodeStats, CodeAndComment, CodeGenerator}
 import org.apache.spark.sql.execution.adaptive.DisableAdaptiveExecutionSuite
 import org.apache.spark.sql.execution.aggregate.{HashAggregateExec, SortAggregateExec}
@@ -1733,4 +1733,70 @@ class WholeStageCodegenSuite extends SharedSparkSession
     assert(sinhPattern.findAllIn(inlineCode).length == 1,
       "sinh(v) should be evaluated only once per input row without function splitting")
   }
+
+  test("SPARK-59295: a nested With in a branch is emitted once per scope under whole-stage") {
+    // A whole-stage `Project` passes its input as local variables, which is where
+    // `CommonExprSlots.fill` could not put a definition in a method before. Both `nullif`s survive
+    // the rewrite inside the branch and each reads its definition twice, so pasting the bodies
+    // would leave the innermost one 4 times over. The project list mentions `a` and `b` more than
+    // once, which is what has `ProjectExec` evaluate them before generating this expression -- the
+    // case below covers the other one.
+    val marker = 1234567
+    val df = spark.range(0, 10, 1, 1).selectExpr("cast(id as int) as a", "cast(id as int) + 1 as b")
+    val query = df.selectExpr(
+      s"CASE WHEN a < 0 THEN NULL ELSE nullif(nullif(a + b + $marker, b), a) END AS r")
+    assert(query.queryExecution.executedPlan.exists(_.expressions.exists(_.exists {
+      case _: With => true
+      case _ => false
+    })), "the rewrite left no With to generate")
+    // Pin the threshold: below the innermost body's length that body would go into a method of its
+    // own and the count would be 1, for a reason unrelated to nesting.
+    val source = withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
+      genCode(query).map(_.body).mkString("\n")
+    }
+    val emitted = marker.toString.r.findAllMatchIn(source).size
+    assert(emitted == 2, s"the innermost definition was emitted $emitted times")
+    // One method for the outer definition, called at both of its references. The name carries the
+    // operator's fresh-name prefix, as in `project_computeCommonExpr_0`.
+    val declared = "private void \\w*computeCommonExpr_[0-9]+\\(".r.findAllMatchIn(source).size
+    assert(declared == 1, source)
+    // Runs it too, so the generated source above is known to compile.
+    checkAnswer(query, (0 until 10).map(i => Row(2 * i + marker + 1)))
+  }
+
+  test("SPARK-59295: a definition reading an input variable the operator has not evaluated") {
+    // Such a variable's code is written against the scope of the operator producing the row and
+    // names a local of it -- here the column batch's row index, and after an exchange the input
+    // adapter's row -- so it cannot go into a method along with the body that reads it. `b` is read
+    // once and only inside a definition, which is what leaves it unevaluated: `ProjectExec`
+    // evaluates an attribute up front only where the project list mentions it more than once.
+    val marker = 1234567
+    withTempPath { dir =>
+      val path = dir.getCanonicalPath
+      spark.range(0, 10).selectExpr("cast(id as int) as a", "cast(id as int) + 1 as b")
+        .write.mode(SaveMode.Overwrite).parquet(path)
+      val query = spark.read.parquet(path).selectExpr(
+        s"CASE WHEN a < 0 THEN NULL ELSE nullif(nullif(a + $marker, b), a) END AS r")
+      assert(query.queryExecution.executedPlan.exists(_.expressions.exists(_.exists {
+        case _: With => true
+        case _ => false
+      })), "the rewrite left no With to generate")
+      // Pin the threshold for the two assertions below: under the inner definition's length that
+      // body would go into a method of its own, whatever this case is about.
+      val code = withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
+        genCode(query)
+      }
+      // Compiling it is the check: before this was refused, the method named the row index of the
+      // column batch and janino rejected it, which drops the whole stage.
+      code.foreach(CodeGenerator.compile)
+      val source = code.map(_.body).mkString("\n")
+      assert(!source.contains("computeCommonExpr"),
+        "the definition reads an unevaluated variable, so it cannot go in a method")
+      // Which leaves the inner definition pasted at its two references inside the outer one, itself
+      // pasted at two: 4, against the 2 the case above gets from a method.
+      val emitted = marker.toString.r.findAllMatchIn(source).size
+      assert(emitted == 4, s"the innermost definition was emitted $emitted times")
+      checkAnswer(query, (0 until 10).map(i => Row(i + marker)))
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org