(spark) branch branch-4.x updated: [SPARK-59774][SQL] Warn once, and name the remedy, when a whole-stage codegen method is too long to be JIT compiled

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new e9323989a0a2 [SPARK-59774][SQL] Warn once, and name the remedy, when a whole-stage codegen method is too long to be JIT compiled
e9323989a0a2 is described below

commit e9323989a0a272befb2c22e600eced6e8e28531d
Author: Max Gekk <ma...@gmail.com>
AuthorDate: Fri Sep 25 12:02:05 2026 +0200

    [SPARK-59774][SQL] Warn once, and name the remedy, when a whole-stage codegen method is too long to be JIT compiled
    
    ### What changes were proposed in this pull request?
    
    `CodeCompiler.computeByteCodeStats` logs, at INFO, every generated method whose bytecode is past HotSpot's JIT limit of 8000 bytes (`CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT`). This PR raises that line to WARN in the one case where it costs the user and they can act on it, and only the first time, on the driver. All of these must hold:
    
    - the method belongs to a whole-stage codegen stage (`GeneratedIteratorForCodegenStage*`);
    - it runs per row, not once (not `<init>`, `<clinit>` or `init`);
    - the stage keeps whole-stage codegen: the method is at or below `spark.sql.codegen.hugeMethodLimit`, so the stage does not fall back;
    - the JVM leaves huge methods to the interpreter: `-XX:+DontCompileHugeMethods`, read through `Utils.getVMOptionValue`;
    - the method is compiled on the driver, not by a task: the driver compiles each stage before its tasks do.
    
    Every other report, including every one after the first WARN, keeps the existing INFO line unchanged.
    
    Before:
    ```
    INFO CodeGenerator: Generated method too long to be JIT compiled: <class>.<method> is 9513 bytes
    ```
    After, the first time on the driver:
    ```
    WARN CodeGenerator: Generated method too long to be JIT compiled: <class>.processNext is 9513 bytes, so it runs interpreted on every row. Setting spark.sql.codegen.hugeMethodLimit to 8000 runs such stages without whole-stage codegen instead. Further methods over the limit are logged at INFO.
    ```
    
    ### Why are the changes needed?
    
    HotSpot does not JIT-compile a method past 8000 bytes (`-XX:+DontCompileHugeMethods`, on by default), so the method runs interpreted, typically several times slower. On stock Spark 4.2.0, widening a projection's consume method from 7945 to 8677 bytes steps it from about 770 to about 4100 ns a row. Spark detects this, but only at INFO. `spark-shell` and `pyspark` run at WARN and never show it, and `spark.sql.codegen.hugeMethodLimit`, which could act on it, defaults to 65535, which no c [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, in the log only: at most one WARN, from the driver, and only in the case above. Everything else is unchanged.
    
    ### How was this patch tested?
    
    A new test in `CodeGenerationSuite` covers each case: a stage that falls back, the run-once methods, a class outside whole-stage codegen, a compile under a `TaskContext`, the first WARN (which names the setting and 8000), and INFO afterwards. The SPARK-25113 test asserts that a projection's huge method is still logged at INFO.
    ```
    build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.CodeGenerationSuite -- -z SPARK-25113 -z SPARK-59774"
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #59020 from MaxGekk/SPARK-59774.
    
    Authored-by: Max Gekk <ma...@gmail.com>
    Signed-off-by: Max Gekk <ma...@gmail.com>
    (cherry picked from commit 9b25ad22b3d2ee1236865f6b0b46578f9a92eb34)
    Signed-off-by: Max Gekk <ma...@gmail.com>
---
 .../expressions/codegen/CodeCompiler.scala         | 54 ++++++++++++++++++++--
 .../catalyst/expressions/CodeGenerationSuite.scala | 53 +++++++++++++++++++--
 2 files changed, 99 insertions(+), 8 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala
index 2a64f5d00713..72440d05fc19 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala
@@ -318,6 +318,55 @@ object CodeCompiler extends Logging {
     }
   }
 
+  /**
+   * Whether this JVM leaves a method past HotSpot's size limit to the interpreter: HotSpot's
+   * `-XX:+DontCompileHugeMethods`, on by default. False when it is off, or when the JVM is not
+   * HotSpot and has no such option.
+   */
+  private lazy val jitSkipsHugeMethods: Boolean =
+    Utils.getVMOptionValue("DontCompileHugeMethods").contains("true")
+
+  /** Generated methods that run once per class or per partition, not per row. */
+  private val runOnceMethods = Set("<init>", "<clinit>", "init")
+
+  private val hugeMethodWarned = new java.util.concurrent.atomic.AtomicBoolean(false)
+
+  /** Lets a test see the first warning again. */
+  private[catalyst] def resetHugeMethodWarning(): Unit = hugeMethodWarned.set(false)
+
+  /**
+   * Reports a generated method whose bytecode is past HotSpot's JIT limit
+   * ([[CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT]]). The first time in the driver that such a
+   * method will run interpreted on every row of a whole-stage codegen stage, the report is a
+   * warning that names the remedy: the JVM skips huge methods, the method is not one that runs
+   * once, and `spark.sql.codegen.hugeMethodLimit` does not already make the stage fall back.
+   * Every other report is INFO: every one after that first warning, every one from a generated
+   * class outside whole-stage codegen, where the setting does not apply, and every one on an
+   * executor, since the driver compiles each stage before its tasks do and has reported it.
+   */
+  private[catalyst] def logHugeMethod(className: String, methodName: String, size: Int): Unit = {
+    val limit = CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT
+    val wholeStage = className.contains("GeneratedIteratorForCodegenStage")
+    // `WholeStageCodegenExec` falls back on the largest method of the stage, so with a limit
+    // between 8000 and 65535 another method can make the stage fall back while this one is under
+    // the limit. At the default and at the recommended 8000 the two checks agree.
+    val interpretedPerRow = jitSkipsHugeMethods && wholeStage && TaskContext.get() == null &&
+      !runOnceMethods.contains(methodName) && size <= SQLConf.get.hugeMethodLimit
+    if (interpretedPerRow && hugeMethodWarned.compareAndSet(false, true)) {
+      logWarning(log"Generated method too long to be JIT compiled: " +
+        log"${MDC(LogKeys.CLASS_NAME, className)}.${MDC(LogKeys.METHOD_NAME, methodName)} is " +
+        log"${MDC(LogKeys.BYTECODE_SIZE, size)} bytes, so it runs interpreted on every row. " +
+        log"Setting " +
+        log"${MDC(LogKeys.CONFIG, SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key)} to " +
+        log"${MDC(LogKeys.HUGE_METHOD_LIMIT, limit)} runs such stages without whole-stage " +
+        log"codegen instead. Further methods over the limit are logged at INFO.")
+    } else {
+      logInfo(log"Generated method too long to be JIT compiled: " +
+        log"${MDC(LogKeys.CLASS_NAME, className)}.${MDC(LogKeys.METHOD_NAME, methodName)} is " +
+        log"${MDC(LogKeys.BYTECODE_SIZE, size)} bytes")
+    }
+  }
+
   /**
    * Compute bytecode statistics for a set of compiled classes. Both backends
    * produce the same map shape (className -> classfile bytes), so the analysis is
@@ -338,10 +387,7 @@ object CodeCompiler extends Logging {
             val byteCodeSize = attr.code.length
             CodegenMetrics.METRIC_GENERATED_METHOD_BYTECODE_SIZE.update(byteCodeSize)
             if (byteCodeSize > CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT) {
-              logInfo(log"Generated method too long to be JIT compiled: " +
-                log"${MDC(LogKeys.CLASS_NAME, cf.getThisClassName)}." +
-                log"${MDC(LogKeys.METHOD_NAME, method.getName)} is " +
-                log"${MDC(LogKeys.BYTECODE_SIZE, byteCodeSize)} bytes")
+              logHugeMethod(cf.getThisClassName, method.getName, byteCodeSize)
             }
             byteCodeSize
           }
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala
index d713ce4e8217..4dc761a04b1f 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala
@@ -21,7 +21,7 @@ import java.sql.Timestamp
 
 import org.apache.logging.log4j.Level
 
-import org.apache.spark.SparkFunSuite
+import org.apache.spark.{SparkFunSuite, TaskContext}
 import org.apache.spark.metrics.source.CodegenMetrics
 import org.apache.spark.sql.Row
 import org.apache.spark.sql.catalyst.InternalRow
@@ -34,7 +34,7 @@ import org.apache.spark.sql.catalyst.util.DateTimeTestUtils.LA
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 import org.apache.spark.unsafe.types.UTF8String
-import org.apache.spark.util.ThreadUtils
+import org.apache.spark.util.{ThreadUtils, Utils}
 
 /**
  * Additional tests for code generation.
@@ -573,8 +573,53 @@ class CodeGenerationSuite extends SparkFunSuite with ExpressionEvalHelper {
       val actual = proj(null)
       assert(actual.getInt(0) == x)
     }
-    assert(appender.loggingEvents
-      .exists(_.getMessage().getFormattedMessage.contains("Generated method too long")))
+    // A projection is not a whole-stage codegen stage, where the remedy applies, so its huge
+    // method is reported at INFO, as before.
+    val lines = appender.loggingEvents
+      .filter(_.getMessage().getFormattedMessage.contains("Generated method too long"))
+    assert(lines.nonEmpty && lines.forall(_.getLevel == Level.INFO))
+  }
+
+  test("SPARK-59774: a huge whole-stage method warns once, with the remedy, where it applies") {
+    assume(Utils.getVMOptionValue("DontCompileHugeMethods").contains("true"),
+      "the JVM compiles huge methods, so nothing is interpreted")
+    val stage = "org.apache.spark.sql.catalyst.expressions.GeneratedClass$" +
+      "GeneratedIteratorForCodegenStage1"
+    def logged(className: String, method: String, size: Int): Seq[(Level, String)] = {
+      val appender = new LogAppender("huge method")
+      withLogAppender(appender, loggerNames = Seq(classOf[CodeGenerator[_, _]].getName),
+          Some(Level.INFO)) {
+        CodeCompiler.logHugeMethod(className, method, size)
+      }
+      appender.loggingEvents.toSeq.map(e => (e.getLevel, e.getMessage.getFormattedMessage))
+    }
+    CodeCompiler.resetHugeMethodWarning()
+    // With the setting at the JIT limit the stage falls back, so the method never runs: INFO.
+    withSQLConf(SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key ->
+        CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT.toString) {
+      assert(logged(stage, "processNext", 9513).map(_._1) == Seq(Level.INFO))
+    }
+    // A method that runs once per class or partition costs little interpreted: INFO.
+    assert(logged(stage, "init", 9513).map(_._1) == Seq(Level.INFO))
+    assert(logged(stage, "<init>", 9513).map(_._1) == Seq(Level.INFO))
+    // Outside whole-stage codegen the setting does not apply: INFO.
+    assert(logged("org.apache.spark.sql.catalyst.expressions.GeneratedClass$" +
+      "SpecificUnsafeProjection", "apply", 9513).map(_._1) == Seq(Level.INFO))
+    // An executor compiles the stage after the driver has reported it: INFO.
+    TaskContext.setTaskContext(TaskContext.empty())
+    try {
+      assert(logged(stage, "processNext", 9513).map(_._1) == Seq(Level.INFO))
+    } finally {
+      TaskContext.unset()
+    }
+    // The per-row method of a stage that keeps whole-stage codegen, on the driver: a warning
+    // naming the setting and the value, the first time...
+    val warned = logged(stage, "processNext", 9513)
+    assert(warned.size == 1 && warned.head._1 == Level.WARN)
+    assert(warned.head._2.contains(s"${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key} to " +
+      s"${CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT}"))
+    // ...and INFO after that, so executors and recompiles do not repeat it.
+    assert(logged(stage, "processNext", 9600).map(_._1) == Seq(Level.INFO))
   }
 
   test("SPARK-51527: spark.sql.codegen.logLevel") {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org