(spark) branch branch-4.x updated: [SPARK-59813][YARN] Dump thread stacks when SparkContext initialization times out in ApplicationMaster

[ch...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new e336c2e106d8 [SPARK-59813][YARN] Dump thread stacks when SparkContext initialization times out in ApplicationMaster
e336c2e106d8 is described below

commit e336c2e106d8268e94d16fa6e5823bd942eac592
Author: singer-bin <26...@users.noreply.github.com>
AuthorDate: Wed Oct 14 15:55:13 2026 +0800

    [SPARK-59813][YARN] Dump thread stacks when SparkContext initialization times out in ApplicationMaster
    
    ## What changes were proposed in this pull request?
    
    When SparkContext initialization times out in YARN cluster mode (`spark.yarn.am.waitTime`), the ApplicationMaster fails the application with exit code 13 and only advises to "check earlier log output for errors". However, in real production incidents the driver thread is often blocked *silently* (slow/contended node, a blocking call with no logging, etc.) and there is **no** earlier log output at all, leaving operators with no way to tell where the driver thread was stuck.
    
    This PR logs a full thread dump when the timeout fires, with the user application thread (driver) sorted to the front since it is usually the most relevant one. The behavior is gated by a new config `spark.yarn.am.sparkContextInitTimeout.threadDump` (default `true`), following the same pattern as `spark.task.reaper.threadDump` on the executor side. The dump is best-effort: any failure while capturing it is swallowed so it never masks the original timeout error.
    
    ## Why are the changes needed?
    
    Production example: a long-running SQL engine (Kyuubi) failed with `java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]` and exit code 13. The driver stdout showed ~80 seconds of complete silence right after `SparkEnv` initialization finished. Cross-checking the HDFS NameNode and Hive Metastore logs proved the driver issued zero external I/O during the hang, so the usual suspects (slow NN/HMS calls) were all ruled out — but without a thread dump there  [...]
    
    ## Does this PR introduce any user-facing change?
    
    Yes: a new configuration `spark.yarn.am.sparkContextInitTimeout.threadDump` (default `true`). When enabled, an error-level thread dump is logged on the SparkContext-initialization-timeout failure path.
    
    ## How was this patch tested?
    
    - Built the module and its dependency chain: `./build/mvn -Pyarn -pl resource-managers/yarn -am package -DskipTests`
    - `scalastyle:check` on `resource-managers/yarn` and `checkstyle:check` on `common/utils-java` both pass
    - `ApplicationMasterSuite` passes
    
    The `runDriver` timeout path has no existing test harness (it requires a live YARN RM), and this is a logging-only change on a failure path, so no new test is added.
    
    Closes #58991 from singer-bin/dev/am-timeout-threaddump.
    
    Lead-authored-by: singer-bin <26...@users.noreply.github.com>
    Co-authored-by: singerzhang <si...@tencent.com>
    Signed-off-by: Cheng Pan <ch...@apache.org>
    (cherry picked from commit b4316defc89eff702118411119acbe65454d18d9)
    Signed-off-by: Cheng Pan <ch...@apache.org>
---
 .../spark/deploy/yarn/ApplicationMaster.scala      | 25 ++++++++++++++++++++++
 .../apache/spark/deploy/yarn/config/package.scala  |  8 +++++++
 2 files changed, 33 insertions(+)

diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
index 9c9f6f3bc313..0cfe0a14ed11 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
@@ -528,6 +528,9 @@ private[spark] class ApplicationMaster(
         logError(log"SparkContext did not initialize after waiting for " +
             log"${MDC(LogKeys.TIMEOUT, totalWaitTime)} ms. " +
             log"Please check earlier log output for errors. Failing the application.")
+        if (sparkConf.get(AM_SC_INIT_TIMEOUT_THREAD_DUMP)) {
+          dumpThreadStacksOnScInitTimeout()
+        }
         finish(FinalApplicationStatus.FAILED,
           ApplicationMaster.EXIT_SC_NOT_INITED,
           "Timed out waiting for SparkContext.")
@@ -536,6 +539,28 @@ private[spark] class ApplicationMaster(
     }
   }
 
+  /**
+   * Log the stack traces of all threads to help diagnose what the user application thread
+   * (driver) was blocked on when SparkContext initialization timed out. The user thread is
+   * sorted to the front since it is usually the most relevant one. Failures of the dump
+   * itself are swallowed so that they never mask the original timeout error.
+   */
+  private def dumpThreadStacksOnScInitTimeout(): Unit = {
+    try {
+      val dumps = Utils.getThreadDump()
+      val sorted = Option(userClassThread) match {
+        case Some(t) => dumps.sortBy(d => if (d.threadId == t.getId) 0 else 1)
+        case None => dumps
+      }
+      logError(log"Thread dump on SparkContext initialization timeout:\n" +
+        log"${MDC(LogKeys.THREAD, sorted.map(_.toString).mkString("\n"))}")
+    } catch {
+      case NonFatal(dumpError) =>
+        logWarning("Failed to capture thread dump on SparkContext initialization timeout.",
+          dumpError)
+    }
+  }
+
   private def runExecutorLauncher(): Unit = {
     val hostname = Utils.localHostNameForURI()
     val amCores = sparkConf.get(AM_CORES)
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config/package.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config/package.scala
index 7147cbfcf276..c6a0a3f18665 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config/package.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config/package.scala
@@ -251,6 +251,14 @@ package object config extends Logging {
     .timeConf(TimeUnit.MILLISECONDS)
     .createWithDefaultString("100s")
 
+  private[spark] val AM_SC_INIT_TIMEOUT_THREAD_DUMP =
+    ConfigBuilder("spark.yarn.am.sparkContextInitTimeout.threadDump")
+      .doc(s"Whether to print all thread stacks when SparkContext initialization times out " +
+        s"(see ${AM_MAX_WAIT_TIME.key}).")
+      .version("4.4.0")
+      .booleanConf
+      .createWithDefault(true)
+
   private[spark] val YARN_METRICS_NAMESPACE = ConfigBuilder("spark.yarn.metrics.namespace")
     .doc("The root namespace for AM metrics reporting.")
     .version("2.4.0")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org