[jira] [Created] (SPARK-57191) [YARN] Driver hangs indefinitely when job submission / monitor thread fails

["Rohan Arora (Jira)" <ji...@apache.org>]

Rohan Arora created SPARK-57191:
-----------------------------------

             Summary: [YARN] Driver hangs indefinitely when job submission / monitor thread fails
                 Key: SPARK-57191
                 URL: https://issues.apache.org/jira/browse/SPARK-57191
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 4.1.2
            Reporter: Rohan Arora


h4. *Overview*

In Spark-on-YARN client mode deployment, if a fatal uncaught exception is thrown within the asynchronous application-submission or application-monitoring thread (e.g., during initialisation inside {{YarnClientSchedulerBackend}} or YARN {{{}Client.scala{}}}), the Spark Driver process hangs indefinitely instead of shutting down or throwing the exception to the main thread.
h4. *Root Cause Analysis*
 # {*}Asynchronous Execution{*}: When YARN client mode starts, {{YarnClientSchedulerBackend}} submits the Spark application context to YARN and monitors it asynchronously (e.g., utilising the internal {{MonitorThread}} or Scala standard {{Future}} contexts).
 # {*}Exception Swallowing/Isolation{*}: If a fatal exception occurs in these background threads (such as network failure, credential expiration, or {{OutOfMemoryError}} during the initial handshake), the exception is either swallowed by Scala {{Future}} execution pools or isolate-trapped in a thread not guarded by Spark’s custom {{{}SparkUncaughtExceptionHandler{}}}.
 # {*}Blocker Threads Inactive{*}: Main threads (like the one executing {{SparkContext.init}} or {{{}waitForApplication{}}}) remain indefinitely blocked waiting on the future completion or lock notification.
 # {*}Zombie JVM State{*}: Since the driver process has already spun up active non-daemon threads (such as heartbeats, Spark UI HTTP server, and log appenders), the JVM does not exit naturally, leaving the driver in a zombie/hung state.

h4. *Impact on Managed Environments*

In orchestration and managed environments (such as cloud platform agents, workflows, schedulers), the agent continues to report the job driver process as active. The scheduler cannot distinguish this hung driver from a driver performing legitimate post-execution cleanup (like metastore synchronization or final file renaming). This leads to resource leakages, orphaned driver processes, and long job timeout durations for customers.
h4. *Proposed Solution*
 * {*}Exception Propagation{*}: Ensure that worker thread closures and background futures executing YARN submissions are wrapped in robust {{try-catch}} blocks that propagate exceptions to Spark's uncaught exception handler ({{{}ThreadUtils.runInNewThread{}}} should be leveraged for thread instantiation).
 * {*}Explicit Teardown on Failure{*}: On critical failures inside the submission or monitoring loops, explicitly trigger {{SparkContext.stop()}} or standard JVM termination ({{{}System.exit(exitCode){}}}) so that the main thread does not block infinitely on states that will never resolve.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org