[jira] [Resolved] (SPARK-57191) [YARN] Driver hangs indefinitely when job submission / monitor thread fails

["Kousuke Saruta (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kousuke Saruta resolved SPARK-57191.
------------------------------------
    Fix Version/s: 4.2.0
         Assignee: Shrirang Mhalgi
       Resolution: Fixed

Issue resolved by https://github.com/apache/spark/pull/56274

> [YARN] Driver hangs indefinitely when job submission / monitor thread fails
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-57191
>                 URL: https://issues.apache.org/jira/browse/SPARK-57191
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 4.1.2
>            Reporter: Rohan Arora
>            Assignee: Shrirang Mhalgi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0
>
>
> h4. *Overview*
> In Spark-on-YARN client mode deployment, if a fatal uncaught exception is thrown within the asynchronous application-submission or application-monitoring thread (e.g., during initialisation inside {{YarnClientSchedulerBackend}} or YARN {{{}Client.scala{}}}), the Spark Driver process hangs indefinitely instead of shutting down or throwing the exception to the main thread.
> h4. *Root Cause Analysis*
>  # {*}Asynchronous Execution{*}: When YARN client mode starts, {{YarnClientSchedulerBackend}} submits the Spark application context to YARN and monitors it asynchronously (e.g., utilising the internal {{MonitorThread}} or Scala standard {{Future}} contexts).
>  # {*}Exception Swallowing/Isolation{*}: If a fatal exception occurs in these background threads (such as network failure, credential expiration, or {{OutOfMemoryError}} during the initial handshake), the exception is either swallowed by Scala {{Future}} execution pools or isolate-trapped in a thread not guarded by Spark’s custom {{{}SparkUncaughtExceptionHandler{}}}.
>  # {*}Blocker Threads Inactive{*}: Main threads (like the one executing {{SparkContext.init}} or {{{}waitForApplication{}}}) remain indefinitely blocked waiting on the future completion or lock notification.
>  # {*}Zombie JVM State{*}: Since the driver process has already spun up active non-daemon threads (such as heartbeats, Spark UI HTTP server, and log appenders), the JVM does not exit naturally, leaving the driver in a zombie/hung state.
> h4. *Impact on Managed Environments*
> In orchestration and managed environments (such as cloud platform agents, workflows, schedulers), the agent continues to report the job driver process as active. The scheduler cannot distinguish this hung driver from a driver performing legitimate post-execution cleanup (like metastore synchronization or final file renaming). This leads to resource leakages, orphaned driver processes, and long job timeout durations for customers.
> h4. *Proposed Solution*
>  * {*}Exception Propagation{*}: Ensure that worker thread closures and background futures executing YARN submissions are wrapped in robust {{try-catch}} blocks that propagate exceptions to Spark's uncaught exception handler ({{{}ThreadUtils.runInNewThread{}}} should be leveraged for thread instantiation).
>  * {*}Explicit Teardown on Failure{*}: On critical failures inside the submission or monitoring loops, explicitly trigger {{SparkContext.stop()}} or standard JVM termination ({{{}System.exit(exitCode){}}}) so that the main thread does not block infinitely on states that will never resolve.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org