(kyuubi) branch branch-1.11 updated: [KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown

[ch...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

pan3793 pushed a commit to branch branch-1.11
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/branch-1.11 by this push:
     new 71c362449c [KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown
71c362449c is described below

commit 71c362449c7c3490ed4739f802f0cbbbb6d54449
Author: Wu Shuwen <mi...@outlook.com>
AuthorDate: Wed Oct 28 11:16:54 2026 +0800

    [KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown
    
    Fixes #7590.
    
    ### Why are the changes needed?
    
    When SparkContext stops before engine shutdown reaches `stopServer()`, the latch in `SparkSQLEngine.main` may never be released. The main thread polls the latch and checks SparkContext; if it is stopped first, the code now throws `KyuubiException` through the existing engine error path. That path records the abnormal termination and diagnostics instead of returning normally (which could make YARN report the application as succeeded). Earlier PRs #7592 and #7680 are closed and unmerged.
    
    ### How was this patch tested?
    
    - `git diff --check` passed for this review follow-up.
    - Maven/Spotless and the Spark test suite were not rerun for this revision because the protected shared Zinc-cache incident prohibits Maven reruns. The earlier 3/3 `SparkSQLEngineSuite` run was on commit `e24adad`, before this follow-up, and does not validate the current diff.
    
    ### Was this patch assisted by generative AI tooling?
    
    Assisted-by: Pi worker dev-w11stew-c1-4w (openai-codex/gpt-6-luna).
    
    Closes #7797 from dajiaohuang/kyuubi-7590-spark-context-stop.
    
    Closes #7590
    
    17c57661f [dajiaohuang] Report abnormal SparkContext termination
    9ace3f81e [dajiaohuang] Inline SparkContext termination wait
    e24adad24 [dajiaohuang] Simplify SparkContext termination wait
    bfb8d04f2 [dajiaohuang] [KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown
    
    Authored-by: Wu Shuwen <mi...@outlook.com>
    Signed-off-by: Cheng Pan <ch...@apache.org>
    (cherry picked from commit d953184bc743ab7c1752895e7f10013f4b2ff19f)
    Signed-off-by: Cheng Pan <ch...@apache.org>
---
 .../scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala    | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index c793d846ee..856098e06b 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -410,7 +410,14 @@ object SparkSQLEngine extends Logging {
         try {
           startEngine(spark)
           // blocking main thread
-          countDownLatch.await()
+          // The engine may not reach stopServer after SparkContext shutdown starts.
+          // Fail through the engine error path so abnormal termination is reported.
+          while (!countDownLatch.await(1, TimeUnit.SECONDS)) {
+            if (spark.sparkContext.isStopped) {
+              throw new KyuubiException(
+                "SparkContext stopped before SparkSQLEngine, terminating the engine main thread")
+            }
+          }
         } catch {
           case e: KyuubiException =>
             currentEngine match {