[spark] branch branch-3.1 updated: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
     new c625eb4  [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
c625eb4 is described below

commit c625eb4f9f970108d93bf3342c7ccb7ec873dc27
Author: skotlov <sk...@joom.com>
AuthorDate: Thu Apr 8 16:51:38 2021 -0700

    [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
    
    ### What changes were proposed in this pull request?
    Close SparkContext after the Main method has finished, to allow SparkApplication on K8S to complete
    
    ### Why are the changes needed?
    if I don't call the method sparkContext.stop() explicitly, then a Spark driver process doesn't terminate even after its Main method has been completed. This behaviour is different from spark on yarn, where the manual sparkContext stopping is not required. It looks like, the problem is in using non-daemon threads, which prevent the driver jvm process from terminating.
    So I have inserted code that closes sparkContext automatically.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Manually on the production AWS EKS environment in my company.
    
    Closes #32081 from kotlovs/close-spark-context-on-exit.
    
    Authored-by: skotlov <sk...@joom.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
    (cherry picked from commit ab97db75b2ab3ebb4d527610e2801df89cc23e2d)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index bb3a20d..17950d6 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -952,6 +952,12 @@ private[spark] class SparkSubmit extends Logging {
     } catch {
       case t: Throwable =>
         throw findCause(t)
+    } finally {
+      try {
+        SparkContext.getActive.foreach(_.stop())
+      } catch {
+        case e: Throwable => logError(s"Failed to close SparkContext: $e")
+      }
     }
   }
 

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org