[spark] branch branch-3.1 updated: [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
     new ab94702  [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env
ab94702 is described below

commit ab94702a4f3a81942fc26c13d84574506a70eff2
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Jul 18 22:26:23 2021 -0700

    [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env
    
    ### What changes were proposed in this pull request?
    
    According to the discussion on https://github.com/apache/spark/pull/32283 , this PR aims to limit the feature of SPARK-34674 to K8s environment only.
    
    ### Why are the changes needed?
    
    To reduce the behavior change in non-K8s environment.
    
    ### Does this PR introduce _any_ user-facing change?
    
    The change behavior is consistent with 3.1.1 and older Spark releases.
    
    ### How was this patch tested?
    
    N/A
    
    Closes #33403 from dongjoon-hyun/SPARK-36193.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit fd3e9ce0b9ee09c7dce9f2e029fe96eac51eab96)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index fa86da9..818b263 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -953,8 +953,8 @@ private[spark] class SparkSubmit extends Logging {
       case t: Throwable =>
         throw findCause(t)
     } finally {
-      if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
-        !isThriftServer(args.mainClass)) {
+      if (args.master.startsWith("k8s") && !isShell(args.primaryResource) &&
+          !isSqlShell(args.mainClass) && !isThriftServer(args.mainClass)) {
         try {
           SparkContext.getActive.foreach(_.stop())
         } catch {

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org