You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/07/19 05:26:50 UTC
[spark] branch master updated: [SPARK-36193][CORE] Recover
SparkSubmit.runMain not to stop SparkContext in non-K8s env
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fd3e9ce [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env
fd3e9ce is described below
commit fd3e9ce0b9ee09c7dce9f2e029fe96eac51eab96
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Jul 18 22:26:23 2021 -0700
[SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env
### What changes were proposed in this pull request?
According to the discussion on https://github.com/apache/spark/pull/32283 , this PR aims to limit the feature of SPARK-34674 to K8s environment only.
### Why are the changes needed?
To reduce the behavior change in non-K8s environment.
### Does this PR introduce _any_ user-facing change?
The change behavior is consistent with 3.1.1 and older Spark releases.
### How was this patch tested?
N/A
Closes #33403 from dongjoon-hyun/SPARK-36193.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index a65be54..8124650 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -957,8 +957,8 @@ private[spark] class SparkSubmit extends Logging {
case t: Throwable =>
throw findCause(t)
} finally {
- if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
- !isThriftServer(args.mainClass)) {
+ if (args.master.startsWith("k8s") && !isShell(args.primaryResource) &&
+ !isSqlShell(args.mainClass) && !isThriftServer(args.mainClass)) {
try {
SparkContext.getActive.foreach(_.stop())
} catch {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org