You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/03/14 15:49:35 UTC
[spark] branch master updated: [SPARK-42785][K8S][CORE] When spark submit without `--deploy-mode`, avoid facing NPE in Kubernetes Case
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 767253bb621 [SPARK-42785][K8S][CORE] When spark submit without `--deploy-mode`, avoid facing NPE in Kubernetes Case
767253bb621 is described below
commit 767253bb6219f775a8a21f1cdd0eb8c25fa0b9de
Author: zwangsheng <22...@qq.com>
AuthorDate: Tue Mar 14 08:49:13 2023 -0700
[SPARK-42785][K8S][CORE] When spark submit without `--deploy-mode`, avoid facing NPE in Kubernetes Case
### What changes were proposed in this pull request?
After https://github.com/apache/spark/pull/37880 when user spark submit without `--deploy-mode XXX` or `–conf spark.submit.deployMode=XXXX`, may face NPE with this code.
### Why are the changes needed?
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#164
```scala
args.deployMode.equals("client") &&
```
Of course, submit without `deployMode` is not allowed and will throw an exception and terminate the application, but we should leave it to the later logic to give the appropriate hint instead of giving a NPE.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Closes #40414 from zwangsheng/SPARK-42785.
Authored-by: zwangsheng <22...@qq.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index 7563b093522..506dd0d916a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -162,7 +162,7 @@ private[spark] class SparkSubmit extends Logging {
// deploy mode with k8s resource manager, the spark submit in the driver container
// is done in client mode.
val isKubernetesClusterModeDriver = args.master.startsWith("k8s") &&
- args.deployMode.equals("client") &&
+ "client".equals(args.deployMode) &&
args.toSparkConf().getBoolean("spark.kubernetes.submitInDriver", false)
if (isKubernetesClusterModeDriver) {
logInfo("Running driver with proxy user. Cluster manager: Kubernetes")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org