You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/04/03 02:42:37 UTC
[spark] branch master updated: [SPARK-38770][K8S] Remove `renameMainAppResource` from `baseDriverContainer`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 22b6ea0 [SPARK-38770][K8S] Remove `renameMainAppResource` from `baseDriverContainer`
22b6ea0 is described below
commit 22b6ea0253f0fc6bcc82c1872153de0ca5b30f9c
Author: Qian.Sun <qi...@gmail.com>
AuthorDate: Sat Apr 2 19:41:17 2022 -0700
[SPARK-38770][K8S] Remove `renameMainAppResource` from `baseDriverContainer`
### What changes were proposed in this pull request?
This PR aims to simply steps to re-write primary resource in k8s spark application.
### Why are the changes needed?
Re-write primary resource uses `renameMainAppResource` twice.
* First `renameMainAppResource` in `baseDriverContainer` in is introduced by #23546
* #25870 refactors `renameMainAppResource` and introduces `renameMainAppResource` in `configureForJava`.
Refactoring and `renameMainAppResource` in `configureForJava` makes `renameMainAppResource` in `baseDriverContainer` useless.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
* Pass the GA.
* Pass k8s IT.
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=r -Dspark.kubernetes.test.imageRepo=kubespark "kubernetes-integration-tests/test"
[info] KubernetesSuite:
[info] - Run SparkPi with no resources (17 seconds, 443 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (17 seconds, 858 milliseconds)
[info] - Run SparkPi with a very long application name. (30 seconds, 450 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (18 seconds, 596 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (18 seconds, 534 milliseconds)
[info] - Run SparkPi with an argument. (21 seconds, 853 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (14 seconds, 285 milliseconds)
[info] - All pods have the same service account by default (13 seconds, 800 milliseconds)
[info] - Run extraJVMOptions check on driver (7 seconds, 825 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (15 seconds, 242 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (15 seconds, 491 milliseconds)
[info] - Run SparkPi with env and mount secrets. (26 seconds, 967 milliseconds)
[info] - Run PySpark on simple pi.py example (20 seconds, 318 milliseconds)
[info] - Run PySpark to test a pyfiles example (25 seconds, 659 milliseconds)
[info] - Run PySpark with memory customization (25 seconds, 608 milliseconds)
[info] - Run in client mode. (14 seconds, 620 milliseconds)
[info] - Start pod creation from template (19 seconds, 916 milliseconds)
[info] - SPARK-38398: Schedule pod creation from template (19 seconds, 966 milliseconds)
[info] - PVs with local hostpath storage on statefulsets (22 seconds, 380 milliseconds)
[info] - PVs with local hostpath and storageClass on statefulsets (26 seconds, 935 milliseconds)
[info] - PVs with local storage (30 seconds, 75 milliseconds)
[info] - Launcher client dependencies (2 minutes, 48 seconds)
[info] - SPARK-33615: Launcher client archives (1 minute, 26 seconds)
[info] - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON (1 minute, 47 seconds)
[info] - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python (1 minute, 51 seconds)
[info] - Launcher python client dependencies using a zip file (1 minute, 51 seconds)
[info] - Test basic decommissioning (59 seconds, 765 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (1 minute, 3 seconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 58 seconds)
[info] - Test decommissioning timeouts (58 seconds, 754 milliseconds)
[info] - SPARK-37576: Rolling decommissioning (1 minute, 15 seconds)
[info] Run completed in 29 minutes, 15 seconds.
[info] Total number of tests run: 31
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2020 s (33:40), completed 2022-4-2 12:35:52
```
PS. #23546 introduces deleted code and `DepsTestsSuite`. `DepsTestsSuite` can check re-write primary resource. This PR can pass `DepsTestsSuite`, which can prove deletion about `renameMainAppResource` in `baseDriverContainer` does not affect the process about re-write primary resource.
Closes #36044 from dcoliversun/SPARK-38770.
Authored-by: Qian.Sun <qi...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../spark/deploy/k8s/features/DriverCommandFeatureStep.scala | 8 +-------
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala
index 8015a1a..e9fbcd7 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala
@@ -120,12 +120,6 @@ private[spark] class DriverCommandFeatureStep(conf: KubernetesDriverConf)
}
private def baseDriverContainer(pod: SparkPod, resource: String): ContainerBuilder = {
- // re-write primary resource, app jar is also added to spark.jars by default in SparkSubmit
- val resolvedResource = if (conf.mainAppResource.isInstanceOf[JavaMainAppResource]) {
- KubernetesUtils.renameMainAppResource(resource, Option(conf.sparkConf), false)
- } else {
- resource
- }
var proxyUserArgs = Seq[String]()
if (!conf.proxyUser.isEmpty) {
proxyUserArgs = proxyUserArgs :+ "--proxy-user"
@@ -136,7 +130,7 @@ private[spark] class DriverCommandFeatureStep(conf: KubernetesDriverConf)
.addToArgs(proxyUserArgs: _*)
.addToArgs("--properties-file", SPARK_CONF_PATH)
.addToArgs("--class", conf.mainClass)
- .addToArgs(resolvedResource)
+ .addToArgs(resource)
.addToArgs(conf.appArgs: _*)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org