You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/01/25 17:36:53 UTC
[spark] branch branch-3.2 updated: [SPARK-38022][K8S][TESTS] Use relativePath for K8s remote file test in `BasicTestsSuite`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 727dbe3 [SPARK-38022][K8S][TESTS] Use relativePath for K8s remote file test in `BasicTestsSuite`
727dbe3 is described below
commit 727dbe35f468e6e5cf0ba6abbd83676a19046a60
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Jan 25 09:29:57 2022 -0800
[SPARK-38022][K8S][TESTS] Use relativePath for K8s remote file test in `BasicTestsSuite`
### What changes were proposed in this pull request?
This PR aims to use `relativePath` for K8s remote file test in `BasicTestsSuite`.
### Why are the changes needed?
To make `Run SparkRemoteFileTest using a remote data file` test pass.
**BEFORE**
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 3 seconds)
[info] The code passed to eventually never returned normally. Attempted 190 times over 3.012265011116667 minutes. Last failure message: false was not true. (KubernetesSuite.scala:452)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608 milliseconds)
...
```
Closes #35318 from dongjoon-hyun/SPARK-38022.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 277322851f3c96f812c7da115f00f66bb6f11f6b)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
index 1c12123..8c753ea 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
+++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
@@ -102,8 +102,8 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
test("Run SparkRemoteFileTest using a remote data file", k8sTestTag) {
assert(sys.props.contains("spark.test.home"), "spark.test.home is not set!")
TestUtils.withHttpServer(sys.props("spark.test.home")) { baseURL =>
- sparkAppConf
- .set("spark.files", baseURL.toString + REMOTE_PAGE_RANK_DATA_FILE)
+ sparkAppConf.set("spark.files", baseURL.toString +
+ REMOTE_PAGE_RANK_DATA_FILE.replace(sys.props("spark.test.home"), "").substring(1))
runSparkRemoteCheckAndVerifyCompletion(appArgs = Array(REMOTE_PAGE_RANK_FILE_NAME))
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org