You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/08 14:50:30 UTC
[GitHub] [spark] Yikun opened a new pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Yikun opened a new pull request #35772:
URL: https://github.com/apache/spark/pull/35772
### What changes were proposed in this pull request?
Support pyDockerfile and rDockerfile in SBT K8s IT
### Why are the changes needed?
Enable users to specify `pyDockerfile` and `rDockerfile` separately.
### Does this PR introduce _any_ user-facing change?
No, test only
### How was this patch tested?
```
build/sbt -Pkubernetes -Pkubernetes-integration-tests \
-Dtest.exclude.tags=minikube -Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.pyDockerFile=/Users/yikun/code/Dockerfile.py \
-Dspark.kubernetes.test.rDockerFile=/Users/yikun/code/Dockerfile.r \
-Dspark.kubernetes.test.DockerFile=/Users/yikun/code/Dockerfile "kubernetes-integration-tests/test"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822239491
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -294,3 +294,13 @@ In addition, you can run a single test selectively.
-Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very long application name"'
+
+You can also specify your specific dockerfile to build JVM/Python/R based image to test.
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
Review comment:
This looks misleading because this suggests it's a Python script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822276591
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -294,3 +294,13 @@ In addition, you can run a single test selectively.
-Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very long application name"'
+
+You can also specify your specific dockerfile to build JVM/Python/R based image to test.
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
+ -Dspark.kubernetes.test.rDockerFile=/path/to/Dockerfile.r \
Review comment:
/path/to/r/Dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822238657
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -288,6 +288,16 @@ The following is an example to rerun tests with the pre-built image.
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/runIts'
+You can also specify your specific dockerfile to build JVM/Python/R based image to test:
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
+ -Dspark.kubernetes.test.rDockerFile=/path/to/Dockerfile.r \
+ 'kubernetes-integration-tests/test'
Review comment:
Although this is a minor comment, it would be great to be consistent with the above examples.
- First, this is not setting `deployMode=docker-desktop`. If then, why do we have `-Dtest.exclude.tags=minikube`?
- Second, in the same way, we had better use `-Dspark.kubernetes.test.imageTag=2022-03-06` consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822239813
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -294,3 +294,13 @@ In addition, you can run a single test selectively.
-Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very long application name"'
+
+You can also specify your specific dockerfile to build JVM/Python/R based image to test.
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
+ -Dspark.kubernetes.test.rDockerFile=/path/to/Dockerfile.r \
Review comment:
ditto. `Dockerfile.r` looks like R script. It should not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822239116
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -294,3 +294,13 @@ In addition, you can run a single test selectively.
-Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very long application name"'
+
+You can also specify your specific dockerfile to build JVM/Python/R based image to test.
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
Review comment:
Is this tested, @Yikun ? It looks wrong because the correct one is `spark.kubernetes.test.dockerFile`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35772:
URL: https://github.com/apache/spark/pull/35772#issuecomment-1062039725
Thank you, @Yikun . cc @williamhyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35772:
URL: https://github.com/apache/spark/pull/35772#issuecomment-1061878593
For my case,
- case 1: I want to set pyDockerfile separately, do some custom dev.
- case 2: I always to run IT slowly due to build py/r binding images do `apt update` and `apt install`again and again (because base images change), so I just write a `all-in-one` Dockerfile to add all deps (py+r+base) at first line, then image docker build will re-use previous cache. it can reduce about 1+ minute in my env (this all in one dockerfile could be a new contribution if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35772:
URL: https://github.com/apache/spark/pull/35772#issuecomment-1061878593
For my case,
- case 1: I want to set pyDockerfile separately, do some custom dev.
- case 2: I always to run IT slowly due to build py/r binding images do `apt update` and `apt install`again and again (because base images change), so I just write a `all-in-one` Dockerfile to add all deps (py+r+base) at first line, then image docker build will re-use previous cache. it can reduce about 1+ minute in my env (this all in one dockerfile could be a contribution if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35772:
URL: https://github.com/apache/spark/pull/35772#issuecomment-1061878593
For my case,
- case 1: I want to set pyDockerfile separately, do some custom dev.
- case 2: I always to run IT slowly due to build py/r binding images do `apt update` and `apt install`again and again (because base images change), so I just write a `all-in-one` Dockerfile to add all deps (py+r+base) at first line, then image build will re-use cache. it can reduce about 1+ minute in my env (this all in one dockerfile could be a contribution if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35772:
URL: https://github.com/apache/spark/pull/35772#issuecomment-1061878593
For my case,
- case 1: I want to set pyDockerfile separately, do some custom dev.
- case 2: I always to run IT slowly due to build py/r binding images do `apt update` and `apt install`again and again (because base images change), so I just write a `all-in-one` Dockerfile to add all deps (py+r+base) at first line, then image build will re-use cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35772:
URL: https://github.com/apache/spark/pull/35772#issuecomment-1061859145
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822275246
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -288,6 +288,16 @@ The following is an example to rerun tests with the pre-built image.
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/runIts'
+You can also specify your specific dockerfile to build JVM/Python/R based image to test:
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
+ -Dspark.kubernetes.test.rDockerFile=/path/to/Dockerfile.r \
+ 'kubernetes-integration-tests/test'
Review comment:
> First, this is not setting deployMode=docker-desktop. If then, why do we have -Dtest.exclude.tags=minikube?
we set the docker-desktop, maybe some display error on github?
> Second, in the same way, we had better use -Dspark.kubernetes.test.imageTag=2022-03-06 consistently.
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35772:
URL: https://github.com/apache/spark/pull/35772
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r821917465
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -288,6 +288,16 @@ The following is an example to rerun tests with the pre-built image.
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/runIts'
+You can also specify your specific dockerfile to build JVM/Python/R based image to test:
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
+ -Dspark.kubernetes.test.rDockerFile=/path/to/Dockerfile.r \
+ 'kubernetes-integration-tests/test'
Review comment:
Could you move this to the end of this doc, @Yikun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822237614
##########
File path: project/SparkBuild.scala
##########
@@ -646,6 +646,10 @@ object KubernetesIntegrationTests {
val javaImageTag = sys.props.get("spark.kubernetes.test.javaImageTag")
val dockerFile = sys.props.getOrElse("spark.kubernetes.test.dockerFile",
s"$sparkHome/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17")
+ val pyDockerFile = sys.props.getOrElse("spark.kubernetes.test.pyDockerFile",
+ s"$bindingsDir/python/Dockerfile")
+ val rDockerFile = sys.props.getOrElse("spark.kubernetes.test.rDockerFile",
+ s"$bindingsDir/R/Dockerfile")
Review comment:
indentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822276438
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -294,3 +294,13 @@ In addition, you can run a single test selectively.
-Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very long application name"'
+
+You can also specify your specific dockerfile to build JVM/Python/R based image to test.
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
Review comment:
I have to say sorry, I only tested it with new comming configurations. will do a complete test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822276528
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -294,3 +294,13 @@ In addition, you can run a single test selectively.
-Dspark.kubernetes.test.deployMode=docker-desktop \
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very long application name"'
+
+You can also specify your specific dockerfile to build JVM/Python/R based image to test.
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
Review comment:
/path/to/py/Dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35772: [SPARK-38452][K8S][TESTS] Support pyDockerfile and rDockerfile in SBT K8s IT
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35772:
URL: https://github.com/apache/spark/pull/35772#discussion_r822228161
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -288,6 +288,16 @@ The following is an example to rerun tests with the pre-built image.
-Dspark.kubernetes.test.imageTag=2022-03-06 \
'kubernetes-integration-tests/runIts'
+You can also specify your specific dockerfile to build JVM/Python/R based image to test:
+
+ build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ -Dspark.kubernetes.test.DockerFile=/path/to/Dockerfile \
+ -Dspark.kubernetes.test.pyDockerFile=/path/to/Dockerfile.py \
+ -Dspark.kubernetes.test.rDockerFile=/path/to/Dockerfile.r \
+ 'kubernetes-integration-tests/test'
Review comment:
done, :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org