You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/12/04 23:22:04 UTC
[spark] branch master updated: [SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 32922ee [SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17
32922ee is described below
commit 32922eeb3250408523991f21c943de02cd1ebea3
Author: Kousuke Saruta <sa...@oss.nttdata.com>
AuthorDate: Sat Dec 4 15:20:29 2021 -0800
[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17
### What changes were proposed in this pull request?
This PR aims to support K8s integration tests for Java 17 using Maven and SBT.
The new system property `spark.kubernetes.test.dockerFile` is introduced to specify a Dockerfile.
By setting `Dockerfile.java17` to the property, the integration tests run with Java 17.
This PR also revised the change brought by SPARK-37354 (#34628) by changing `SparkBuild.scala` so that it can recognize the system property `spark.kubernetes.test.javaImageTag`, like the integration tests with Maven do.
If both `spark.kubernetes.test.dockerFile` and `spark.kubernetes.test.javaImageTag` are set, `spark.kubernetes.test.dockerFile` is preferred.
### Why are the changes needed?
To ensure Spark works on K8s with Java 17.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Confirmed that the intended version of Java is used for each pattern.
* Neither `spark.kubernetes.test.javaImageTag` nor `spark.kubernetes.test.dockerFile` is set (SBT)
```
# Run the integration tests to create the container image
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests "kubernetes-integration-tests/test"
# Create and login the container.
docker run -it <hash> /bin/bash
185b40ef8aaa56c:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
```
* Neither `spark.kubernetes.test.javaImageTag` nor `spark.kubernetes.test.dockerFile` is set (Maven)
```
# Run the integration tests to create the container image
$ build/mvn -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test
# Create and login the container.
docker run -it <hash> /bin/bash
18536ae5c5c21f4:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
```
* `spark.kubernetes.test.javaImageTag` is set (SBT):
```
# Run the integration tests to create the container image
$ build/sbt -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests "kubernetes-integration-tests/test"
# Create and login the container.
docker run -it <hash> /bin/bash
1855b896d0e5dd8:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)
```
* `spark.kubernetes.test.javaImageTag` is set (Maven):
```
# Run the integration tests to create the container image
$ build/mvn -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test
# Create and login the container.
docker run -it <hash> /bin/bash
1855d3c228e7521:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)
```
* `spark.kubernetes.test.dockerFile` is set (SBT)
```
$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"
# Create and login the container.
docker run -it <hash> /bin/bash
185550515c76582:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```
* `spark.kubernetes.test.dockerFile` is set (Maven)
```
build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test
# Create and login the container.
docker run -it <hash> /bin/bash
185be35bde0ebfa:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```
* Both `spark.kubernetes.test.javaImageTag` and `spark.kubernetes.test.dockerFile` are set (SBT)
```
$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"
# Create and login the container.
docker run -it <hash> /bin/bash
1857ed1891b2921:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```
* Both `spark.kubernetes.test.javaImageTag` and `spark.kubernetes.test.dockerFile` are set (Maven)
```
$ build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test
# Create and login the container.
docker run -it <hash> /bin/bash
18584f3a8b7b4a6:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)
```
Closes #34790 from sarutak/kube-integration-test-java17.
Authored-by: Kousuke Saruta <sa...@oss.nttdata.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
project/SparkBuild.scala | 12 +++++++++---
resource-managers/kubernetes/integration-tests/pom.xml | 4 ++++
.../integration-tests/scripts/setup-integration-test-env.sh | 13 +++++++++++--
3 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index f592ecd..d9e63e8 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -639,13 +639,19 @@ object KubernetesIntegrationTests {
if (shouldBuildImage) {
val dockerTool = s"$sparkHome/bin/docker-image-tool.sh"
val bindingsDir = s"$sparkHome/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings"
+ val dockerFile = sys.props.get("spark.kubernetes.test.dockerFile")
+ val javaImageTag = sys.props.getOrElse("spark.kubernetes.test.javaImageTag", "8-jre-slim")
+ val extraOptions = if (dockerFile.isDefined) {
+ Seq("-f", s"${dockerFile.get}")
+ } else {
+ Seq("-b", s"java_image_tag=$javaImageTag")
+ }
val cmd = Seq(dockerTool, "-m",
"-t", imageTag.value,
"-p", s"$bindingsDir/python/Dockerfile",
- "-R", s"$bindingsDir/R/Dockerfile",
- "-b", s"java_image_tag=${sys.env.getOrElse("JAVA_IMAGE_TAG", "8-jre-slim")}",
+ "-R", s"$bindingsDir/R/Dockerfile") ++
+ extraOptions :+
"build"
- )
val ec = Process(cmd).!
if (ec != 0) {
throw new IllegalStateException(s"Process '${cmd.mkString(" ")}' exited with $ec.")
diff --git a/resource-managers/kubernetes/integration-tests/pom.xml b/resource-managers/kubernetes/integration-tests/pom.xml
index d281e38..74d4ade 100644
--- a/resource-managers/kubernetes/integration-tests/pom.xml
+++ b/resource-managers/kubernetes/integration-tests/pom.xml
@@ -43,6 +43,7 @@
<spark.kubernetes.test.master></spark.kubernetes.test.master>
<spark.kubernetes.test.namespace></spark.kubernetes.test.namespace>
<spark.kubernetes.test.serviceAccountName></spark.kubernetes.test.serviceAccountName>
+ <spark.kubernetes.test.dockerFile>N/A</spark.kubernetes.test.dockerFile>
<test.exclude.tags></test.exclude.tags>
<test.include.tags></test.include.tags>
@@ -112,6 +113,9 @@
<argument>--spark-tgz</argument>
<argument>${spark.kubernetes.test.sparkTgz}</argument>
+ <argument>--docker-file</argument>
+ <argument>${spark.kubernetes.test.dockerFile}</argument>
+
<argument>--test-exclude-tags</argument>
<argument>"${test.exclude.tags}"</argument>
</arguments>
diff --git a/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh b/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
index beda56c..562d1d8 100755
--- a/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
+++ b/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
@@ -26,6 +26,7 @@ IMAGE_TAG="N/A"
JAVA_IMAGE_TAG="8-jre-slim"
SPARK_TGZ="N/A"
MVN="$TEST_ROOT_DIR/build/mvn"
+DOCKER_FILE="N/A"
EXCLUDE_TAGS=""
# Parse arguments
@@ -59,6 +60,10 @@ while (( "$#" )); do
SPARK_TGZ="$2"
shift
;;
+ --docker-file)
+ DOCKER_FILE="$2"
+ shift
+ ;;
--test-exclude-tags)
EXCLUDE_TAGS="$2"
shift
@@ -97,8 +102,12 @@ then
IMAGE_TAG=${VERSION}_$(uuidgen)
cd $SPARK_INPUT_DIR
- # OpenJDK base-image tag (e.g. 8-jre-slim, 11-jre-slim)
- JAVA_IMAGE_TAG_BUILD_ARG="-b java_image_tag=$JAVA_IMAGE_TAG"
+ if [[ $DOCKER_FILE == "N/A" ]]; then
+ # OpenJDK base-image tag (e.g. 8-jre-slim, 11-jre-slim)
+ JAVA_IMAGE_TAG_BUILD_ARG="-b java_image_tag=$JAVA_IMAGE_TAG"
+ else
+ JAVA_IMAGE_TAG_BUILD_ARG="-f $DOCKER_FILE"
+ fi
# Build PySpark image
LANGUAGE_BINDING_BUILD_ARGS="-p $DOCKER_FILE_BASE_PATH/bindings/python/Dockerfile"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org