You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/07/29 18:41:03 UTC
(spark) branch branch-4.x updated: [SPARK-55199][K8S][DOC] Improve K8s integration tests `README`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 40404fe546a6 [SPARK-55199][K8S][DOC] Improve K8s integration tests `README`
40404fe546a6 is described below
commit 40404fe546a62508ec7f461622a760184ebe6db5
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Jul 29 11:40:41 2026 -0700
[SPARK-55199][K8S][DOC] Improve K8s integration tests `README`
### What changes were proposed in this pull request?
This PR improves the K8s integration tests `README.md` to match the current scripts and build:
- Document the default mode which builds Docker images directly from the current source tree, and
remove the stale `TODO` which was already implemented.
- Add a new `Running the YuniKorn Integration Tests` section, symmetric to the Volcano section.
- Document missing script flags (`--image-repo`, `--include-tags`, `--exclude-tags`,
`--default-exclude-tags`, `--java-version`, `--hadoop-profile`, `--skip-building-dependencies`)
and properties (`test.include.tags`, `test.exclude.tags`, `test.default.exclude.tags`).
- Fix the default value of `spark.kubernetes.test.dockerFile` and remove an outdated note.
### Why are the changes needed?
The README has drifted from the actual behavior. Accurate documentation helps developers run the
K8s integration tests correctly.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57632 from dongjoon-hyun/SPARK-55199.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 8bee0297e94b1750c86aec76e69afd7145d6b279)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../kubernetes/integration-tests/README.md | 93 +++++++++++++++++++---
1 file changed, 83 insertions(+), 10 deletions(-)
diff --git a/resource-managers/kubernetes/integration-tests/README.md b/resource-managers/kubernetes/integration-tests/README.md
index c8809dd6a516..20f4651d8618 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -5,9 +5,6 @@ title: Spark on Kubernetes Integration Tests
# Running the Kubernetes Integration Tests
-Note that the integration test framework is currently being heavily revised and
-is subject to change.
-
The simplest way to run the integration tests is to install and run Minikube, then run the following from this
directory:
@@ -84,7 +81,11 @@ image tag that you have built by other means already, pass the tag to the test s
where if you still want to use images that were built before by the test framework:
./dev/dev-run-integration-tests.sh --image-tag $(cat target/imageTag.txt)
-
+
+The Docker image repository is customised via the `--image-repo <repo>` flag, this is the repository
+that contains the images when a specific image tag is set, or to which freshly built images are
+pushed. The default is `docker.io/kubespark`.
+
### Customising the Image Names
If your image names do not follow the standard Spark naming convention - `spark`, `spark-py` and `spark-r` - then you can customise the names using several options.
@@ -99,8 +100,10 @@ Alternatively if you use completely custom names then you can set each individua
## Spark Distribution Under Test
-The Spark code to test is handed to the integration test system via a tarball. Here is the option that is used to
-specify the tarball:
+By default, when neither `--spark-tgz` nor `--image-tag` is specified, the Docker images are built
+directly from the current source tree, so no extra option is needed.
+
+Alternatively, the Spark code to test can be handed to the integration test system via a tarball:
* `--spark-tgz <path-to-tgz>` - set `<path-to-tgz>` to point to a tarball containing the Spark distribution to test.
@@ -110,9 +113,6 @@ runnable distribution please see the
[Building Spark](https://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution)
documentation.
-**TODO:** Don't require the packaging of the built Spark artifacts into this tarball, just read them out of the current
-tree.
-
## Customizing the Namespace and Service Account
If no namespace is specified then a temporary namespace will be created and deleted during the test run. Similarly if
@@ -129,6 +129,24 @@ to get, list, watch, and create pods. For clusters with RBAC turned on, it's imp
granted to the service account in the namespace through an appropriate role and role binding. A reference RBAC
configuration is provided in `dev/spark-rbac.yaml`.
+## Selecting the Tests to Run
+
+The tests are tagged, and the set of tests to run can be controlled via the following flags:
+
+* `--include-tags <tags>` - comma separated list of tags to include in addition to the always
+ included `k8s` tag.
+* `--exclude-tags <tags>` - comma separated list of tags to exclude.
+* `--default-exclude-tags <tags>` - overrides the default list of excluded tags, which is
+ `org.apache.spark.deploy.k8s.integrationtest.YuniKornTag`. Pass an empty string to clear the
+ default exclusions, e.g. to run the YuniKorn tests.
+
+## Customising the Build
+
+* `--java-version <version>` - Java version to build Spark and the tests with. The default is `17`.
+* `--hadoop-profile <profile>` - Hadoop Maven profile to build with. The default is `hadoop-3`.
+* `--skip-building-dependencies` - skip rebuilding the modules the integration tests depend on
+ (drops Maven's `-am` flag). Useful for iterating when the current tree is already built.
+
# Running the Test Directly
If you prefer to run just the integration tests directly, then you can customise the behaviour via passing system
@@ -251,7 +269,7 @@ to the wrapper scripts and using the wrapper scripts will simply set these appro
<td>
The path to the custom Dockerfile
</td>
- <td><code>N/A</code></td>
+ <td><code>Dockerfile</code></td>
</tr>
<tr>
<td><code>spark.kubernetes.test.namespace</code></td>
@@ -294,6 +312,29 @@ to the wrapper scripts and using the wrapper scripts will simply set these appro
</td>
<td></td>
</tr>
+ <tr>
+ <td><code>test.include.tags</code></td>
+ <td>
+ Comma separated list of test tags to include, e.g. <code>k8s</code>, <code>minikube</code>,
+ <code>local</code>, <code>r</code>, <code>volcano</code>.
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td><code>test.exclude.tags</code></td>
+ <td>
+ Comma separated list of test tags to exclude.
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td><code>test.default.exclude.tags</code></td>
+ <td>
+ The default list of excluded test tags. Set to an empty string to clear the default
+ exclusions, e.g. to run the YuniKorn tests.
+ </td>
+ <td><code>org.apache.spark.deploy.k8s.integrationtest.YuniKornTag</code></td>
+ </tr>
</table>
# Running the Kubernetes Integration Tests with SBT
@@ -363,3 +404,35 @@ You can also specify `volcano` tag to only run Volcano test:
kubectl delete -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.14.2/installer/volcano-development.yaml
+# Running the YuniKorn Integration Tests
+
+`YuniKornSuite` runs the Kubernetes integration tests with the
+[Apache YuniKorn](https://yunikorn.apache.org/) scheduler instead of the default scheduler.
+The suite is tagged with `YuniKornTag` which is excluded by default via
+`test.default.exclude.tags`.
+
+## Requirements
+
+- Apache YuniKorn 1.8.0.
+
+## Installation
+
+ helm repo add yunikorn https://apache.github.io/yunikorn-release
+ helm repo update
+ helm install yunikorn yunikorn/yunikorn --namespace yunikorn --version 1.8.0 \
+ --create-namespace --set embedAdmissionController=false
+
+## Run tests
+
+Clear the default excluded tags to enable the YuniKorn tests:
+
+ build/sbt -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dtest.default.exclude.tags= \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/testOnly *YuniKornSuite'
+
+## Cleanup YuniKorn
+
+ helm uninstall yunikorn --namespace yunikorn
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org