You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/08 15:37:32 UTC
[GitHub] [spark] Yikun opened a new pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Yikun opened a new pull request #35773:
URL: https://github.com/apache/spark/pull/35773
### What changes were proposed in this pull request?
There are some requirements on “Minikube should run with a minimum of 4 CPUs and 6G of memory” on [IT README](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/integration-tests/README.md).
### Why are the changes needed?
Make sure Volcano IT can be completed in `4U6G` cluster.
### Does this PR introduce _any_ user-facing change?
No, test only
### How was this patch tested?
```
$ k get pod
Capacity:
cpu: 4
memory: 6081840Ki
$ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
-Dtest.exclude.tags=minikube -Dtest.include.tags=volcano -Dspark.kubernetes.test.namespace=default \
-Dspark.kubernetes.test.deployMode=docker-desktop "kubernetes-integration-tests/test"
[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (9 seconds, 19 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (17 seconds, 633 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (15 seconds, 220 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (16 seconds, 413 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (16 seconds, 333 milliseconds)
[info] Run completed in 1 minute, 19 seconds.
[info] Total number of tests run: 5
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35773:
URL: https://github.com/apache/spark/pull/35773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823355970
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
Review comment:
Please add an example for Minikube start command for this explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353188
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
Review comment:
Please don't miss `-Psparkr` because we can run SparkR tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063505082
In theory, min res for IT in theory: `driver res * n parallel + executor res * 1 + extra left`.
- min cpu requirement: `1 * n + 1 * 1 + 1`
- min mem requirment: `1.5 * n + 1.5 * 1 + 1`
- for queue in this PR, need about 6U9G before
- for priority in this PR, need about 5U7G before.
- Left volcano tests can be compelted in 4U6G
I just thought can we reduce the limit of volcano IT. 😂 otherwise, need more res for dev and ci.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063251732
If you are busy, let me make a documentation PR for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1065784315
Thanks, @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1064337812
Actually, I reported this issue from the first time and I hope it was resolved at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062402651
@dongjoon-hyun emm, to make sure not misleading, we also can't create >= 3 jobs in parallel without volcano in 4U6G cluster.
So, better to make the volcano test case have less resources if not break any regression.
Doc is also need to be added will address in next patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062402651
@dongjoon-hyun emm, to make sure not misleading, we also can't create 2 jobs in parallel without volcano in 4U6G cluster.
So, better to make the test case have less resources if not break any regression.
Doc is also need to be added will address in next patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062406351
I will also test all test cases (including previous inherit k8s suite) to make sure can passed in 4U6G env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][TESTS] Add K8S IT doc for volcano test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353462
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
Review comment:
Let's remove `all Kubernetes and` part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1064335951
Hi, @Yikun . It seems that the volcano image is outdated. This instruction only work on Minikube and doesn't work in the production EKS ARM64 environment. Could you describe the limitation?
```
$ k apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development-arm64.yaml
$ k get pods -n volcano-system
NAME READY STATUS RESTARTS AGE
volcano-admission-54b4798bff-jkww9 0/1 CrashLoopBackOff 6 6m26s
volcano-admission-init-9s6sh 0/1 Completed 0 6m26s
volcano-controllers-865f496899-6mmqq 0/1 CrashLoopBackOff 6 6m24s
volcano-scheduler-5f8844cd99-nlmgl 1/1 Running 0 6m22s
$ k logs volcano-admission-54b4798bff-jkww9 -n volcano-system
unknown flag: --admission-conf
Usage of /vc-webhook-manager:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1064980529
```
docker rmi volcanosh/vc-scheduler-arm64:latest
docker rmi volcanosh/vc-webhook-manager-arm64:latest
docker rmi volcanosh/vc-controller-manager-arm64:latest
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development-arm64.yaml
```
Could you try to first cleanup image then apply new installation?
This is really not a good UX to install latest multiple times, I also submit a issue: https://github.com/volcano-sh/volcano/issues/2072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063652894
```
[info] VolcanoSuite:
[info] - Run SparkPi with no resources (11 seconds, 363 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (12 seconds, 181 milliseconds)
[info] - Run SparkPi with a very long application name. (10 seconds, 876 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (11 seconds, 932 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (10 seconds, 756 milliseconds)
[info] - Run SparkPi with an argument. (10 seconds, 989 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (13 seconds, 562 milliseconds)
[info] - All pods have the same service account by default (10 seconds, 703 milliseconds)
[info] - Run extraJVMOptions check on driver (5 seconds, 625 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (10 seconds, 795 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (16 seconds, 211 milliseconds)
[info] - Run SparkPi with env and mount secrets. (19 seconds, 830 milliseconds)
[info] - Run PySpark on simple pi.py example (11 seconds, 677 milliseconds)
[info] - Run PySpark to test a pyfiles example (16 seconds, 518 milliseconds)
[info] - Run PySpark with memory customization (11 seconds, 920 milliseconds)
[info] - Run in client mode. (10 seconds, 330 milliseconds)
[info] - Start pod creation from template (13 seconds, 8 milliseconds)
[info] - SPARK-38398: Schedule pod creation from template (12 seconds, 59 milliseconds)
[info] - Test basic decommissioning (45 seconds, 509 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (44 seconds, 664 milliseconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 43 seconds)
[info] - Test decommissioning timeouts (47 seconds, 531 milliseconds)
[info] - SPARK-37576: Rolling decommissioning (1 minute, 7 seconds)
[info] - Run SparkPi with volcano scheduler (10 seconds, 844 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (32 seconds, 654 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (32 seconds, 610 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (14 seconds, 323 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (26 seconds, 385 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (20 seconds, 209 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (17 seconds, 427 milliseconds)
[info] Run completed in 12 minutes, 58 seconds.
[info] Total number of tests run: 30
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 30, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
$ k get node -oymal
capacity:
cpu: "6"
memory: 9159716Ki
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][TESTS] Add K8S IT doc for volcano test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353344
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/test'
Review comment:
Please use `"kubernetes-integration-tests/testOnly *.VolcanoSuite"'`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823355970
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
Review comment:
Please add an example for Minikube start command for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063505082
In theory, min res for IT in theory: `driver res * n parallel + executor res * 1 + extra left`.
- min cpu requirement: `1 * n + 1 * 1 + 1`
- min mem requirment: `1.5 * n + 1.5 * 1 + 1`
- for queue in this PR, need about 6U9G before
- for priority in this PR, need about 5U7G before.
- Other volcano tests can be compelted in 4U6G
I just thought can we reduce the limit of volcano IT. 😂 otherwise, need more res for dev and ci.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1065784068
@dongjoon-hyun Thanks, using v1.5 is reasonable for now.
As I mentioned before, volcano will release v1.5.1 to include some fixes also include we found in spark K8S IT issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1064335951
Hi, @Yikun . It seems that the volcano image is outdated. This instruction only work on Minikube and doesn't work in the production EKS ARM64 environment. Could you describe the limitation?
```
$ k get pods -n volcano-system
NAME READY STATUS RESTARTS AGE
volcano-admission-54b4798bff-jkww9 0/1 CrashLoopBackOff 6 6m26s
volcano-admission-init-9s6sh 0/1 Completed 0 6m26s
volcano-controllers-865f496899-6mmqq 0/1 CrashLoopBackOff 6 6m24s
volcano-scheduler-5f8844cd99-nlmgl 1/1 Running 0 6m22s
$ k logs volcano-admission-54b4798bff-jkww9 -n volcano-system
unknown flag: --admission-conf
Usage of /vc-webhook-manager:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063224341
It seems that you misunderstand my comment. I proposed you to use the higher minimum requirement when you add *volcano* module testing section into [document.](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/integration-tests/README.md)
> @dongjoon-hyun emm, to make sure not misleading, we also **can't create >= 3 jobs in parallel without volcano in 4U6G cluster**.
>
> So, better to make the volcano test case have less resources if not break any regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353344
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/test'
Review comment:
~Please use `"kubernetes-integration-tests/testOnly *.VolcanoSuite"'`.~ Never mind.
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/test'
+
+You can also specify `volcano` tag to only run Volcano test:
Review comment:
~Yes, this is possible, but I guess we don't need this example because `test.include.tags` is preferred in `Maven`-only.~ Never mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823354093
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/test'
+
+You can also specify `volcano` tag to only run Volcano test:
Review comment:
Yes, this is possible, but I guess we don't need this example because `test.include.tags` is preferred in `Maven`-only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062325739
@dongjoon-hyun Yes, it was also failed in 4U6G. My previous env have 8U16G, so it passed.
Cluster have 4U means we can only use about 3U for resources.
Also means we can only submitted <= 2 PI jobs (1 driver + 1 executor) in parallel **if no resource control**. Otherwise, all drivers are running to occupy all resource, executors will pending, then stack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063505082
In theory, min res for IT in theory: `driver res * n parallel + executor res * 1 + extra left`.
- min cpu requirement: `1 * n + 1 * 1 + 1`
- min mem requirment: `1.5 * n + 1.5 * 1 + 1`
- for queue in this PR, need about 6U9G before
- for priority in this PR, need about 5U7G before.
- Left volcano tests can be compelted in 4U6G
I just thought can we reduce the limit of volcano IT. 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353462
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
Review comment:
Let's remove `all Kubernetes and` part because the title of this section is `Running the Volcano Integration Tests`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823355970
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
Review comment:
~Please add an example for Minikube start command for this explicitly.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1064335951
Hi, @Yikun . It seems that the volcano image is outdated. This instruction only work on Minikube and doesn't work in the production EKS ARM64 environment. Could you describe the limitation?
```
$ kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development-arm64.yaml
$ k get pods -n volcano-system
NAME READY STATUS RESTARTS AGE
volcano-admission-54b4798bff-jkww9 0/1 CrashLoopBackOff 6 6m26s
volcano-admission-init-9s6sh 0/1 Completed 0 6m26s
volcano-controllers-865f496899-6mmqq 0/1 CrashLoopBackOff 6 6m24s
volcano-scheduler-5f8844cd99-nlmgl 1/1 Running 0 6m22s
$ k logs volcano-admission-54b4798bff-jkww9 -n volcano-system
unknown flag: --admission-conf
Usage of /vc-webhook-manager:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1065328425
No, we never use `volcanosh/*-latest` again, @Yikun . Please specify the version number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063505082
In theory, minRes for IT in theory: `driver cpu * n parallel + executor cpu * 1 + extra left`.
- min cpu requirement: `1 * n + 1 * 1 + 1`
- min mem requirment: `1.5 * n + 1 * 1.5 + 1`
- for queue in this PR, need about 6U9G before
- for priority in this PR, need about 5U7G before.
- Left volcano tests can be compelted in 4U6G
I just thought can we reduce the limit of volcano IT. 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063521812
@dongjoon-hyun OK, then we can convert this to a doc PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1065328425
No, we never use `latest` again, @Yikun . Please specify the version number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062402651
@dongjoon-hyun emm, to make sure not misleading, we also can't create 2 jobs in parallel without volcano in 4U6G cluster.
So, better to make the volcano test case have less resources if not break any regression.
Doc is also need to be added will address in next patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1064617528
@dongjoon-hyun Thanks for report this, will take a look soon today, also cc @william-wang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][TESTS] Add K8S IT doc for volcano test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353188
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
Review comment:
Please don't miss `-Psparkr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823353462
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
Review comment:
~Let's remove `all Kubernetes and` part because the title of this section is `Running the Volcano Integration Tests`.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062402651
@dongjoon-hyun emm, to make sure not misleading, we also **can't create >= 3 jobs in parallel without volcano in 4U6G cluster**.
So, better to make the volcano test case have less resources if not break any regression.
Doc is also need to be added will address in next patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1065329242
Also, this README should be changed because it turns out this instruction is too fragile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35773: [SPARK-38453][K8S][DOCS] Add `volcano` section to K8s IT `README.md`
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1065784068
@dongjoon-hyun Thanks, using v1.5 is reasonable for now.
As I mentioned before, volcano will release v1.5.1 to include some fixes also include we found in spark K8S IT issues. I will also update this doc to min require to v1.5.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1062402651
@dongjoon-hyun emm, to make sure not misleading, we also can't create 2 jobs in parallel without volcano in 4U6G cluster.
So, better to make the test case have less resource if not break any regression.
Doc is also need to be added will address in next patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063515295
Our first priority is having a complete test coverage for Volcano's all features. It's totally fine to require more resources for Kube **Batch**-style custom schedulers, @Yikun .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35773: [SPARK-38453][K8S][TESTS] Add K8S IT doc for volcano test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35773:
URL: https://github.com/apache/spark/pull/35773#discussion_r823354093
##########
File path: resource-managers/kubernetes/integration-tests/README.md
##########
@@ -305,3 +305,24 @@ You can also specify your specific dockerfile to build JVM/Python/R based image
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
'kubernetes-integration-tests/test'
+
+# Running the Volcano Integration Tests
+
+Prerequisites
+- Install Volcano according to [link](https://volcano.sh/en/docs/installation/).
+- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano test cases.
+
+You can specify `-Pvolcano` to enable volcano module to run all Kubernetes and Volcano tests
+
+ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
+ -Dtest.exclude.tags=minikube \
+ -Dspark.kubernetes.test.deployMode=docker-desktop \
+ 'kubernetes-integration-tests/test'
+
+You can also specify `volcano` tag to only run Volcano test:
Review comment:
Yes, this is possible, but I guess we don't need this because `test.include.tags` is preferred in `Maven`-only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun removed a comment on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun removed a comment on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063652894
```
[info] VolcanoSuite:
[info] - Run SparkPi with no resources (11 seconds, 363 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (12 seconds, 181 milliseconds)
[info] - Run SparkPi with a very long application name. (10 seconds, 876 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (11 seconds, 932 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (10 seconds, 756 milliseconds)
[info] - Run SparkPi with an argument. (10 seconds, 989 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (13 seconds, 562 milliseconds)
[info] - All pods have the same service account by default (10 seconds, 703 milliseconds)
[info] - Run extraJVMOptions check on driver (5 seconds, 625 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (10 seconds, 795 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (16 seconds, 211 milliseconds)
[info] - Run SparkPi with env and mount secrets. (19 seconds, 830 milliseconds)
[info] - Run PySpark on simple pi.py example (11 seconds, 677 milliseconds)
[info] - Run PySpark to test a pyfiles example (16 seconds, 518 milliseconds)
[info] - Run PySpark with memory customization (11 seconds, 920 milliseconds)
[info] - Run in client mode. (10 seconds, 330 milliseconds)
[info] - Start pod creation from template (13 seconds, 8 milliseconds)
[info] - SPARK-38398: Schedule pod creation from template (12 seconds, 59 milliseconds)
[info] - Test basic decommissioning (45 seconds, 509 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (44 seconds, 664 milliseconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 43 seconds)
[info] - Test decommissioning timeouts (47 seconds, 531 milliseconds)
[info] - SPARK-37576: Rolling decommissioning (1 minute, 7 seconds)
[info] - Run SparkPi with volcano scheduler (10 seconds, 844 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (32 seconds, 654 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (32 seconds, 610 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (14 seconds, 323 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (26 seconds, 385 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (20 seconds, 209 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (17 seconds, 427 milliseconds)
[info] Run completed in 12 minutes, 58 seconds.
[info] Total number of tests run: 30
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 30, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
$ k get node -oymal
capacity:
cpu: "6"
memory: 9159716Ki
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35773: [SPARK-38453][K8S][TESTS] Make sure Volcano IT can be completed in `4U6G` cluster
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35773:
URL: https://github.com/apache/spark/pull/35773#issuecomment-1063653384
```
[info] VolcanoSuite:
[info] - Run SparkPi with no resources (11 seconds, 363 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (12 seconds, 181 milliseconds)
[info] - Run SparkPi with a very long application name. (10 seconds, 876 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (11 seconds, 932 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (10 seconds, 756 milliseconds)
[info] - Run SparkPi with an argument. (10 seconds, 989 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (13 seconds, 562 milliseconds)
[info] - All pods have the same service account by default (10 seconds, 703 milliseconds)
[info] - Run extraJVMOptions check on driver (5 seconds, 625 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (10 seconds, 795 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (16 seconds, 211 milliseconds)
[info] - Run SparkPi with env and mount secrets. (19 seconds, 830 milliseconds)
[info] - Run PySpark on simple pi.py example (11 seconds, 677 milliseconds)
[info] - Run PySpark to test a pyfiles example (16 seconds, 518 milliseconds)
[info] - Run PySpark with memory customization (11 seconds, 920 milliseconds)
[info] - Run in client mode. (10 seconds, 330 milliseconds)
[info] - Start pod creation from template (13 seconds, 8 milliseconds)
[info] - SPARK-38398: Schedule pod creation from template (12 seconds, 59 milliseconds)
[info] - Test basic decommissioning (45 seconds, 509 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (44 seconds, 664 milliseconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 43 seconds)
[info] - Test decommissioning timeouts (47 seconds, 531 milliseconds)
[info] - SPARK-37576: Rolling decommissioning (1 minute, 7 seconds)
[info] - Run SparkPi with volcano scheduler (10 seconds, 844 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (32 seconds, 654 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (32 seconds, 610 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (14 seconds, 323 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (26 seconds, 385 milliseconds)
[info] - SPARK-38423: Run SparkPi Jobs with priorityClassName (20 seconds, 209 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (17 seconds, 427 milliseconds)
[info] Run completed in 12 minutes, 58 seconds.
[info] Total number of tests run: 30
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 30, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
$ k get node -oymal
capacity:
cpu: "6"
memory: 9159716Ki
```
All test passed in 6U9G cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org