You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/19 00:10:07 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
dongjoon-hyun opened a new pull request #32226:
URL: https://github.com/apache/spark/pull/32226
### What changes were proposed in this pull request?
This PR aims to support a new configuration, `spark.kubernetes.driver.service.deleteOnTermination`, to clean up `Driver Service` resource during app termination.
### Why are the changes needed?
The K8s service is one of the important resources and sometimes it's controlled by quota.
```
$ k describe quota
Name: service
Namespace: default
Resource Used Hard
-------- ---- ----
services 1 3
```
Apache Spark creates a service for driver whose lifecycle is the same with driver pod.
It means a new Spark job submission will fails if the number of completed Spark jobs equals the number of service quota.
```
$ k get pod
NAME READY STATUS RESTARTS AGE
org-apache-spark-examples-sparkpi-a32c9278e7061b4d-driver 0/1 Completed 0 31m
org-apache-spark-examples-sparkpi-a9f1f578e721ef62-driver 0/1 Completed 0 78s
$ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 80m
org-apache-spark-examples-sparkpi-a32c9278e7061b4d-driver-svc ClusterIP None <none> 7078/TCP,7079/TCP,4040/TCP 31m
org-apache-spark-examples-sparkpi-a9f1f578e721ef62-driver-svc ClusterIP None <none> 7078/TCP,7079/TCP,4040/TCP 80s
$ k describe quota
Name: service
Namespace: default
Resource Used Hard
-------- ---- ----
services 3 3
$ bin/spark-submit...
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException:
Failure executing: POST at: https://192.168.64.50:8443/api/v1/namespaces/default/services.
Message: Forbidden! User minikube doesn't have permission.
services "org-apache-spark-examples-sparkpi-843f6978e722819c-driver-svc" is forbidden:
exceeded quota: service, requested: services=1, used: services=3, limited: services=3.
```
### Does this PR introduce _any_ user-facing change?
Yes, this PR adds a new configuration, `spark.kubernetes.driver.service.deleteOnTermination`, and enables it by default.
The change is documented at the migration guide.
### How was this patch tested?
Manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822258169
**[Test build #137589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137589/testReport)** for PR 32226 at commit [`5277a0e`](https://github.com/apache/spark/commit/5277a0e6ee21fce3fd72d6e95db55bf9fe6a5da2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822298093
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822687097
Ok, I am fine with being in the safe side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822111816
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42112/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822272719
**[Test build #137589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137589/testReport)** for PR 32226 at commit [`5277a0e`](https://github.com/apache/spark/commit/5277a0e6ee21fce3fd72d6e95db55bf9fe6a5da2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822531897
Thank you for your review and approval, @viirya .
@attilapiros . I fully agree with you technically because I also don't use it, but I want to be careful here and want to make no surprise to all users because this is very old behavior from the initial K8s support. In terms of the release management, in general, for non-correctness behavior changes, we had better provide a fallback for one or two releases. After one or two releases, we can remove this config later and will be no-op safely. In addition, this removal of driver service is not an urgent bug either although this is a production issue in some environments. WDYT?
> I am thinking about the advantage of keeping the driver service (if there is any) after the app terminates.
> Because if we come up with some then we have to mention it in the "Does this PR introduce any user-facing change?" section as those are lost by this change. If there is none we can remove "spark.kubernetes.driver.service.deleteOnTermination" flag.
>
> The logs can be still accessed as we get those from the PODs which are still available. The UI already stopped as the app stopped, right? Then what could be lost by removing the service?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822147118
Hi, @viirya and @attilapiros .
Could you review this please, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822115793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822105620
**[Test build #137538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137538/testReport)** for PR 32226 at commit [`5277a0e`](https://github.com/apache/spark/commit/5277a0e6ee21fce3fd72d6e95db55bf9fe6a5da2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822716221
Merged to master for Apache Spark 3.2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822258169
**[Test build #137589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137589/testReport)** for PR 32226 at commit [`5277a0e`](https://github.com/apache/spark/commit/5277a0e6ee21fce3fd72d6e95db55bf9fe6a5da2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822298093
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822112824
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42112/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822115793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822102144
**[Test build #137538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137538/testReport)** for PR 32226 at commit [`5277a0e`](https://github.com/apache/spark/commit/5277a0e6ee21fce3fd72d6e95db55bf9fe6a5da2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822210338
I am thinking about the advantage of keeping the driver service (if there is any) after the app terminates.
Because if we come up with some then we have to mention it in the "Does this PR introduce any user-facing change?" section as those are lost by this change. If there is none we can remove "spark.kubernetes.driver.service.deleteOnTermination" flag.
The logs can be still accessed as we get those from the PODs which are still available. The UI already stopped as the app stopped, right? Then what could be lost by removing the service?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #32226:
URL: https://github.com/apache/spark/pull/32226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822102144
**[Test build #137538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137538/testReport)** for PR 32226 at commit [`5277a0e`](https://github.com/apache/spark/commit/5277a0e6ee21fce3fd72d6e95db55bf9fe6a5da2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32226: [SPARK-35131][K8S] Support early driver service clean-up during app termination
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32226:
URL: https://github.com/apache/spark/pull/32226#issuecomment-822716108
Thank you so much, @attilapiros !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org