You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/08 09:13:41 UTC
[GitHub] [spark] attilapiros opened a new pull request #33261: [K8S][SPARK-35334] Make Spark more resilient to intermittent K8s flakiness
attilapiros opened a new pull request #33261:
URL: https://github.com/apache/spark/pull/33261
### What changes were proposed in this pull request?
Setting `kubernetes.request.retry.backoffLimit` by default to 3 when the user haven't specified any value for it.
This way when k8s API servers gives back HTTP response >=500 then an exponential backoff will be triggered (where `kubernetes.request.retry.backoffInterval` is 1000ms by default).
For details please check https://github.com/fabric8io/kubernetes-client/issues/3087.
### Why are the changes needed?
We experienced some internal K8s errors for example when the `etcdserver` leader election was ongoing the error was propagated to the API client and caused an issue in Spark:
```
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at:
https://kubernetes.default.svc/api/v1/namespaces/dex-app-bl24w4z9/pods/sparkpi-10-fcd3f6781a874212-driver. Message: etcdserver:
leader changed. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=etcdserver: leader changed,
metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null,
status=Failure, additionalProperties={}).
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Running the integration tests along with `log4j.logger.org.apache.spark.deploy.k8s.SparkKubernetesClientFactory=DEBUG` the log4j config. It produced the following log:
```
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: 21/07/08 11:01:14 DEBUG org.apache.spark.deploy.k8s.SparkKubernetesClientFactory: Kubernetes client config: {
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestConfig" : {
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "username" : null,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "password" : null,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "oauthToken" : null,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "oauthTokenProvider" : null,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "impersonateUsername" : null,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "impersonateGroups" : [ null ],
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "impersonateExtras" : { },
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "watchReconnectInterval" : 1000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "watchReconnectLimit" : -1,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "connectionTimeout" : 10000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "uploadConnectionTimeout" : 10000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "uploadRequestTimeout" : 120000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestRetryBackoffLimit" : 3,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestRetryBackoffInterval" : 1000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestTimeout" : 10000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "rollingTimeout" : 900000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "scaleTimeout" : 600000,
21/07/08 11:01:14.873 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "loggingInterval" : 20000,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "websocketTimeout" : 5000,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "websocketPingInterval" : 0,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "maxConcurrentRequests" : 64,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "maxConcurrentRequestsPerHost" : 5,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "impersonateGroup" : null
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "contexts" : [ {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "context" : {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "cluster" : "talos-default",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "namespace" : "default",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "user" : "admin@talos-default"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "admin@talos-default"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: }, {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "context" : {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "cluster" : "arn:aws:eks:us-west-2:392479084068:cluster/mow",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "user" : "arn:aws:eks:us-west-2:392479084068:cluster/mow"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "arn:aws:eks:us-west-2:392479084068:cluster/mow"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: }, {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "context" : {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "cluster" : "minikube",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "extensions" : [ {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "context_info"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: } ],
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "namespace" : "default",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "user" : "minikube"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "minikube"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: }, {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "context" : {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "cluster" : "",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "user" : ""
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "mow"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: } ],
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "currentContext" : {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "context" : {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "cluster" : "minikube",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "extensions" : [ {
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "context_info"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: } ],
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "namespace" : "default",
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "user" : "minikube"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "name" : "minikube"
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: },
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "maxConcurrentRequests" : 64,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "maxConcurrentRequestsPerHost" : 5,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "autoConfigure" : false,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "trustCerts" : false,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "disableHostnameVerification" : false,
21/07/08 11:01:14.874 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "masterUrl" : "https://192.168.64.127:8443/",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "apiVersion" : "v1",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "namespace" : "a0993113b8084cd3868b3052e698b17f",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "caCertFile" : "/Users/attilazsoltpiros/.minikube/ca.crt",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "clientCertFile" : "/Users/attilazsoltpiros/.minikube/profiles/minikube/client.crt",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "clientKeyFile" : "/Users/attilazsoltpiros/.minikube/profiles/minikube/client.key",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "clientKeyAlgo" : "RSA",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "clientKeyPassphrase" : "changeit",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "watchReconnectInterval" : 1000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "watchReconnectLimit" : -1,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "connectionTimeout" : 10000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "uploadConnectionTimeout" : 10000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "uploadRequestTimeout" : 120000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestRetryBackoffLimit" : 3,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestRetryBackoffInterval" : 1000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestTimeout" : 10000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "rollingTimeout" : 900000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "scaleTimeout" : 600000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "loggingInterval" : 20000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "websocketTimeout" : 5000,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "websocketPingInterval" : 0,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "impersonateGroups" : [ null ],
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "impersonateExtras" : { },
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "http2Disable" : false,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "noProxy" : [ ],
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "tlsVersions" : [ "TLS_1_2" ],
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "errorMessages" : {
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "401" : "Unauthorized! Token may have expired! Please log-in again.",
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "403" : "Forbidden! User minikube doesn't have permission."
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: }
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: }
```
Which contains the expected values:
```
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestRetryBackoffLimit" : 3,
21/07/08 11:01:14.875 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: "requestRetryBackoffInterval" : 1000,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876322103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877430429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876275413
**[Test build #140786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140786/testReport)** for PR 33261 at commit [`ddb0d4f`](https://github.com/apache/spark/commit/ddb0d4f4c04d2147b9f1cfeedacf3b2060a9881a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877413439
**[Test build #140866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140866/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876311037
**[Test build #140792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140792/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877326253
For sure, I'm interested in all K8s stuff w/Spark thanks for looping me in :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877430429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877342017
Kubernetes integration test unable to build dist.
exiting with code: 141
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876322103
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140792/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876310080
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45300/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876275413
**[Test build #140786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140786/testReport)** for PR 33261 at commit [`ddb0d4f`](https://github.com/apache/spark/commit/ddb0d4f4c04d2147b9f1cfeedacf3b2060a9881a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876331400
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45300/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876311037
**[Test build #140792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140792/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-879619575
According to https://github.com/fabric8io/kubernetes-client/issues/3087, this is only applicable at 5.5.0+?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #33261:
URL: https://github.com/apache/spark/pull/33261#discussion_r668029007
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -74,6 +77,11 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
kubeContext.map("context " + _).getOrElse("current context") +
" from users K8S config file")
+ // if backoff limit is not set then set it to 3
+ if (getSystemPropertyOrEnvVar(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY) == null) {
+ System.setProperty(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY, "3")
+ }
+
Review comment:
That would be too late and its value would be set already to 0 and not null.
See this line:
https://github.com/attilapiros/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L418
So we cannot not differentiate whether 0 is set as an intended value chosen by the user or just the default value of the kubernetes-client.
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -102,6 +110,8 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val httpClientWithCustomDispatcher = baseHttpClient.newBuilder()
.dispatcher(dispatcher)
.build()
+ logDebug("Kubernetes client config: " +
Review comment:
I would like to keep it as close to `DefaultKubernetesClient` as possible.
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -102,6 +110,8 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val httpClientWithCustomDispatcher = baseHttpClient.newBuilder()
.dispatcher(dispatcher)
.build()
+ logDebug("Kubernetes client config: " +
Review comment:
As that is the one which we are mostly interested in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877406091
**[Test build #140866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140866/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876276854
**[Test build #140786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140786/testReport)** for PR 33261 at commit [`ddb0d4f`](https://github.com/apache/spark/commit/ddb0d4f4c04d2147b9f1cfeedacf3b2060a9881a).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877406091
**[Test build #140866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140866/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876669469
cc @holdenk (this is not for upscaling still it might be interesting for you)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876276878
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140786/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros closed pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros closed pull request #33261:
URL: https://github.com/apache/spark/pull/33261
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876340985
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876345401
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877458467
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876346243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #33261:
URL: https://github.com/apache/spark/pull/33261#discussion_r667965962
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -74,6 +77,11 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
kubeContext.map("context " + _).getOrElse("current context") +
" from users K8S config file")
+ // if backoff limit is not set then set it to 3
+ if (getSystemPropertyOrEnvVar(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY) == null) {
+ System.setProperty(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY, "3")
+ }
+
Review comment:
The [autoConfigure](https://github.com/fabric8io/kubernetes-client/blob/4e4dcc73bfd6e0d811e3aba32ffa7956b71d951d/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L418) helps us set backofflimit with [priority](https://github.com/fabric8io/kubernetes-client/blob/5bfcac2d8a144f66b1323df5988dd2761fc7e561/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/utils/Utils.java#L80-L92) :
1. System.getProperty(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY)
2. System.getenv(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY)
3. then [config.getRequestRetryBackoffLimit()](https://github.com/fabric8io/kubernetes-client/blob/4e4dcc73bfd6e0d811e3aba32ffa7956b71d951d/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L418)
So, it works, but looks like a little bit flaky for me (it dependents more internal implements of k8s-client), how about respects user property/env/config after config build() like:
```
if config.getRequestRetryBackoffLimit() != null:
config.setRequestRetryBackoffLimit("3")
```
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -102,6 +110,8 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val httpClientWithCustomDispatcher = baseHttpClient.newBuilder()
.dispatcher(dispatcher)
.build()
+ logDebug("Kubernetes client config: " +
Review comment:
nit: I think the log can be moved before L109(`HttpClientUtils.createHttpClient(config)`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-879864718
> According to [fabric8io/kubernetes-client#3087](https://github.com/fabric8io/kubernetes-client/issues/3087), this is only applicable at 5.5.0+?
@dongjoon-hyun Yes, you're right. This is why we needed https://github.com/apache/spark/pull/33233.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877327189
Jenkins retest this please.
I really like this, something I've run into when I've turned on fail-over on my own clusters.
LGTM if you've run the the K8s integration tests locally, otherwise LGTM pending passing K8s integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877426006
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877403954
Jenkins retest this please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876276878
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140786/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #33261:
URL: https://github.com/apache/spark/pull/33261#discussion_r668308678
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -74,6 +77,11 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
kubeContext.map("context " + _).getOrElse("current context") +
" from users K8S config file")
+ // if backoff limit is not set then set it to 3
+ if (getSystemPropertyOrEnvVar(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY) == null) {
+ System.setProperty(KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY, "3")
+ }
+
Review comment:
> So we cannot not differentiate whether 0 is set as an intended value chosen by the user or just the default value of the kubernetes-client.
OK, thanks for the explanation, make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877443420
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876321879
**[Test build #140792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140792/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877458467
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-879027857
Thanks for the reviews! Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-877333314
**[Test build #140859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140859/testReport)** for PR 33261 at commit [`64df5f1`](https://github.com/apache/spark/commit/64df5f1180fa7f62f87cf695dfd45b9233e646ed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33261: [SPARK-35334][K8S] Make Spark more resilient to intermittent K8s flakiness
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33261:
URL: https://github.com/apache/spark/pull/33261#issuecomment-876276854
**[Test build #140786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140786/testReport)** for PR 33261 at commit [`ddb0d4f`](https://github.com/apache/spark/commit/ddb0d4f4c04d2147b9f1cfeedacf3b2060a9881a).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org