You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/29 16:55:51 UTC
[GitHub] [spark] Reamer opened a new pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Reamer opened a new pull request #34143:
URL: https://github.com/apache/spark/pull/34143
### What changes were proposed in this pull request?
This pull request removes the configuration that disables the WebsocketPingInterval. The default value of the [fabric8io/kubernetes-client](https://github.com/fabric8io/kubernetes-client/blob/master/README.md) library is 30 seconds.
### Why are the changes needed?
With a periodical websocket ping, the tunnel timeout in loadbalancers should work as expected.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
It is not possible for me to write a test for this particular infrastructure. I have checked the setting locally with my infrastructure with a small JUnit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930453612
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930484385
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950625232
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49051/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-951654525
I added you to the Apache Spark contributor group and assigned SPARK-36890 to you.
Welcome to the Apache Spark community, @Reamer .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Reamer commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
Reamer commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950590709
I did a rebase to the current master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950688677
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49051/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Reamer commented on a change in pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #34143:
URL: https://github.com/apache/spark/pull/34143#discussion_r720336905
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
> Are you using this code in your production?
At the moment I do not use this code in production.
> I'm wondering if there is any possibility of side-effect at the corner case.
I don't think we will have any side effects if we remove this hardcoded value. As I wrote, a regular ping supports the intermediate network components to keep the session alive. Of course, we have a bit more network traffic due to the regular ping, but that is desired.
> After this PR removes this hard-coded value, is there a way to control this?
Because Spark uses ConfigBuilder in the following way.
```
ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
```
and autoConfigure has the call [`configFromSysPropsOrEnvVars(config);`](https://github.com/fabric8io/kubernetes-client/blob/61bdd64c15cb951cf6fda8260202d086d1ef16c9/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L273-L289), we should be able to set a custom value via an environment variable or system property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950591750
**[Test build #144580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144580/testReport)** for PR 34143 at commit [`7d18cc7`](https://github.com/apache/spark/commit/7d18cc7554bff3a38119b572a9733e9b6962278d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Reamer commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
Reamer commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-946446027
How can we move forward with the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930552743
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48238/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34143:
URL: https://github.com/apache/spark/pull/34143#discussion_r718801875
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
Thank you for making a PR, @Reamer . This is an ancient code came from SPARK-18278.
- https://github.com/apache/spark/pull/19468/files#diff-16671cf9d0bff95338158f5fe51d661c482abbcf0fba8515d920849effad66ebR69
Are you using this code in your production? I'm wondering if there is any possibility of side-effect at the corner case, @Reamer . After this PR removes this hard-coded value, is there a way to control this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950631542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144580/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950603989
**[Test build #144580 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144580/testReport)** for PR 34143 at commit [`7d18cc7`](https://github.com/apache/spark/commit/7d18cc7554bff3a38119b572a9733e9b6962278d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950688677
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49051/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34143:
URL: https://github.com/apache/spark/pull/34143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950661130
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49051/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Reamer commented on a change in pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #34143:
URL: https://github.com/apache/spark/pull/34143#discussion_r721146856
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
> Are you using this code in your production?
Today I took the time to [build Spark with the fix](https://github.com/apache/spark/compare/v3.1.2...Reamer:k8s_api_websocket_ping_3_1) by myself and with that I can confirm the fix is working in my production environment.
I was able to reproduce the erroneous behavior using the environment variable `CUBERNET_WEBSOCKET_PING_INTERVAL`. Other users can adjust this environment variable for their needs, but I think that the default value from the library is more than sufficient.
```
pdallig@W-PDALLIG:~/Apps/spark-3.1.2-bin-hadoop3.2$ export KUBERNETES_WEBSOCKET_PING_INTERVAL=0
pdallig@W-PDALLIG:~/Apps/spark-3.1.2-bin-hadoop3.2$ ./bin/pyspark
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
21/10/04 10:16:29 WARN Utils: Your hostname, W-PDALLIG resolves to a loopback address: 127.0.1.1; using 172.16.56.172 instead (on interface eno1)
21/10/04 10:16:29 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/pdallig/Apps/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/10/04 10:16:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/10/04 10:16:31 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
21/10/04 10:16:31 INFO KubernetesClientUtils: Spark configuration files loaded from Some(/home/pdallig/Apps/spark-3.1.2-bin-hadoop3.2/conf) : log4j.properties,spark-env.sh
21/10/04 10:16:31 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/10/04 10:16:31 INFO KubernetesClientUtils: Spark configuration files loaded from Some(/home/pdallig/Apps/spark-3.1.2-bin-hadoop3.2/conf) : log4j.properties,spark-env.sh
21/10/04 10:16:32 INFO KubernetesClientUtils: Spark configuration files loaded from Some(/home/pdallig/Apps/spark-3.1.2-bin-hadoop3.2/conf) : log4j.properties,spark-env.sh
21/10/04 10:16:32 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.1.2
/_/
Using Python version 3.6.9 (default, Jan 26 2021 15:33:00)
Spark context Web UI available at http://172.16.56.172:4040
Spark context available as 'sc' (master = k8s://https://api.ocp.cloud.mycompany.com:443, app id = spark-application-1633335391515).
SparkSession available as 'spark'.
>>> 21/10/04 10:17:15 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
21/10/04 10:17:56 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Traceback (most recent call last):
File "/home/pdallig/Apps/spark-3.1.2-bin-hadoop3.2/python/pyspark/context.py", line 285, in signal_handler
raise KeyboardInterrupt()
KeyboardInterrupt
>>>
21/10/04 10:18:04 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
```
@dongjoon-hyun How can I help further so that the pull request is merged? Should I prepare pull requests for the stable branches?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930506638
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48238/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930484385
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930474828
**[Test build #143727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143727/testReport)** for PR 34143 at commit [`c72b5ce`](https://github.com/apache/spark/commit/c72b5ce70953eca86f9ab7212bbde6d3d38200c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Reamer commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
Reamer commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-951668867
Thanks @dongjoon-hyun
Should we backport this feature to lower versions of Spark (e.g. 3.2, 3.1, 3.0)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930453612
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950631542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144580/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34143:
URL: https://github.com/apache/spark/pull/34143#discussion_r718801875
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
This is an ancient code came from SPARK-18278.
- https://github.com/apache/spark/pull/19468/files#diff-16671cf9d0bff95338158f5fe51d661c482abbcf0fba8515d920849effad66ebR69
Are you using this code in your production? I'm wondering if there is any possibility of side-effect at the corner case, @Reamer . After this PR, is there a way to control this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930552743
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48238/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930377838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930377838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930377838
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930474828
**[Test build #143727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143727/testReport)** for PR 34143 at commit [`c72b5ce`](https://github.com/apache/spark/commit/c72b5ce70953eca86f9ab7212bbde6d3d38200c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930377838
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930546336
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48238/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34143: [SPARK-36890] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34143:
URL: https://github.com/apache/spark/pull/34143#discussion_r718801875
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
This is an ancient code came from SPARK-18278.
- https://github.com/apache/spark/pull/19468/files#diff-16671cf9d0bff95338158f5fe51d661c482abbcf0fba8515d920849effad66ebR69
Are you using this code in your production? I'm wondering if there is any possibility of side-effect at the corner case, @Reamer . After this PR, is there a way to control this?
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
This is an ancient code came from SPARK-18278.
- https://github.com/apache/spark/pull/19468/files#diff-16671cf9d0bff95338158f5fe51d661c482abbcf0fba8515d920849effad66ebR69
Are you using this code in your production? I'm wondering if there is any possibility of side-effect at the corner case, @Reamer . After this PR removes this hard-coded value, is there a way to control this?
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
Thank you for making a PR, @Reamer . This is an ancient code came from SPARK-18278.
- https://github.com/apache/spark/pull/19468/files#diff-16671cf9d0bff95338158f5fe51d661c482abbcf0fba8515d920849effad66ebR69
Are you using this code in your production? I'm wondering if there is any possibility of side-effect at the corner case, @Reamer . After this PR removes this hard-coded value, is there a way to control this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930474828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930474828
**[Test build #143727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143727/testReport)** for PR 34143 at commit [`c72b5ce`](https://github.com/apache/spark/commit/c72b5ce70953eca86f9ab7212bbde6d3d38200c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34143:
URL: https://github.com/apache/spark/pull/34143#discussion_r718801875
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
##########
@@ -88,7 +88,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
.withApiVersion("v1")
.withMasterUrl(master)
- .withWebsocketPingInterval(0)
Review comment:
This is an ancient code came from SPARK-18278.
- https://github.com/apache/spark/pull/19468/files#diff-16671cf9d0bff95338158f5fe51d661c482abbcf0fba8515d920849effad66ebR69
Are you using this code in your production? I'm wondering if there is any possibility of side-effect at the corner case, @Reamer . After this PR removes this hard-coded value, is there a way to control this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-930484142
**[Test build #143727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143727/testReport)** for PR 34143 at commit [`c72b5ce`](https://github.com/apache/spark/commit/c72b5ce70953eca86f9ab7212bbde6d3d38200c0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-950591750
**[Test build #144580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144580/testReport)** for PR 34143 at commit [`7d18cc7`](https://github.com/apache/spark/commit/7d18cc7554bff3a38119b572a9733e9b6962278d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-931516589
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34143: [SPARK-36890][K8S] Use default WebsocketPingInterval for Kubernetes watches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34143:
URL: https://github.com/apache/spark/pull/34143#issuecomment-931516589
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org