[GitHub] [spark] redsk commented on pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

[GitBox <gi...@apache.org>]

redsk commented on pull request #29533:
URL: https://github.com/apache/spark/pull/29533#issuecomment-714650777


   @jkleckner I tried this patch in production but it does not seem to work. 
   
   Distribution:
   ```
   git clone https://github.com/apache/spark.git
   git checkout branch-3.0
   git fetch origin pull/29533/head:backport-SPARK-24266-to-branch-3.0
   git checkout backport-SPARK-24266-to-branch-3.0
   git checkout -b rebased-backport-SPARK-24266-to-branch-3.0
   git rebase branch-3.0
   export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g"
   ./dev/make-distribution.sh --name spark-3.0-24266 --tgz -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
   ```
   
   Image creation:
   ```
   ./bin/docker-image-tool.sh -r my-registry -t my-tag -n -u 0 -b java_image_tag=11-jre-slim build
   ./bin/docker-image-tool.sh -r my-registry -t my-tag push
   ```
   
   When I execute my long-running spark application I get
   ```
   ...
   20/10/22 17:30:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
   io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 1543007015 (1543067888)
   	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
   	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
   	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
   	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
   	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
   	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
   	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
   	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   	at java.base/java.lang.Thread.run(Unknown Source)
   ```
   
   I'm not sure if I made a mistake or there's a problem in the patch. Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org