[jira] [Created] (SPARK-50337) Error logged due to race condition when shutting down kubernetes client

["Rocco Verhoef (Jira)" <ji...@apache.org>]

Rocco Verhoef created SPARK-50337:
-------------------------------------

             Summary: Error logged due to race condition when shutting down kubernetes client
                 Key: SPARK-50337
                 URL: https://issues.apache.org/jira/browse/SPARK-50337
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes, Spark Core
    Affects Versions: 3.5.3
            Reporter: Rocco Verhoef


While running spark 3.5.3 on kubernetes, we sometimes get the error message below in our logs.

Looking throught the code, we start 2 components to validate driver/executor state in the k8 cluster. One is `ExecutorPodsWatchSnapshotSource`, which does a watch on all events coming k8 and is fine. The other is `ExecutorPodsPollingSnapshotSource`, which periodically pulls the snapshot state from k8. which is likely there in case the event watcher gets disconnects for a period (standard pattern).

Class `ExecutorPodsPollingSnapshotSource` has a thread running periodically in the background to fetch the snapshot. On shutdown, it interrupts the future. Depending where the future is at, this leads to different types of exceptions being raised within that thread. If it raises on `InterruptedException`, then class `Utils.tryLogNonFatalError` swallows it and doesn't print an error. But, as per below, the  `InterruptedException` gets wrapped then it prints the below error.

I guess the solution should be in the direction of checking the root cause of the exception.

 

```
WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
ERROR Utils: Uncaught exception in thread kubernetes-executor-pod-polling-sync
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list] for kind: [Pod] with name: [null] in namespace: [test-namespace] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:422)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:388)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:92)
at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource$PollRunnable.$anonfun$run$1(ExecutorPodsPollingSnapshotSource.scala:92)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1375)
at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsPollingSnapshotSource$PollRunnable.run(ExecutorPodsPollingSnapshotSource.scala:75)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.InterruptedIOException
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:505)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:420)
... 11 more
Caused by: java.lang.InterruptedException
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:502)
... 12 more
```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org