(spark) branch branch-4.3 updated: [SPARK-38794][K8S] Create executor ConfigMap before requesting executors

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 8253c336b2a5 [SPARK-38794][K8S] Create executor ConfigMap before requesting executors
8253c336b2a5 is described below

commit 8253c336b2a58fb8b8d24716fa4a3b7593b717c1
Author: DongjunHwang <do...@hyundai.com>
AuthorDate: Mon Aug 17 13:28:14 2026 -0700

    [SPARK-38794][K8S] Create executor ConfigMap before requesting executors
    
    ### What changes were proposed in this pull request?
    Reorders two calls in `KubernetesClusterSchedulerBackend.start()` so that the executor
    ConfigMap (`setUpExecutorConfigMap`) is created *before* executor allocation is triggered
    (`podAllocator.setTotalExpectedExecutors`), instead of after. A regression test is added to
    `KubernetesClusterSchedulerBackendSuite` asserting this ordering.
    
    ### Why are the changes needed?
    `podAllocator.setTotalExpectedExecutors()` asynchronously submits executor pod creation
    requests via a background thread pool
    (`ExecutorPodsSnapshotsStoreImpl.notifySubscribers()` -> `subscribersExecutor.submit(...)`).
    Previously, this was called *before* the shared executor ConfigMap was created. This is a
    genuine race: if an executor pod is scheduled and its kubelet attempts to mount the shared
    ConfigMap volume before the driver has finished creating it, the pod fails to start with:
    
    MountVolume.SetUp failed for volume "spark-conf-volume-exec" :
    configmap "spark-exec-...-conf-map" not found
    
    Spark's executor retry logic (and the kubelet's own mount retry) normally recovers from this
    within a few seconds, so today this surfaces only as a transient `FailedMount` warning event
    rather than a hard failure. We reproduced this repeatedly on a production cluster during
    completely normal executor startup (no fault injection needed) and captured it via real-time
    Kubernetes event logs.
    
    This change closes the race window by giving the (synchronous) ConfigMap creation a head
    start over the async executor pod requests. It significantly narrows, but does not perfectly
    eliminate, the window, since a fully deterministic fix (e.g. blocking executor requests until
    the ConfigMap is confirmed synced to all nodes) would be a larger behavioral change.
    
    Note: this addresses a different failure mode than the "ConfigMap creation permanently fails
    (e.g. due to insufficient RBAC permissions) and the driver never retries" scenario also
    described in SPARK-38794. That scenario is not addressed by this change.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    - Added `KubernetesClusterSchedulerBackendSuite` test
      `SPARK-38794: executor ConfigMap is created before executors are requested`, using
      Mockito's `inOrder` to assert the ConfigMap is created before `setTotalExpectedExecutors`
      is called. Verified it fails without the fix and passes with it.
    - Ran the full `kubernetes/core` test module against current master: 381/381 passing.
    - Ran `./dev/scalafmt`; no reformatting was needed for the changed lines.
    - Reproduced and verified the fix against a real Kubernetes cluster across multiple driver
      restarts with dynamic allocation triggering executor creation, observing zero
      `FailedMount`/`configmap not found` events with the fix applied, versus intermittent
      occurrences without it.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Yes. Generated-by: Claude Sonnet 4.5 and Claude Sonnet 5 (via opencode CLI)
    
    Closes #58031 from HwangDongJun/fix/spark-38794-executor-configmap-race.
    
    Authored-by: DongjunHwang <do...@hyundai.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 8b61d2017b4e3b04ae20210808728c716bbb55ec)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../cluster/k8s/KubernetesClusterSchedulerBackend.scala          | 9 ++++++---
 .../cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala     | 9 ++++++++-
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 0784b82a85de..255d76fece56 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -111,15 +111,18 @@ private[spark] class KubernetesClusterSchedulerBackend(
     super.start()
     // Must be called before setting the executors
     podAllocator.start(applicationId(), this)
+    // SPARK-38794: Create the executor ConfigMap before requesting executors. Executor
+    // allocation is asynchronous (background thread pool), so requesting executors first
+    // can race with ConfigMap creation, causing transient "configmap ... not found" mounts.
+    if (!conf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)) {
+      setUpExecutorConfigMap(podAllocator.driverPod)
+    }
     val defaultProfile = scheduler.sc.resourceProfileManager.defaultResourceProfile
     val initExecs = Map(defaultProfile -> initialExecutors)
     podAllocator.setTotalExpectedExecutors(initExecs)
     lifecycleManager.start(this)
     watchEvents.start(applicationId())
     pollEvents.start(applicationId())
-    if (!conf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)) {
-      setUpExecutorConfigMap(podAllocator.driverPod)
-    }
   }
 
   override def stop(): Unit = {
diff --git a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
index cf172eb096d4..5e4ec0ca6115 100644
--- a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
+++ b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
@@ -28,7 +28,7 @@ import io.fabric8.kubernetes.client.dsl.base.PatchContext
 import org.jmock.lib.concurrent.DeterministicScheduler
 import org.mockito.{ArgumentCaptor, Mock, MockitoAnnotations}
 import org.mockito.ArgumentMatchers.{any, eq => mockitoEq}
-import org.mockito.Mockito.{atLeastOnce, mock, never, spy, verify, when}
+import org.mockito.Mockito.{atLeastOnce, inOrder, mock, never, spy, verify, when}
 import org.scalatest.BeforeAndAfter
 
 import org.apache.spark.{SparkConf, SparkContext, SparkEnv, SparkFunSuite}
@@ -160,6 +160,13 @@ class KubernetesClusterSchedulerBackendSuite extends SparkFunSuite with BeforeAn
     verify(configMapResource).create()
   }
 
+  test("SPARK-38794: executor ConfigMap is created before executors are requested") {
+    schedulerBackendUnderTest.start()
+    val ordered = inOrder(configMapResource, podAllocator)
+    ordered.verify(configMapResource).create()
+    ordered.verify(podAllocator).setTotalExpectedExecutors(Map(defaultProfile -> 3))
+  }
+
   test("SPARK-56684: kubernetesClient is exposed within the k8s package") {
     assert(schedulerBackendUnderTest.kubernetesClient eq kubernetesClient)
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org