You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HwangDongJun (via GitHub)" <gi...@apache.org> on 2026/10/19 04:49:16 UTC
Re: [PR] [SPARK-38079][K8S] Gate the driver pod until the resources it mounts exist [spark]
HwangDongJun commented on code in PR #58187:
URL: https://github.com/apache/spark/pull/58187#discussion_r4310415737
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala:
##########
@@ -131,48 +147,80 @@ private[spark] class Client(
.withName(configMapName)
.endConfigMap()
.endVolume()
+ .addToNodeSelector(
+ pinnedNodeName.map(nodeName => Map("kubernetes.io/hostname" -> nodeName))
+ .getOrElse(Map.empty[String, String]).asJava)
+ // SPARK-38079: holds the pod unschedulable -- so kubelet cannot attempt to mount
+ // anything on it -- until its pre-resources exist (see below). Only removed once
+ // that is true, right before the watch loop; see the "Remove the pre-resources
+ // scheduling gate" step below for why this makes an ownerless-resource window,
+ // and its consequences (a shutdown hook), unnecessary.
+ .addNewSchedulingGate(PRE_RESOURCES_SCHEDULING_GATE)
.endSpec()
.build()
val driverPodName = resolvedDriverPod.getMetadata.getName
- // setup resources before pod creation
- val preKubernetesResources = resolvedDriverSpec.driverPreKubernetesResources
+ // SPARK-38079: the driver's own base config map (mounted as SPARK_CONF_VOLUME_DRIVER
+ // above) must also be created before the pod is schedulable, to avoid a "configmap ...
+ // not found" mount race between the driver pod and the config map it depends on.
+ val preKubernetesResources = resolvedDriverSpec.driverPreKubernetesResources ++ Seq(configMap)
+
+ var watch: Watch = null
+ val createdDriverPod: Pod =
+ try {
+ kubernetesClient.pods().inNamespace(conf.namespace).resource(resolvedDriverPod).create()
Review Comment:
Agreed, and fixed by going back to `master`'s order: `driverPreKubernetesResources` are applied before the pod again, so the `PodGroup` exists at the pod CREATE. The gated pod is created after them, and the pre-resources' owner references are refreshed after the pod exists, as on `master`.
I reproduced this on Volcano v1.11.0. The earlier code was rejected by `validatepod.volcano.sh` with `failed to get PodGroup for pod ...`, and the new code is created normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org