[GitHub] spark pull request #21067: [SPARK-23980][K8S] Resilient Spark driver on Kube...

[mccheah <gi...@git.apache.org>]

Github user mccheah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21067#discussion_r190961469
  
    --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala ---
    @@ -67,12 +68,19 @@ private[spark] class BasicExecutorFeatureStep(
         }
       private val executorLimitCores = kubernetesConf.get(KUBERNETES_EXECUTOR_LIMIT_CORES)
     
    -  override def configurePod(pod: SparkPod): SparkPod = {
    -    val name = s"$executorPodNamePrefix-exec-${kubernetesConf.roleSpecificConf.executorId}"
    +  // If the driver pod is killed, the new driver pod will try to
    +  // create new executors with the same name, but it will fail
    +  // and hangs indefinitely because a terminating executors blocks
    +  // the creation of the new ones, so to avoid that apply salt
    +  private val executorNameSalt = Random.alphanumeric.take(4).mkString("").toLowerCase
    --- End diff --
    
    Can't you just use `KubernetesClusterSchedulerBackend#applicationId`? Believe that's actually newly generated every time, albeit confusingly since that app ID isn't tied to any fields on the driver itself.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org