[GitHub] [spark] shockdm commented on a change in pull request #29496: [SPARK-24266][K8S][2.4] Restart the watcher when we receive a version changed from k8s

[GitBox <gi...@apache.org>]

shockdm commented on a change in pull request #29496:
URL: https://github.com/apache/spark/pull/29496#discussion_r482203205



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala
##########
@@ -177,4 +188,27 @@ private[k8s] class LoggingPodStatusWatcherImpl(
   private def formatTime(time: String): String = {
     if (time != null ||  time != "") time else "N/A"
   }
+
+  override def watchOrStop(sId: String): Boolean = if (hasCompleted()) {

Review comment:
       This line is different in the `master` branch. It check whether the client was set to wait for the completion of the driver, not simply checking `hasCompleted()`, which should normally return false - causing `spark-submit` to terminate immediately. 
   
   For `2.4.7` port you may want to add additional parameter to the `LoggingPodStatusWatcherImpl`, specifically `waitForCompletion: Boolean`, that can be checked in this if statement. In current master they are using `KubernetesDriverConf` instead of passing parameters separately: https://github.com/apache/spark/blob/5e6173ebefcbf3d85ba10234b35fe043f8497d32/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala#L40. 
   
   However in `2.4.6`, `KubernetesDriverConf` is not yet introduced, and instead the `appId` and `maybeLoggingInterval` are passed individually. Simple solution is to add `waitForCompletion` as a parameter, and use it in the above if statement. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org