You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zwangsheng (via GitHub)" <gi...@apache.org> on 2023/02/22 07:04:41 UTC

[GitHub] [spark] zwangsheng opened a new pull request, #40118: [SPARK-26365] In kuberentes + cluster deploy-mode, spark submit should pass driver exit code

zwangsheng opened a new pull request, #40118:
URL: https://github.com/apache/spark/pull/40118

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Find spark driver container exit code and set as spark submit exit code.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Many users face this problem [SPARK-26365](https://issues.apache.org/jira/browse/SPARK-26365)
   
   > When launching apps using spark-submit in a kubernetes cluster, if the Spark applications fails (returns exit code = 1 for example), spark-submit will still exit gracefully and return exit code = 0.
   
   As a comparison, in yarn cluster mode, spark submit will throw exception when application be failed or killed.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   Yes
   
   Before, when application (driver container) on kubernetes fail, spark-submit still return 0 as exit code.
   After this, spark submit will use driver's exit code or throw exception when not found driver's exit code.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   In local; If need unit test, willing to add


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151379705


##########
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcherSuite.scala:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.submit
+
+import io.fabric8.kubernetes.api.model.{PodBuilder, PodStatusBuilder}
+import io.fabric8.kubernetes.client.Watcher.Action
+
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.deploy.k8s.Constants.DEFAULT_DRIVER_CONTAINER_NAME
+import org.apache.spark.deploy.k8s.KubernetesTestConf
+
+class LoggingPodStatusWatcherSuite extends SparkFunSuite {
+
+  private val pod = new PodBuilder()
+    .withNewMetadata()
+      .withName("test-pod")
+      .withNamespace("spark")
+      .withUid("test-spark-uid")
+      .withCreationTimestamp("2020-01-01T00:00:00Z")
+    .endMetadata()
+    .withNewSpec()
+      .withServiceAccount("spark")
+      .withNodeName("test-node")
+    .endSpec()
+    .build()
+
+
+  test("[SPARK-26365] Get watched driver container exit code") {
+    import org.apache.spark.SparkException
+    val sparkConf = new SparkConf()
+    val kubernetesConf = KubernetesTestConf.createDriverConf(sparkConf)
+    val watcher = new LoggingPodStatusWatcherImpl(kubernetesConf)
+
+    val errorCode = 1234
+    val completedCode = 0
+
+    val errorState = new PodStatusBuilder()
+      .withPhase("Failed")
+      .addNewContainerStatus()
+        .withName(DEFAULT_DRIVER_CONTAINER_NAME)
+        .withNewState()
+          .withNewTerminated()
+            .withExitCode(errorCode)
+          .endTerminated()
+        .endState()
+      .endContainerStatus()
+      .build()
+
+    val completedState = new PodStatusBuilder()
+      .withPhase("Succeeded")
+      .addNewContainerStatus()
+        .withName(DEFAULT_DRIVER_CONTAINER_NAME)
+          .withNewState()
+            .withNewTerminated()
+              .withExitCode(completedCode)
+            .endTerminated()
+          .endState()
+        .endContainerStatus()
+      .build()
+
+    val runningState = new PodStatusBuilder()
+      .withPhase("Running")
+      .build()
+
+    val errorPod = new PodBuilder(pod).withStatus(errorState).build()
+    watcher.eventReceived(Action.MODIFIED, errorPod)
+
+    assert(watcher.getDriverExitCode().nonEmpty)
+    assert(watcher.getDriverExitCode().get == errorCode)
+
+    val completedPod = new PodBuilder(pod).withStatus(completedState).build()
+    watcher.eventReceived(Action.MODIFIED, completedPod)
+
+    assert(watcher.getDriverExitCode().nonEmpty)
+    assert(watcher.getDriverExitCode().get == completedCode)
+
+    val runningPod = new PodBuilder(pod).withStatus(runningState).build()
+    watcher.eventReceived(Action.MODIFIED, runningPod)
+
+    // non-completed case
+    assertThrows[SparkException] {
+      watcher.getDriverExitCode()
+    }

Review Comment:
   We need to check the exception message because the function throws two different `SparkException`s.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zwangsheng commented on pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #40118:
URL: https://github.com/apache/spark/pull/40118#issuecomment-1487916674

   > Overall I like it, but I would want to see a unit test (does not need to go all the way to exit but something to show that the code would match).
   
   Thanks for your review.
   
   Added unit test for `LoggingPodStatusWatcher` get driver exit code after receive event.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zwangsheng commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151451312


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -95,6 +100,27 @@ private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
     this.notifyAll()
   }
 
+  override def getDriverExitCode(): Option[Int] = synchronized {
+    if (hasCompleted()) {
+      val driverContainerName = conf.get(KUBERNETES_DRIVER_PODTEMPLATE_CONTAINER_NAME)
+        .getOrElse(DEFAULT_DRIVER_CONTAINER_NAME)
+      pod.foreach { p =>
+        try {
+          return Some(p.getStatus.getContainerStatuses.asScala
+            .filter(driverContainerName == _.getName)
+            .head.getState.getTerminated.getExitCode)
+        } catch {
+          case _: NullPointerException =>

Review Comment:
   We can normalize this exception and throw it because this is not the expected behavior (Driver container completed without exitCode)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151374762


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -95,6 +100,27 @@ private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
     this.notifyAll()
   }
 
+  override def getDriverExitCode(): Option[Int] = synchronized {
+    if (hasCompleted()) {
+      val driverContainerName = conf.get(KUBERNETES_DRIVER_PODTEMPLATE_CONTAINER_NAME)
+        .getOrElse(DEFAULT_DRIVER_CONTAINER_NAME)
+      pod.foreach { p =>
+        try {
+          return Some(p.getStatus.getContainerStatuses.asScala
+            .filter(driverContainerName == _.getName)
+            .head.getState.getTerminated.getExitCode)
+        } catch {
+          case _: NullPointerException =>

Review Comment:
   Sorry but this could hide other important underlying exception. Do you think we can use more safer way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zwangsheng commented on pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #40118:
URL: https://github.com/apache/spark/pull/40118#issuecomment-1610487468

   > Just following back up, @zwangsheng do you have cycles?
   
   Thanks for watching this, sry to be late. I added unit test for `LoggingPodStatusWatcher` and fixed comments, would you @holdenk and @dongjoon-hyun take a review?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151379391


##########
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcherSuite.scala:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.submit
+
+import io.fabric8.kubernetes.api.model.{PodBuilder, PodStatusBuilder}
+import io.fabric8.kubernetes.client.Watcher.Action
+
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.deploy.k8s.Constants.DEFAULT_DRIVER_CONTAINER_NAME
+import org.apache.spark.deploy.k8s.KubernetesTestConf
+
+class LoggingPodStatusWatcherSuite extends SparkFunSuite {
+
+  private val pod = new PodBuilder()
+    .withNewMetadata()
+      .withName("test-pod")
+      .withNamespace("spark")
+      .withUid("test-spark-uid")
+      .withCreationTimestamp("2020-01-01T00:00:00Z")
+    .endMetadata()
+    .withNewSpec()
+      .withServiceAccount("spark")
+      .withNodeName("test-node")
+    .endSpec()
+    .build()
+
+
+  test("[SPARK-26365] Get watched driver container exit code") {
+    import org.apache.spark.SparkException
+    val sparkConf = new SparkConf()
+    val kubernetesConf = KubernetesTestConf.createDriverConf(sparkConf)
+    val watcher = new LoggingPodStatusWatcherImpl(kubernetesConf)
+
+    val errorCode = 1234
+    val completedCode = 0
+
+    val errorState = new PodStatusBuilder()
+      .withPhase("Failed")
+      .addNewContainerStatus()
+        .withName(DEFAULT_DRIVER_CONTAINER_NAME)
+        .withNewState()
+          .withNewTerminated()
+            .withExitCode(errorCode)
+          .endTerminated()
+        .endState()
+      .endContainerStatus()
+      .build()
+
+    val completedState = new PodStatusBuilder()
+      .withPhase("Succeeded")
+      .addNewContainerStatus()
+        .withName(DEFAULT_DRIVER_CONTAINER_NAME)
+          .withNewState()
+            .withNewTerminated()
+              .withExitCode(completedCode)
+            .endTerminated()
+          .endState()
+        .endContainerStatus()
+      .build()
+
+    val runningState = new PodStatusBuilder()
+      .withPhase("Running")
+      .build()
+
+    val errorPod = new PodBuilder(pod).withStatus(errorState).build()
+    watcher.eventReceived(Action.MODIFIED, errorPod)
+
+    assert(watcher.getDriverExitCode().nonEmpty)
+    assert(watcher.getDriverExitCode().get == errorCode)
+
+    val completedPod = new PodBuilder(pod).withStatus(completedState).build()
+    watcher.eventReceived(Action.MODIFIED, completedPod)
+
+    assert(watcher.getDriverExitCode().nonEmpty)
+    assert(watcher.getDriverExitCode().get == completedCode)
+
+    val runningPod = new PodBuilder(pod).withStatus(runningState).build()
+    watcher.eventReceived(Action.MODIFIED, runningPod)
+
+    // non-completed case

Review Comment:
   Please split this negative case into a new test case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] holdenk commented on pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #40118:
URL: https://github.com/apache/spark/pull/40118#issuecomment-1609769678

   Just following back up, @zwangsheng do you have cycles?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zwangsheng commented on pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #40118:
URL: https://github.com/apache/spark/pull/40118#issuecomment-1477559688

   Hello @dongjoon-hyun @holdenk , please help review this PR, given that there is a lot of user feedback on JIRA about this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zwangsheng commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151449690


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -95,6 +100,27 @@ private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
     this.notifyAll()
   }
 
+  override def getDriverExitCode(): Option[Int] = synchronized {
+    if (hasCompleted()) {
+      val driverContainerName = conf.get(KUBERNETES_DRIVER_PODTEMPLATE_CONTAINER_NAME)
+        .getOrElse(DEFAULT_DRIVER_CONTAINER_NAME)
+      pod.foreach { p =>
+        try {
+          return Some(p.getStatus.getContainerStatuses.asScala
+            .filter(driverContainerName == _.getName)
+            .head.getState.getTerminated.getExitCode)
+        } catch {
+          case _: NullPointerException =>
+            logError("Fail to find completed driver container exit code")
+            return None
+        }
+      }
+      throw new SparkException("Fail to get driver exit code, when the application completed")
+    } else {
+      throw new SparkException("Call getDriverExitCode() when the application has not completed")

Review Comment:
   Thank you for looking at the code in detail.
   
   Four scenarios are considered here:
   * Case 1: which normal won't be triggered under normal case, placed here to prevent the function from being misused.
   * Case 2: Successful case
   * Case 3:  Only trigger if the app has completed but pod without exit code
   * Case 4: due to we didn't get the specified driver container(Neither `KUBERNETES_DRIVER_PODTEMPLATE_CONTAINER_NAME` nor `DEFAULT_DRIVER_CONTAINER_NAME `)
   > PS. In `KubernetesUtils.loadPodFromTemplate` function logic, we read the first container from user's template file, not hardwired to `KUBERNETES_DRIVER_PODTEMPLATE_CONTAINER_NAME` or `DEFAULT_DRIVER_CONTAINER_NAME`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zwangsheng closed pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng closed pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code
URL: https://github.com/apache/spark/pull/40118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151378818


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -69,7 +74,7 @@ private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
 
   override def onClose(e: WatcherException): Unit = {
     logDebug(s"Stopping watching application $appId with last-observed phase $phase")
-    if(e != null && e.isHttpGone) {
+    if (e != null && e.isHttpGone) {

Review Comment:
   If you don't mind, let's not mix a style fix in this PR.



##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -106,7 +132,7 @@ private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
       }
     }
 
-    if(podCompleted) {
+    if (podCompleted) {

Review Comment:
   ditto. If you don't mind, let's not mix a style fix in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151378382


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -95,6 +100,27 @@ private[k8s] class LoggingPodStatusWatcherImpl(conf: KubernetesDriverConf)
     this.notifyAll()
   }
 
+  override def getDriverExitCode(): Option[Int] = synchronized {
+    if (hasCompleted()) {
+      val driverContainerName = conf.get(KUBERNETES_DRIVER_PODTEMPLATE_CONTAINER_NAME)
+        .getOrElse(DEFAULT_DRIVER_CONTAINER_NAME)
+      pod.foreach { p =>
+        try {
+          return Some(p.getStatus.getContainerStatuses.asScala
+            .filter(driverContainerName == _.getName)
+            .head.getState.getTerminated.getExitCode)
+        } catch {
+          case _: NullPointerException =>
+            logError("Fail to find completed driver container exit code")
+            return None
+        }
+      }
+      throw new SparkException("Fail to get driver exit code, when the application completed")
+    } else {
+      throw new SparkException("Call getDriverExitCode() when the application has not completed")

Review Comment:
   It seems that this method is designed to return four cases, but why `NullPointerException` is converted `None` instead of case (1)?
   - Case 1: SparkException: In case of `hasCompleted==false`
   - Case 2: Some(getExitCode): Successful case
   - Case 3: None: In case of `hasCompleted==true`, but `NullPointerException`
   - Case 4: SparkException: `Fail to get driver exit code, when the application completed`????



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151381916


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala:
##########
@@ -202,9 +203,13 @@ private[spark] class Client(
           }
         }
       }
-    } else {
-      logInfo(s"Deployed Spark application ${conf.appName} with application ID ${conf.appId} " +
-        s"and submission ID $sId into Kubernetes")
+      // no-test case exit with the driver's exit code
+      if (!sys.props.get(IS_TESTING.key).getOrElse("false").toBoolean) {
+        watcher.getDriverExitCode().foreach { exitCode =>

Review Comment:
   This PR seems to propose to introduce a new `SparkException` point to this class. In general, it's not good, as you know.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] holdenk commented on pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #40118:
URL: https://github.com/apache/spark/pull/40118#issuecomment-1487539939

   Overall I like it, but I would want to see a unit test (does not need to go all the way to exit but something to show that the code would match).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151375355


##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala:
##########
@@ -16,18 +16,23 @@
  */
 package org.apache.spark.deploy.k8s.submit
 
+import scala.collection.JavaConverters._
+
 import io.fabric8.kubernetes.api.model.Pod
 import io.fabric8.kubernetes.client.{Watcher, WatcherException}
 import io.fabric8.kubernetes.client.Watcher.Action
 
+import org.apache.spark.SparkException
 import org.apache.spark.deploy.k8s.Config._
+import org.apache.spark.deploy.k8s.Constants.DEFAULT_DRIVER_CONTAINER_NAME
 import org.apache.spark.deploy.k8s.KubernetesDriverConf
 import org.apache.spark.deploy.k8s.KubernetesUtils._
 import org.apache.spark.internal.Logging
 
 private[k8s] trait LoggingPodStatusWatcher extends Watcher[Pod] {
   def watchOrStop(submissionId: String): Boolean
   def reset(): Unit
+  def getDriverExitCode(): Option[Int]

Review Comment:
   Could you add some function documents about `SparkException`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40118:
URL: https://github.com/apache/spark/pull/40118#discussion_r1151379128


##########
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcherSuite.scala:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.submit
+
+import io.fabric8.kubernetes.api.model.{PodBuilder, PodStatusBuilder}
+import io.fabric8.kubernetes.client.Watcher.Action
+
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.deploy.k8s.Constants.DEFAULT_DRIVER_CONTAINER_NAME
+import org.apache.spark.deploy.k8s.KubernetesTestConf
+
+class LoggingPodStatusWatcherSuite extends SparkFunSuite {
+
+  private val pod = new PodBuilder()
+    .withNewMetadata()
+      .withName("test-pod")
+      .withNamespace("spark")
+      .withUid("test-spark-uid")
+      .withCreationTimestamp("2020-01-01T00:00:00Z")
+    .endMetadata()
+    .withNewSpec()
+      .withServiceAccount("spark")
+      .withNodeName("test-node")
+    .endSpec()
+    .build()
+
+
+  test("[SPARK-26365] Get watched driver container exit code") {

Review Comment:
   ```
   - test("[SPARK-26365] Get watched driver container exit code") {
   + test("SPARK-26365: Get watched driver container exit code") {
   ```



##########
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcherSuite.scala:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.submit
+
+import io.fabric8.kubernetes.api.model.{PodBuilder, PodStatusBuilder}
+import io.fabric8.kubernetes.client.Watcher.Action
+
+import org.apache.spark.SparkConf
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.deploy.k8s.Constants.DEFAULT_DRIVER_CONTAINER_NAME
+import org.apache.spark.deploy.k8s.KubernetesTestConf
+
+class LoggingPodStatusWatcherSuite extends SparkFunSuite {
+
+  private val pod = new PodBuilder()
+    .withNewMetadata()
+      .withName("test-pod")
+      .withNamespace("spark")
+      .withUid("test-spark-uid")
+      .withCreationTimestamp("2020-01-01T00:00:00Z")
+    .endMetadata()
+    .withNewSpec()
+      .withServiceAccount("spark")
+      .withNodeName("test-node")
+    .endSpec()
+    .build()
+
+
+  test("[SPARK-26365] Get watched driver container exit code") {

Review Comment:
   ```scala
   - test("[SPARK-26365] Get watched driver container exit code") {
   + test("SPARK-26365: Get watched driver container exit code") {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org