[GitHub] [spark] venkata91 commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level

[GitBox <gi...@apache.org>]

venkata91 commented on a change in pull request #33253:
URL: https://github.com/apache/spark/pull/33253#discussion_r701325301



##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##########
@@ -1208,6 +1232,33 @@ private[spark] class AppStatusListener(
     }
   }
 
+  private def killedTaskSummaryForSpeculationStageSummary(
+      reason: TaskEndReason,
+      oldSummary: Map[String, Int],
+      isSpeculative: Boolean): Map[String, Int] = {
+    reason match {
+      case k: TaskKilled if k.reason.contains("another attempt succeeded") =>
+        if (isSpeculative) {
+          oldSummary.updated("original attempt succeeded",
+            oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+        } else {
+          oldSummary.updated("speculated attempt succeeded",
+            oldSummary.getOrElse("speculated attempt succeeded", 0) + 1)
+        }
+      // If the stage is finished and speculative tasks get killed, then the
+      // kill reason is "stage finished"
+      case k: TaskKilled if k.reason.contains("Stage finished") =>
+        if (isSpeculative) {
+          oldSummary.updated("original attempt succeeded",
+            oldSummary.getOrElse("original attempt succeeded", 0) + 1)
+        } else {
+          oldSummary

Review comment:
       @sarutak Did you get a chance to think about the above issue? Any thoughts? Gentle reminder




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org