[GitHub] [spark] mridulm commented on a change in pull request #34735: [SPARK-37481][Core][WebUI] Fix disappearance of skipped stages after they retry

[GitBox <gi...@apache.org>]

mridulm commented on a change in pull request #34735:
URL: https://github.com/apache/spark/pull/34735#discussion_r766030145



##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -718,6 +718,12 @@ private[spark] class DAGScheduler(
                 // read from merged output as the MergeStatuses are not available.
                 if (!mapStage.isAvailable || !mapStage.shuffleDep.shuffleMergeFinalized) {
                   missing += mapStage
+                } else if (mapStage.nextAttemptId == 0) {
+                  // Forward the nextAttemptId if skipped and get visited for the first time.
+                  // Otherwise, once it gets retried,
+                  // 1) the stuffs in stage info become distorting, e.g. task num, input byte, e.t.c
+                  // 2) the first attempt starts from 0-idx, it will not be marked as a retry
+                  mapStage.nextAttemptId += 1

Review comment:
       Agree with Tom, this is specific to the skipped -> running transition; not applicable for retry which, as you @Ngone51 mentioned, will increment via `makeNewStageAttempt`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org