[GitHub] [spark] cloud-fan commented on a change in pull request #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files

[GitBox <gi...@apache.org>]

cloud-fan commented on a change in pull request #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
URL: https://github.com/apache/spark/pull/24110#discussion_r272032917
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
 ##########
 @@ -150,6 +150,11 @@ private[spark] class DAGScheduler(
    */
   private[scheduler] val shuffleIdToMapStage = new HashMap[Int, ShuffleMapStage]
   private[scheduler] val jobIdToActiveJob = new HashMap[Int, ActiveJob]
+  /**
+   * Mapping from shuffle id to the indeterminate stage attempt id. Only includes the retried
+   * stages whose output deterministic level is indeterminate.
+   */
+  private[scheduler] val shuffleIdToIndeterminateStageAttemptId = new HashMap[Int, Int]
 
 Review comment:
   why do we need this map? I think we just use the stage attempt id, right?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org