[GitHub] [spark] otterc commented on a change in pull request #33034: WIP: [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle

[GitBox <gi...@apache.org>]

otterc commented on a change in pull request #33034:
URL: https://github.com/apache/spark/pull/33034#discussion_r673411883



##########
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##########
@@ -1450,11 +1450,11 @@ private[spark] object MapOutputTracker extends Logging {
           val remainingMapStatuses = if (mergeStatus != null && mergeStatus.totalSize > 0) {
             // If MergeStatus is available for the given partition, add location of the
             // pre-merged shuffle partition for this partition ID. Here we create a
-            // ShuffleBlockId with mapId being SHUFFLE_PUSH_MAP_ID to indicate this is
+            // ShufflePushBlockId with mapId being SHUFFLE_PUSH_MAP_ID to indicate this is
             // a merged shuffle block.
             splitsByAddress.getOrElseUpdate(mergeStatus.location, ListBuffer()) +=
-              ((ShuffleBlockId(shuffleId, SHUFFLE_PUSH_MAP_ID, partId), mergeStatus.totalSize,
-                SHUFFLE_PUSH_MAP_ID))
+              ((ShufflePushBlockId(shuffleId, mergeStatus.shuffleSequenceId,

Review comment:
       I think instead of using `ShufflePushBlockId` on the fetch side to represent a merged block, we should probably create a new block id that identifies a merged blocked which doesn't have mapId. Maybe  `ShuffleMergedBlockId(shuffleId, shuffleSequenceId, reduceId)`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org