[GitHub] [spark] Ngone51 commented on a change in pull request #34934: [SPARK-37675][CORE][SHUFFLE] Return PushMergedRemoteMetaFailedFetchResult if no available push-merged block

[GitBox <gi...@apache.org>]

Ngone51 commented on a change in pull request #34934:
URL: https://github.com/apache/spark/pull/34934#discussion_r772341805



##########
File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala
##########
@@ -145,16 +145,23 @@ private class PushBasedFetchHelper(
         logDebug(s"Received the meta of push-merged block for ($shuffleId, $shuffleMergeId," +
           s" $reduceId) from ${req.address.host}:${req.address.port}")
         try {
-          iterator.addToResultsQueue(PushMergedRemoteMetaFetchResult(shuffleId, shuffleMergeId,
-            reduceId, sizeMap((shuffleId, reduceId)), meta.readChunkBitmaps(), address))
+          val bitmaps = meta.readChunkBitmaps()
+          if (bitmaps.nonEmpty) {
+            iterator.addToResultsQueue(PushMergedRemoteMetaFetchResult(shuffleId, shuffleMergeId,
+              reduceId, sizeMap((shuffleId, reduceId)), bitmaps, address))
+          } else {
+            logInfo(s"No available push-merged block for ($shuffleId, $shuffleMergeId," +

Review comment:
       Hmm...I thought we only raise the merged status request when there are merged blocks. Because we ensured this when fetching merge status:
   https://github.com/apache/spark/blob/f1532a291179665a3b69dad640a770ecfcbed629/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L1537
   
   But why bitmaps could be empty here? cc @mridulm @otterc 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org