[GitHub] [spark] mridulm commented on pull request #33896: [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle

[GitBox <gi...@apache.org>]

mridulm commented on pull request #33896:
URL: https://github.com/apache/spark/pull/33896#issuecomment-964755478


   Agree, we need to handle this
   
   On Tue, Nov 9, 2021 at 9:00 PM wuyi ***@***.***> wrote:
   
   > ***@***.**** commented on this pull request.
   > ------------------------------
   >
   > In core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
   > <https://github.com/apache/spark/pull/33896#discussion_r746212040>:
   >
   > > @@ -327,10 +338,32 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
   >          s"stop.")
   >        return false
   >      } else {
   > +      if (remainingBlocks.isEmpty && pushRequests.isEmpty && deferredPushRequests.isEmpty) {
   >
   > @venkata91 <https://github.com/venkata91> Let me take an example:
   > Assuming we have request 1 and request 2 in total and each has 10 blocks.
   > Then, we start request 1 and request 2 together. When we reach here, the
   > emptyremainingBlocks is only for request 1. And we'd see pushRequests and
   > deferredPushRequests are both empty. However, request 2 is still ongoing
   > and its remaining blocks could be not empty.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/33896#discussion_r746212040>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAMETFBC2TD4KIYPIWPKWPLULHN4DANCNFSM5DIGN5HQ>
   > .
   > Triage notifications on the go with GitHub Mobile for iOS
   > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
   > or Android
   > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
   >
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org