[GitHub] [spark] venkata91 edited a comment on pull request #33896: [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle

[GitBox <gi...@apache.org>]

venkata91 edited a comment on pull request #33896:
URL: https://github.com/apache/spark/pull/33896#issuecomment-923323263


   Also regarding the comment in the JIRA that not to finalize when there are no blocks pushed due to the threshold. I almost implemented that change, but then I thought anyway we are having a timeout (`spark.shuffle.push.results.timeout (default 10 secs)`) if finalize request don't get responded by then, scheduler moves forward submitting the next stages. Do you think we really get much with avoiding finalize when there are no blocks pushed? Thoughts? cc @Ngone51 @Victsm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org