[GitHub] [spark] mridulm commented on pull request #37922: [WIP][SPARK-40480][SHUFFLE] Remove push-based shuffle data after query finished

[GitBox <gi...@apache.org>]

mridulm commented on PR #37922:
URL: https://github.com/apache/spark/pull/37922#issuecomment-1251349810

   > The push-based shuffle service will auto clean up the old shuffle merge data
   
   Consider the case I mentioned above - stage retry for an `INDETERMINATE` stage.
   We cleanup previous attempts only if it happens to use the same merger in new attempt as well.
   
   The previous attempt's mergers are not reused for the next attempt - and so the previous mergers will continue to hold stale data without cleaning them up - until application terminates.
   Note - any merger which happens to be reused in new stage attempt will cleanup - I am referring to those which are not used: old attempt `mergerLocs` `--` new attempts `mergerLocs`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org