[GitHub] [spark] mridulm commented on pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

[GitBox <gi...@apache.org>]

mridulm commented on PR #35906:
URL: https://github.com/apache/spark/pull/35906#issuecomment-1143029732

   @zhouyejoe As part of looking at the comment [above](https://github.com/apache/spark/pull/35906#discussion_r836949853).
   An additional query - unless there is a cleanup request to ESS (right now, application termination - but in future, as part of SPARK-38005), we need to preserve the finalization marker for a successful application + appAttempt + shuffleId + shuffleMergeId - to prevent future pushes from overwriting data.
   We need to additionally handle this in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org