[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

[GitBox <gi...@apache.org>]

otterc commented on code in PR #35906:
URL: https://github.com/apache/spark/pull/35906#discussion_r888251830


##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -342,6 +389,29 @@ void closeAndDeletePartitionFilesIfNeeded(
     if (cleanupLocalDirs) {
       deleteExecutorDirs(appShuffleInfo);
     }
+    cleanUpAppShuffleInfoInDB(appShuffleInfo);

Review Comment:
   Could you please clarify what's the proposal?
   
   > Do we want to delete the app attempt paths immediately, and do the shuffle deletes async (along with path deletes like here) ?
   
   Are we saying to delete app attempt paths metadata that is save in db should be cleaned up immediately and the merged shuffle metadata that is saved in db would be deleted asynchronously? 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org