[GitHub] [spark] nchammas edited a comment on pull request #31742: [SPARK-33000] Cleanup checkpoint data on shutdown

[GitBox <gi...@apache.org>]

nchammas edited a comment on pull request #31742:
URL: https://github.com/apache/spark/pull/31742#issuecomment-795924048


   In my testing I'm seeing that disk-persisted data (i.e. `.persist(DISK_ONLY)`) is cleaned up on shutdown, so that is somehow not affected by the shutdown interruption to `keepCleaning()` that we discussed. Something else is ensuring that bit of cleanup on shutdown.
   
   The last people to work on this cleanup logic, from what I can tell, were @witgo and @andrewor14 (e.g. #855). But that was ~6 years ago, and I don't know if they are still contributing to the project.
   
   I think it's fine to limit the scope of this fix to checkpoint data (since that's what the ticket is about), but if we can reproduce other problems with the shutdown cleanup then I suppose we could fix them in this PR as well.
   
   Let me see if I can get a committer from the dev list to chime in here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org