[GitHub] [spark] attilapiros edited a comment on pull request #31742: [SPARK-33000] Cleanup checkpoint data on shutdown

[GitBox <gi...@apache.org>]

attilapiros edited a comment on pull request #31742:
URL: https://github.com/apache/spark/pull/31742#issuecomment-796003983


   > When Spark is shut down, all references go out of scope. These changes bring Spark's behavior in line with the intended (and documented) behavior.
   
   I think you are wrong here: checkpointing should outlive the unexpected shutdowns. So there is a very important difference between the reference goes out of scope during a normal execution (in this case cleanup is expected depending on the config you mentioned) and when a references goes out of scope because of an unexpected error (in this case you should keep the checkpoint data).
   
   So even after an unexpected exit the next run of the same app could pick up the checkpointed data.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org