[GitHub] [spark] vanzin commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation

[GitBox <gi...@apache.org>]

vanzin commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#issuecomment-492330721
 
 
   If it's not a lot of work, a config option to enable the new behavior would be good. To be frank I only think it affects standalone, since it's the only backend that has the file cleaner hooked up. So if you are working on a solution to the file cleaning problem, then even that could be avoided (we can live with the issue temporarily).
   
   The cleaning problem, though, is a little trickier, because you'll have to involve the shuffle service in the cleaning process of shuffles and RDDs. That's probably a good thing in any case, and is something that is being added to the API being worked on in SPARK-25299.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org