[GitHub] vanzin commented on issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Release executors on unpersisting RDD

[GitBox <gi...@apache.org>]

vanzin commented on issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Release executors on unpersisting RDD
URL: https://github.com/apache/spark/pull/22015#issuecomment-468437353
 
 
   So, I just happen to be playing with this part of the code, and have a different solution for this. It's part of a much bigger change, so if you want to implement the specific solution for the caching problem here, that would be fine.
   
   The main change in this part is that I'm getting rid to the calls to `blockManagerMaster.hasCachedBlocks`, which are sort of expensive. It's possible to track this information without that call; the listener just needs to process `onBlockUpdated` and `onUnpersistRDD`.
   
   With that info it's also possible to handle `onUnpersistRDD` without the extra changes to the event, since the listener can itself keep track of which executors hold blocks for each RDD.
   
   I think that's a better approach.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org