[GitHub] [spark] attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation

[GitBox <gi...@apache.org>]

attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#issuecomment-493643283
 
 
   The new message `RemoveBlocks` used only when the fetch RDD via shuffle service feature is enabled and done asynchronously: called within a `Future` block.
   
   There is one change I am thinking about right now where I could been overeager. Before my change the old `RemoveRDD` was sent to every executor:
   https://github.com/apache/spark/blob/dfeeda24c0f5d60bf6d2e1868c5290a1f62dc558/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L171-L178
   
   Now I am only sending `RemoveRDD` to those executors where its blocks are registered. And I am wondering whether we miss to delete those blocks which not reported to the master. I can easily change that small logic back to be on the safe side.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org