[GitHub] SongYadong opened a new pull request #23585: [SPARK-26662][CORE] Dispose off-heap memory when MemoryStore is cleared

[GitBox <gi...@apache.org>]

SongYadong opened a new pull request #23585: [SPARK-26662][CORE] Dispose off-heap memory when MemoryStore is cleared
URL: https://github.com/apache/spark/pull/23585
 
 
   ## What changes were proposed in this pull request?
   
   If buffer in `MemoryStore`'s `SerializedMemoryEntry` is allocated off-heap, current `clear()` action doesn't dispose them. These off-heap memory, though, will be freed when process ends, but for application that need to stop and restart `SparkContext` serveral times, this may cause some kind of off-heap memory leak. 
   This PR try to dispose off-heap memory by itself when clearing `MemoryStore`.
   
   ## How was this patch tested?
   
   Tested manually by running code written myself, while observing system resources monitor to verify off-heap memory allocation and free behavior.
   I have tried to add a unit test, but it's hard to find a way counting off-heap memory allocated by Unsafe API. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org