[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing

[GitBox <gi...@apache.org>]

Kimahriman commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333068984

   > 1. Have you run the fix with your production workload for a while and see there is no longer the same memory issue?
   Yes I confirmed that today. I have executors with a 25 GiB heap that I was seeing get to 90+ GiB of resident memory within a single one of our batches. After trying out the fix it held steady at ~25-30 GiB resident memory.
   
   > 2. Did [b8b1fbc](https://github.com/apache/spark/commit/b8b1fbc21c66348d25be3404d3f61099f2a7a9b5) help to figure out the "sort of leaking" memory, or did it report simply the resized memory size and didn't disclose the memory issue?
   Didn't really help, because I could only really see the stats write after the commit when the write batch was supposed to be full. Wasn't really anyway to see in those stats that it wasn't being cleaned up. And yeah after clearing it just reports the resized memory not the actually used memory.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org