[I] Two dynamic off-heap sizing bugs: MinHeapFreeRatio not restored, availability metric reads the wrong pools [gluten]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #13093:
URL: https://github.com/apache/gluten/issues/13093

   Both bugs are in the dynamic off-heap sizing path, gated by `spark.gluten.memory.dynamic.offHeap.sizing.enabled` (experimental, off by default), so they only affect deployments that opt into that flag.
   
   **1. `MinHeapFreeRatio` is never restored after a heap shrink.** `DynamicOffHeapSizingMemoryTarget.shrinkOnHeapMemory` lowers both `MinHeapFreeRatio` and `MaxHeapFreeRatio` so the JVM releases committed heap, but the `finally` block restores only `MaxHeapFreeRatio`. After the first shrink the executor keeps `MinHeapFreeRatio=5` for the rest of its life. The heap then stays tighter and the JVM GCs more often, well after the off-heap pressure that triggered the shrink has passed.
   
   **2. The available off-heap metric reads the wrong pools under dynamic sizing.** `SparkMemoryUtil.getCurrentAvailableOffHeapMemory` sums the off-heap storage and execution pools. Under dynamic sizing, Gluten charges its global reservations to the on-heap pools (`GlobalOffHeapMemoryTarget` acquires with `ON_HEAP`) and the driver plugin forces `spark.memory.offHeap.size=0`, so the off-heap pools are empty and the method returns 0. The Velox columnar shuffle writer uses that value as its `memLimit`, so it drops to the minimum buffer size.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org