You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "r7raul1984 (via GitHub)" <gi...@apache.org> on 2026/09/17 10:03:29 UTC
[PR] [GLUTEN-13044][CORE] Fix exceedsMaxMemoryUsage to use actual used heap instead of committed heap [gluten]
r7raul1984 opened a new pull request, #13045:
URL: https://github.com/apache/gluten/pull/13045
Fixes #13044
DynamicOffHeapSizingMemoryTarget.exceedsMaxMemoryUsage() previously used Runtime.getRuntime().totalMemory() (committed heap) as the on-heap memory pressure indicator. Committed heap is the memory the JVM has reserved from the OS, which includes a potentially large
amount of free pages that have not yet been returned. As a result, the check overstates memory pressure between Spark stages and triggers spurious OOMs.
Example of the failure:
- Stage 0 runs with 2 concurrent tasks (~2 GB peak each); JVM commits ~8.9 GB of a 9 GB heap
- Stage 0 finishes; actual heap usage drops to ~2 GB, but totalMemory() remains ~8.9 GB because the JVM has not yet returned the committed pages to the OS
- Stage 1 starts and requests 8 MB: 8 MB + 0 + 8.9 GB >= 9 GB → OOM triggered
- Actual free heap was ~6.5 GB (freeMemory()); the allocation should have succeeded
Fix:
Replace totalMemory() with totalMemory() - freeMemory() (actual used heap) at all four call sites of exceedsMaxMemoryUsage() — three in borrow() and one in shouldTriggerAsyncOnHeapMemoryShrink(). The parameter is renamed from totalOnHeapMemory to usedOnHeapMemory for
clarity.
// Before
exceedsMaxMemoryUsage(totalHeapMemory, usedOffHeapMemory, size, 1.0)
// After
exceedsMaxMemoryUsage(totalHeapMemory - freeHeapMemory, usedOffHeapMemory, size, 1.0)
Note: freeHeapMemory is already read from Runtime.getRuntime().freeMemory() at the top of borrow(), so no additional JVM calls are introduced.
How was this patch tested?
Existing unit tests in DynamicOffHeapSizingMemoryTargetTest (added in #12425) continue to pass. New tests covering the spurious-OOM scenario between stages will be added as a follow-up.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6, Anthropic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13044][CORE] Fix exceedsMaxMemoryUsage to use actual used heap instead of committed heap [gluten]
Posted by "surendralilhore (via GitHub)" <gi...@apache.org>.
surendralilhore commented on code in PR #13045:
URL: https://github.com/apache/gluten/pull/13045#discussion_r4064305886
##########
gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java:
##########
@@ -148,22 +148,24 @@ public long borrow(long size) {
long freeHeapMemory = Runtime.getRuntime().freeMemory();
long usedOffHeapMemory = USED_OFF_HEAP_BYTES.get();
- // Adds the total JVM memory which is the actual memory the JVM occupied from the operating
- // system into the counter.
- if (exceedsMaxMemoryUsage(totalHeapMemory, usedOffHeapMemory, size, 1.0)) {
+ // Use actual used heap (totalMemory - freeMemory) rather than committed heap (totalMemory).
+ // Committed heap includes pages the JVM holds but hasn't filled yet; using it overstates
+ // pressure and causes spurious OOMs between stages when the JVM hasn't had time to return
+ // committed-but-free pages to the OS.
+ if (exceedsMaxMemoryUsage(totalHeapMemory - freeHeapMemory, usedOffHeapMemory, size, 1.0)) {
Review Comment:
This is breaking changes done by commit https://github.com/apache/gluten/pull/9321
The existing code counts committed heap precisely because future Java heap growth is invisible to the native allocator. If committed heap is high but mostly free, the intended behavior is to run GC and shrink the JVM commitment before granting that space to Velox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13044][CORE] Fix exceedsMaxMemoryUsage to use actual used heap instead of committed heap [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13045:
URL: https://github.com/apache/gluten/pull/13045#issuecomment-5712532282
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13044][CORE] Fix exceedsMaxMemoryUsage to use actual used heap instead of committed heap [gluten]
Posted by "scgkiran (via GitHub)" <gi...@apache.org>.
scgkiran commented on PR #13045:
URL: https://github.com/apache/gluten/pull/13045#issuecomment-5928642666
> Example of the failure:
>
> * Stage 0 runs with 2 concurrent tasks (~2 GB peak each); JVM commits ~8.9 GB of a 9 GB heap
> * Stage 0 finishes; actual heap usage drops to ~2 GB, but totalMemory() remains ~8.9 GB because the JVM has not yet returned the committed pages to the OS
> * Stage 1 starts and requests 8 MB: 8 MB + 0 + 8.9 GB >= 9 GB → OOM triggered
> * Actual free heap was ~6.5 GB (freeMemory()); the allocation should have succeeded
>
In the above example, at stage1, 8MB request should trigger a GC synchronously before OOM is triggered. Isn't it?
[`borrow`](https://github.com/apache/gluten/blob/5d35d9ec26e53bef2c90f950dd2ecc299d87e641/gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java#L187) -> shrinkOnHeapMemory --> [shrinkOnHeapMemoryInternal](https://github.com/apache/gluten/blob/5d35d9ec26e53bef2c90f950dd2ecc299d87e641/gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java#L345) --> System.gc();
After this totalMemory() should come down. What am I missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org