You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2025/10/23 14:18:16 UTC
[I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
wForget opened a new issue, #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932
### Backend
VL (Velox)
### Bug description
#9336 removed the logic of passing the memory change of DynamicOffHeapSizingMemoryTarget to TaskMemoryManager, which will cause spark consumers to not be able to trigger spills from the native side.
> Avoid chaining DynamicOffHeapSizingMemoryTarget with Spark consumer since it basically never triggers spills because of its design. The target will manage OOMs by itself.
I can't understand this description.
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3440217870
@zhztheplayer, I noticed that @wForget mentioned the TaskMemoryManager is not passed to DynamicOffHeapSizingMemoryTarget. I was wondering if this could potentially be related to the issue?
https://github.com/apache/incubator-gluten/commit/083ecfbbe9cbb8d2dffd86c177ac5e1debdac973#diff-b6234f870afb82ba142a4f4e3e358ddb30dc4d5f00b3b9f5b4e9afddc9b4a761R55
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3442435675
> We’ve only tested PR https://github.com/apache/incubator-gluten/pull/9585 extensively internally, but we are still using Gluten 1.3 internally, and RP https://github.com/apache/incubator-gluten/pull/9336 has never been included. Spilling has not been an issue in our internal environment. I didn’t realize this issue until @wForget pointed it out.
Now it gets clearer. That's also why I also raised a concern at https://github.com/apache/incubator-gluten/pull/9585#issuecomment-2909353000. Only for a solution like #9585, passing the memoryTarget to `DynamicOffHeapSizingMemoryTarget` for spilling could potentially make sense.
But again, as a non-blocker comment, perhaps more comprehensive consideration is needed to make the feature more robust, like how to trigger spill when on-heap memory already exceeds 40%, how to avoid process RSS overrun if there're constant on-heap memory allocations that DynamicOffHeapSizingMemoryTarget isn't aware, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3441940021
> I was wondering if this could potentially be related to the issue?
https://github.com/apache/incubator-gluten/commit/083ecfbbe9cbb8d2dffd86c177ac5e1debdac973#diff-b6234f870afb82ba142a4f4e3e358ddb30dc4d5f00b3b9f5b4e9afddc9b4a761R55
@zhli1142015 I was trying to understand why this issue is related to https://github.com/apache/incubator-gluten/pull/9336. As I mentioned, https://github.com/apache/incubator-gluten/pull/9336 was for optimizing the first version of this feature, which was implemented by https://github.com/apache/incubator-gluten/pull/5439.
According to that version of the feature, `DynamicOffHeapSizingMemoryTarget` was using exactly the same capacity as Spark's off-heap memory.
https://github.com/apache/incubator-gluten/blob/55b2e92f54f95f980587d08f0beb430e0a922b15/gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java#L33
So in #9336, we could safely do the simplification as spill was never triggered. I was once thinking about removing the whole feature because it was no longer maintained: https://github.com/apache/incubator-gluten/discussions/8600, #9336 was kind of, at the same time, preparing for that.
@zhli1142015 Would you confirm whether we are on the same page about the above? Thanks.
And after these actions, the good news was we got the feature reworked by https://github.com/apache/incubator-gluten/pull/9585. Design was changed a lot so I think we can do anything we think is right to make sure spill works for the feature. Maybe by introducing the delegated target back, I am not sure.
> By default, spark.memory.fraction is 0.6, meaning that only about 60% of the executor memory (after subtracting 300 MB for overhead) is available for task computation.
I would recommend taking more comprehensive factors into consideration when relying on such mechanism for spilling, because we cannot make sure on-heap memory doesn't exceed 40%. Once it exceeds, how can we trigger spill prior to OOM, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3442268927
@zhztheplayer my understanding is that when DynamicOffHeapSizingMemoryTarget is enabled, the following change causes the memory allocated by Velox not to be reported to the Spark Task Memory Manager, so spilling is not triggered — because DynamicOffHeapSizingMemoryTarget does not call the memoryTarget below. (Please correct me if I'm wrong.)
<img width="2470" height="274" alt="Image" src="https://github.com/user-attachments/assets/2eb943e7-7936-4f69-9443-aaa900f6c8b5" />
Even if the memory used by Velox is counted as on-heap memory, we still need to report the memory allocations to the Spark Task Memory Manager.
Why this is not discovered when doing PR #9585 ?
We’ve only tested PR #9585 extensively internally, but we are still using Gluten 1.3 internally, and RP #9336 has never been included. Spilling has not been an issue in our internal environment. I didn’t realize this issue until @wForget pointed it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #10932: [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled
URL: https://github.com/apache/incubator-gluten/issues/10932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3440418390
From this log, the offheap memory usage has exceeded the max memory of TaskMemoryManager `(3 * 1024 MB -300 MB) * 0.6 = 1663.2 MB`, so it should be enough to trigger spill.
```
25/10/24 09:35:25 INFO DynamicOffHeapSizingMemoryTarget: Updated VM flags: MaxHeapFreeRatio from 30 to 5.
25/10/24 09:35:25 WARN DynamicOffHeapSizingMemoryTarget: Starting full gc to shrink JVM memory: Total On-heap: 519110656, Free On-heap: 337648400, Total Off-heap: 2524971008, Used On-Heap: 181462256, Executor memory: 3113877504.
25/10/24 09:35:26 WARN DynamicOffHeapSizingMemoryTarget: Finished full gc to shrink JVM memory: Total On-heap: 519110656, Free On-heap: 414046152, Total Off-heap: 2524971008, Used On-Heap: 105064504, Executor memory: 3113877504, [GC Retry times: 3].
25/10/24 09:35:26 INFO DynamicOffHeapSizingMemoryTarget: Reverted VM flags back.
25/10/24 09:35:26 WARN DynamicOffHeapSizingMemoryTarget: Failing allocation as unified memory is OOM. Used Off-heap: 2524971008, Used On-Heap: 107549216, Free On-heap: 411561440, Total On-heap: 519110656, Max On-heap: 3113877504, Allocation: 486539264.
25/10/24 09:35:26 INFO TaskMemoryManager: Memory used in task 9695
25/10/24 09:35:26 INFO TaskMemoryManager: 0 bytes of memory were used by task 9695 but are not associated with specific consumers
25/10/24 09:35:26 INFO TaskMemoryManager: 0 bytes of memory are used for execution and 15127767 bytes of memory are used for storage
25/10/24 09:35:26 WARN ManagedReservationListener: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 464.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=1663.2 MiB
spark.gluten.memory.task.offHeap.size.in.bytes=1663.2 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=831.6 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
Dynamic off-heap sizing memory target stats:
DynamicOffHeapSizing.96: Current used bytes: 2.3 GiB, peak bytes: 2.3 GiB
```
spark conf:
```
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
spark.executor.memory=3g
spark.executor.extraJavaOptions=-XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=30 -Xms512m -XX:+UseSerialGC -Dfile.encoding=utf-8 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<LOG_DIR>/gc.log -XX:ErrorFile=<LOG_DIR>/hs_err_pid%p.log
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3437772147
The dynamic off-heap sizing code, before and after #9336, both don't trigger spill.
Before #9336, DynamicOffHeapSizingMemoryTarget essentially caps the available off-heap size within a number which is lower than the Spark off-heap size because of the co-accounted on-heap memory in this target. Because this number is lower than the Spark off-heap size, and when the number is reached, OOM will be reported. So from Spark memory manager's view, it's used off-heap memory never reaches the total off-heap limit. That's why spill was never triggered.
#9336 just simplified the code, so this spill shortage can also exposed to code reader clearly.
I had a [similar question](https://github.com/apache/incubator-gluten/pull/9585#issuecomment-2909353000) about the spill design in @zhli1142015's later rework of this feature, and he may have more insights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 commented on issue #10932:
URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3440105177
The formula defines the amount of memory available to each task — once this threshold is exceeded, the task memory manager triggers a spill.
```
Per-task memory = ((spark.executor.memory - 300MB) * spark.memory.fraction)
/ active tasks count
```
By default, spark.memory.fraction is 0.6, meaning that only about 60% of the executor memory (after subtracting 300 MB for overhead) is available for task computation.
As a result, the memory usage threshold that triggers a spill is much lower than the actual amount of memory required to cause an OOM(137).
The DynamicOffHeapSizingMemoryTarget does not change this logic. Its role is to let the JVM release free memory back to the operating system when Velox needs more memory. Since Spark does not account for released memory, this behavior is transparent to the TaskMemoryManager.
Under this architecture, Velox can theoretically utilize the entire memory allocated to the task, not just the configured off-heap portion.
@wForget In our case, we observed that the spill behavior was not affected by this feature.
Note that the JVM currently does not provide an API for ondemand memory shrinking — shrink only occurs as a side effect of a full GC, and its behavior may vary across different JVM versions.
Could you share your runtime environment, such as the JVM version, GC type, and memory configuration? Also, in what scenario did you observe that a spill should have occurred but instead resulted in an OOM?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org