You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wenwj0 (via GitHub)" <gi...@apache.org> on 2024/10/18 12:45:48 UTC
[I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
wenwj0 opened a new issue, #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605
### Backend
VL (Velox)
### Bug description
I am trying to set `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true"`, but OOM exception occurs.
spark configuration :
```
spark.executor.memory=4g;
spark.executor.memoryOverhead=1G;
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
spark.memory.offHeap.enabled=true
```
and web ui shows:
```
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=597059174
spark.gluten.memory.offHeap.size.in.bytes=2388236697
spark.gluten.memory.task.offHeap.size.in.bytes=597059174
spark.memory.offHeap.size=2388236697
```
And I got the OOM exception:
```
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=3.4 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=876.6 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=876.6 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
Memory consumer stats:
Task.52: Current used bytes: 104.0 MiB, peak bytes: N/A
\- Gluten.Tree.0: Current used bytes: 104.0 MiB, peak bytes: 112.0 MiB
\- root.0: Current used bytes: 104.0 MiB, peak bytes: 112.0 MiB
+- CelebornShuffleWriter.0: Current used bytes: 48.0 MiB, peak bytes: 48.0 MiB
| \- single: Current used bytes: 48.0 MiB, peak bytes: 48.0 MiB
| +- gluten::MemoryAllocator: Current used bytes: 28.8 MiB, peak bytes: 29.0 MiB
| \- root: Current used bytes: 4.2 MiB, peak bytes: 15.0 MiB
| \- default_leaf: Current used bytes: 4.2 MiB, peak bytes: 14.1 MiB
```
It may cause by:
```
24/10/18 17:16:00 WARN org.apache.gluten.memory.memtarget.DynamicOffHeapSizingMemoryTarget: "Failing allocation as unified memory is OOM. Used Off-heap: 406847480, Used On-Heap: 2021017784, Free On-heap: 1796847432, Total On-heap: 3817865216, Max On-heap: 2388236697, Allocation: 8388608."
24/10/18 17:16:00 INFO org.apache.spark.memory.TaskMemoryManager: "Memory used in task 11"
24/10/18 17:16:00 INFO org.apache.spark.memory.TaskMemoryManager: "Acquired by org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer@182a8cbe: 104.0 MiB"
24/10/18 17:16:00 INFO org.apache.spark.memory.TaskMemoryManager: "0 bytes of memory were used by task 11 but are not associated with specific consumers"
24/10/18 17:16:00 INFO org.apache.spark.memory.TaskMemoryManager: "406847480 bytes of memory are used for execution and 1129714 bytes of memory are used for storage"
```
**As we can see above, the used off-heap memory is only 406847480(388MiB), while my off-heap configuration is 2.2GiB.**
Why will throw OOM exception?
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "acvictor (via GitHub)" <gi...@apache.org>.
acvictor commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2425646783
When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true` it will not consider the configured off-heap size https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "wenwj0 (via GitHub)" <gi...@apache.org>.
wenwj0 commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2426101699
> Why are you using the option spark.gluten.memory.dynamic.offHeap.sizing.enabled? @wenwj0
In our scenario, we have various configurations for different executor memory. I try to use this property because it seems that I can use the existing configurations without the need to set extra off-heap memory.
> I believe your case is because on-heap occupies the remaining 2.2 GiB - 388 MiB, which could be considered by-design of the feature. But it would be great if you can share more background information about your use case. From my impression only a few users are using the option.
I agree with you.
MAX_MEMORY_IN_BYTES refers to off-heap memory, hence
(size + usedOffHeapBytesNow + usedOnHeapBytes > MAX_MEMORY_IN_BYTES) =>
(8MiB + 388MiB + 1927MiB = 2323MiB > MAX_MEMORY_IN_BYTES(2GiB)
If the MAX_MEMORY_IN_BYTES is onheap memory or onheap+offheap memory, maybe it works. @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2425701744
Why are you using the option `spark.gluten.memory.dynamic.offHeap.sizing.enabled`? @wenwj0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2425657440
> When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true` it will not consider the configured off-heap size https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L167
Perhaps there was a tiny typo leading to a bug when PR #5439 was iterated? cc @supermem613 Would you like to help confirm? Thanks.
https://github.com/apache/incubator-gluten/pull/5439/commits/8c7cfa59bf9f8c015e16c225d3d14c7801a977dd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2425689065
> > > When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true` it will not consider the configured off-heap size https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L167
> >
> >
> > Perhaps there was a tiny typo leading to a bug when PR #5439 was iterated? cc @supermem613 Would you like to help confirm? Thanks.
> > A commit that looks like to be related in PR #5439 [8c7cfa5](https://github.com/apache/incubator-gluten/commit/8c7cfa59bf9f8c015e16c225d3d14c7801a977dd)
>
> @zhztheplayer what typo do you mean?
Found this line
https://github.com/apache/incubator-gluten/commit/8c7cfa59bf9f8c015e16c225d3d14c7801a977dd#diff-b6234f870afb82ba142a4f4e3e358ddb30dc4d5f00b3b9f5b4e9afddc9b4a761R31
`GlutenConfig.getConf().onHeapMemorySize()` was changed to `GlutenConfig.getConf().offHeapMemorySize()`, which doesn't look like intentional to me, but I am not sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "wenwj0 (via GitHub)" <gi...@apache.org>.
wenwj0 commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2422410857
https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java#L51
I have one question here:
```
if (size + usedOffHeapBytesNow + usedOnHeapBytes > MAX_MEMORY_IN_BYTES) {
...
}
```
Why we need to use the `usedOnHeapBytes` to compare with `MAX_MEMORY_IN_BYTES(offHeapMemorySize)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "acvictor (via GitHub)" <gi...@apache.org>.
acvictor commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2425684538
> > When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true` it will not consider the configured off-heap size https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L167
>
> Perhaps there was a tiny typo leading to a bug when PR #5439 was iterated? cc @supermem613 Would you like to help confirm? Thanks.
>
> A commit that looks like to be related in PR #5439 [8c7cfa5](https://github.com/apache/incubator-gluten/commit/8c7cfa59bf9f8c015e16c225d3d14c7801a977dd)
@zhztheplayer what typo do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2425721316
> `GlutenConfig.getConf().onHeapMemorySize()` was changed to `GlutenConfig.getConf().offHeapMemorySize()`, which doesn't look like intentional to me, but I am not sure.
After reading more code I guess this was intentional... The code tended to set an off-heap size then use the size to cover off-heap + on-heap. The PR used on-heap initially at the time when I was reviewing.
@wenwj0
> As we can see above, the used off-heap memory is only 406847480(388MiB), while my off-heap configuration is 2.2GiB.
I believe your case is because on-heap occupies the remaining 2.2 GiB - 388 MiB, which could be considered by-design of the feature. But it would be great if you can share more background information about your use case. From my impression only a few users are using the option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "wenwj0 (via GitHub)" <gi...@apache.org>.
wenwj0 commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2422413695
PTAL @supermem613 @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2428321458
> onheap+offheap memory
It sounds OK to me to change to on-heap + off-heap, which may simplify the relevant configuration setting code by the way.
cc @supermem613 @zhli1142015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Failing allocation in DynamicOffHeapSizingMemoryTarget [incubator-gluten]
Posted by "supermem613 (via GitHub)" <gi...@apache.org>.
supermem613 commented on issue #7605:
URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2441575272
Sorry, I was offline for the last 10 days. @zhztheplayer is correct, this feature's goal is so that one can configure the configure the executor memory (on-heap + off-heap) and allow the use to fluctuate between the two as needed (e.g. in the case of fallback).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org