You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2025/06/12 08:33:11 UTC
[I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
wForget opened a new issue, #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959
### Backend
VL (Velox)
### Bug description
error log:
```
E20250612 15:38:54.817852 163104 Exceptions.h:67] Line: /data/workspace/gluten-deploy-dist/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp:651, Function:runInternal, Expression: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=2.0 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=2.0 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=1024.0 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.1011: Current used bytes: 2.0 GiB, peak bytes: N/A
+- Gluten.Tree.0: Current used bytes: 1961.0 MiB, peak bytes: 2.0 GiB
| \- root.0: Current used bytes: 1961.0 MiB, peak bytes: 2.0 GiB
| +- UniffleShuffleWriter.0: Current used bytes: 1664.0 MiB, peak bytes: 1720.0 MiB
| | \- single: Current used bytes: 1664.0 MiB, peak bytes: 1720.0 MiB
| | +- gluten::MemoryAllocator: Current used bytes: 1658.6 MiB, peak bytes: 1717.0 MiB
| | \- root: Current used bytes: 0.0 B, peak bytes: 1024.0 KiB
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 164.6 KiB
| +- WholeStageIterator.0: Current used bytes: 297.0 MiB, peak bytes: 297.0 MiB
| | \- single: Current used bytes: 289.0 MiB, peak bytes: 297.0 MiB
| | +- root: Current used bytes: 288.0 MiB, peak bytes: 296.0 MiB
| | | +- task.Gluten_Stage_3_TID_1011_VTID_0: Current used bytes: 288.0 MiB, peak bytes: 296.0 MiB
| | | | +- node.0: Current used bytes: 288.0 MiB, peak bytes: 296.0 MiB
| | | | | +- op.0.0.0.TableScan: Current used bytes: 288.0 MiB, peak bytes: 289.6 MiB
| | | | | \- op.0.0.0.TableScan.test-hive: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- node.1: Current used bytes: 0.0 B, peak bytes: 1024.0 KiB
| | | | \- op.1.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 27.0 KiB
| | | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- ArrowContextInstance.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- OverAcquire.DummyTarget.3: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
| +- OverAcquire.DummyTarget.1: Current used bytes: 0.0 B, peak bytes: 89.1 MiB
| +- OverAcquire.DummyTarget.2: Current used bytes: 0.0 B, peak bytes: 19.2 MiB
| +- OverAcquire.DummyTarget.0: Current used bytes: 0.0 B, peak bytes: 417.6 MiB
| +- IndicatorVectorBase#init.0: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| | \- single: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| | +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- VeloxBatchAppender.0: Current used bytes: 0.0 B, peak bytes: 64.0 MiB
| \- single: Current used bytes: 0.0 B, peak bytes: 64.0 MiB
| +- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- root: Current used bytes: 0.0 B, peak bytes: 64.0 MiB
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 62.0 MiB
\- org.apache.spark.shuffle.writer.WriteBufferManager@15bfb96a: Current used bytes: 31.4 MiB, peak bytes: N/A
at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:43)
at org.apache.gluten.vectorized.ShuffleWriterJniWrapper.nativeEvict(Native Method)
at org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter$1.spill(VeloxUniffleColumnarShuffleWriter.java:169)
at org.apache.gluten.memory.memtarget.Spillers$AppendableSpillerList.spill(Spillers.java:86)
at org.apache.gluten.memory.memtarget.Spillers$WithMinSpillSize.spill(Spillers.java:66)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:80)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:55)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:73)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:55)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:73)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:55)
at org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer.spill(TreeMemoryConsumer.java:115)
at org.apache.spark.memory.TaskMemoryManager.trySpillAndAcquire(TaskMemoryManager.java:227)
at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:190)
at org.apache.spark.memory.MemoryConsumer.acquireMemory(MemoryConsumer.java:137)
at org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer.borrow(TreeMemoryConsumer.java:65)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow0(TreeMemoryTargets.java:126)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow(TreeMemoryTargets.java:118)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow0(TreeMemoryTargets.java:126)
at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow(TreeMemoryTargets.java:118)
at org.apache.gluten.memory.memtarget.OverAcquire.borrow(OverAcquire.java:62)
at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:35)
at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:43)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.GeneralInIterator.hasNext(GeneralInIterator.java:31)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.utils.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter.writeImpl(VeloxUniffleColumnarShuffleWriter.java:135)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:320)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-2987161490
@wForget We have introduced streaming compression for sort-based shuffle in this patch https://github.com/apache/incubator-gluten/pull/9278
The streaming compressor allocates a fixed sized buffer which is counted as overhead memory, therefore the memory allocation won't trigger OOM. We should probably adopt this approach for the hash-based shuffle writer as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-3017350839
@wForget without #9278, we need to compress the whole spilled data once, which needs to allocate a large offheap memory. With #9278, we introduced streaming compression which has a fixed buffer size allocating in overhead memory instead of offheap memory. In this way we will compress the spilled data into several small blocks. This is the way vanilla spark uses.
Can you try and report any issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-2978684717
> The OOM error should be caused by memory allocation when trying to push data to the remote server. I will take a look.
Memory allocation may occur when memoryPayload to blockPayload
https://github.com/apache/incubator-gluten/blob/7377aec44eca57de260fca042f99a955888bb324/cpp/core/shuffle/rss/RssPartitionWriter.cc#L117-L118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget closed issue #9959: [VL][1.2] OOM occurs during shuffle writer spill
URL: https://github.com/apache/incubator-gluten/issues/9959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-2965686385
The shuffle writer seems to use so much memory that it cannot request more memory on spill, should we reduce the shuffle writer memory limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-2977914752
@marin-ma mentioned for remote shuffle, spill shouldn't be triggered. @marin-ma any insight?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-2978269082
@FelixYBW When OOM happens, RSS writer will push data to the remote rather than spill to local disks.
The OOM error should be caused by memory allocation when trying to push data to the remote server. I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-3017791612
> [@wForget](https://github.com/wForget) without [#9278](https://github.com/apache/incubator-gluten/pull/9278), we need to compress the whole spilled data once, which needs to allocate a large offheap memory. With [#9278](https://github.com/apache/incubator-gluten/pull/9278), we introduced streaming compression which has a fixed buffer size allocating in overhead memory instead of offheap memory. In this way we will compress the spilled data into several small blocks. This is the way vanilla spark uses.
>
> Can you try and report any issues?
Thanks, I will try to implement it in hash-based shuffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM occurs during shuffle writer spill [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9959:
URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-2968973089
Can we configure a max memory limit for the hash shuffle writer, and when this limit is reached, reclaim all the allocated memory instead of the minSize?
I tried the following and it worked fine:
@FelixYBW @marin-ma Could you please give some suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org