You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2024/08/15 06:20:13 UTC
[I] [VL] shuffle writter spill not triggered [incubator-gluten]
FelixYBW opened a new issue, #6863:
URL: https://github.com/apache/incubator-gluten/issues/6863
### Backend
VL (Velox)
### Bug description
should be sort based shuffle. @marin-ma
```
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=50.0 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=12.5 GiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=6.3 GiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.102118: Current used bytes: 13.5 GiB, peak bytes: N/A
\- Gluten.Tree.158: Current used bytes: 13.5 GiB, peak bytes: 17.7 GiB
\- root.158: Current used bytes: 13.5 GiB, peak bytes: 17.7 GiB
+- ShuffleWriter.158: Current used bytes: 13.5 GiB, peak bytes: 13.5 GiB
| \- single: Current used bytes: 13.5 GiB, peak bytes: 13.5 GiB
| +- root: Current used bytes: 13.5 GiB, peak bytes: 13.5 GiB
| | \- default_leaf: Current used bytes: 13.5 GiB, peak bytes: 13.5 GiB
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- WholeStageIterator.158: Current used bytes: 22.0 MiB, peak bytes: 152.0 MiB
| \- single: Current used bytes: 22.0 MiB, peak bytes: 152.0 MiB
| +- root: Current used bytes: 16.2 MiB, peak bytes: 152.0 MiB
| | +- task.Gluten_Stage_26_TID_102118_VTID_216: Current used bytes: 16.2 MiB, peak bytes: 152.0 MiB
| | | +- node.2: Current used bytes: 16.1 MiB, peak bytes: 152.0 MiB
| | | | \- op.2.1.0.Aggregation: Current used bytes: 16.1 MiB, peak bytes: 128.6 MiB
| | | +- node.5: Current used bytes: 78.0 KiB, peak bytes: 28.0 MiB
| | | | +- op.5.0.0.HashProbe: Current used bytes: 78.0 KiB, peak bytes: 94.0 KiB
| | | | \- op.5.1.0.HashBuild: Current used bytes: 0.0 B, peak bytes: 132.0 KiB
| | | +- node.4: Current used bytes: 6.3 KiB, peak bytes: 1024.0 KiB
| | | | \- op.4.1.0.FilterProject: Current used bytes: 6.3 KiB, peak bytes: 6.5 KiB
| | | +- node.7: Current used bytes: 0.0 B, peak bytes: 1024.0 KiB
| | | | \- op.7.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 12.0 KiB
| | | +- node.1: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.1.1.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | +- node.6: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | | \- op.6.0.0.FilterProject: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- node.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | | \- op.0.0.0.ValueStream: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ArrowContextInstance.203: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
+- IndicatorVectorBase#init.158: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- single: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- root: Current used bytes: 0.0 B, peak bytes: 0.0 B
| | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ShuffleReader.80.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 4.8 MiB
+- WholeStageIterator.158.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 45.6 MiB
+- ArrowContextInstance.204: Current used bytes: 0.0 B, peak bytes: 0.0 B
+- ShuffleReader.80: Current used bytes: 0.0 B, peak bytes: 16.0 MiB
| \- single: Current used bytes: 0.0 B, peak bytes: 16.0 MiB
| +- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 1648.9 KiB
| \- root: Current used bytes: 0.0 B, peak bytes: 1024.0 KiB
| \- default_leaf: Current used bytes: 0.0 B, peak bytes: 240.0 KiB
+- ShuffleWriter.158.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 4.1 GiB
\- IndicatorVectorBase#init.158.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 0.0 B
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] shuffle writter spill not triggered [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #6863:
URL: https://github.com/apache/incubator-gluten/issues/6863#issuecomment-2290864191
Shuffle spill is triggered but it fails to allocate extra memory during spill
```
at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
at org.apache.gluten.vectorized.ShuffleWriterJniWrapper.nativeEvict(Native Method)
at org.apache.spark.shuffle.ColumnarShuffleWriter$$anon$1.spill(ColumnarShuffleWriter.scala:173)
```
The memory allocation likely stems from compression. It can be fixed via pre-allocating the compression buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] shuffle writter spill not triggered [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #6863: [VL] shuffle writter spill not triggered
URL: https://github.com/apache/incubator-gluten/issues/6863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] shuffle writter spill not triggered [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #6863:
URL: https://github.com/apache/incubator-gluten/issues/6863#issuecomment-2290751274
@FelixYBW Could you provide the full stack trace of the exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] shuffle writter spill not triggered [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6863:
URL: https://github.com/apache/incubator-gluten/issues/6863#issuecomment-2358917584
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] shuffle writter spill not triggered [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6863:
URL: https://github.com/apache/incubator-gluten/issues/6863#issuecomment-2290798514
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org