You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2025/05/28 08:06:28 UTC
Re: [PR] [SPARK-21113][CORE] Read ahead input stream to amortize disk IO cost … [spark]
cxzl25 commented on PR #18317:
URL: https://github.com/apache/spark/pull/18317#issuecomment-2915373677
If we generate a large number of spill files, using ReadAhead's Task in `UnsafeExternalSorter` is easier to fail and OOM. By default, each spill file requires a 2MB in-heap buffer and creates a thread.
Perhaps ReadAhead is better and more stable for `BytesToBytesMap`.
Should we introduce a threshold configuration to control whether UnsafeExternalSorter uses ReadAhead to reduce memory usage on the executor? @mridulm @cloud-fan
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter#getIterator
https://github.com/apache/spark/blob/f423885f65d41632ffe0c20afa47d22d161630da/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L768-L773
https://github.com/apache/spark/blob/f423885f65d41632ffe0c20afa47d22d161630da/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java#L286-L287
```java
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:71)
at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:391)
at org.apache.spark.io.ReadAheadInputStream.<init>(ReadAheadInputStream.java:105)
at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.<init>(UnsafeSorterSpillReader.java:74)
at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.getReader(UnsafeSorterSpillWriter.java:159)
at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getIterator(UnsafeExternalSorter.java:733)
at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray.generateIterator(ExternalAppendOnlyUnsafeRowArray.scala:173)
at org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray.generateIterator(ExternalAppendOnlyUnsafeRowArray.scala:177)
at org.apache.spark.sql.execution.joins.OneSideOuterIterator.advanceBufferUntilBoundConditionSatisfied(SortMergeJoinExec.scala:1240)
at org.apache.spark.sql.execution.joins.OneSideOuterIterator.advanceStream(SortMergeJoinExec.scala:1222)
at org.apache.spark.sql.execution.joins.OneSideOuterIterator.advanceNext(SortMergeJoinExec.scala:1254)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org