You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2026/01/26 07:20:01 UTC
[I] [Bug] Jvm OOM caused by RSS Writer OverlappingCompression [uniffle]
wForget opened a new issue, #2714:
URL: https://github.com/apache/uniffle/issues/2714
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Even when I disabled rss client compression, I encountered an OOM error due to rss writer overlapping compression.
spark conf:
```
spark.rss.client.io.compression.codec=NONE
```
error:
```
26/01/26 14:30:06 ERROR Executor: Exception in task 13.0 in stage 18.0 (TID 3197): Error during calling Java code from native code: java.lang.OutOfMemoryError: Java heap space
at org.apache.spark.shuffle.writer.WriterBuffer.getData(WriterBuffer.java:78)
at org.apache.spark.shuffle.writer.WriteBufferManager.createDeferredCompressedBlock(WriteBufferManager.java:428)
at org.apache.spark.shuffle.writer.WriteBufferManager.createShuffleBlock(WriteBufferManager.java:478)
at org.apache.spark.shuffle.writer.WriteBufferManager.insertIntoBuffer(WriteBufferManager.java:320)
at org.apache.spark.shuffle.writer.WriteBufferManager.addPartitionData(WriteBufferManager.java:250)
at org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter.doAddByte(VeloxUniffleColumnarShuffleWriter.java:311)
at org.apache.spark.shuffle.writer.PartitionPusher.pushPartitionData(PartitionPusher.scala:25)
at org.apache.gluten.vectorized.ShuffleWriterJniWrapper.write(Native Method)
at org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter.writeImpl(VeloxUniffleColumnarShuffleWriter.java:211)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:344)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.executor.Executor$TaskRunner$$Lambda$791/1101567564.apply(Unknown Source)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
### Affects Version(s)
0.10
### Uniffle Server Log Output
```logtalk
```
### Uniffle Engine Log Output
```logtalk
```
### Uniffle Server Configurations
```yaml
```
### Uniffle Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] Jvm OOM caused by RSS Writer OverlappingCompression [uniffle]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #2714:
URL: https://github.com/apache/uniffle/issues/2714#issuecomment-3798197791
cc @zuston
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Bug] Jvm OOM caused by RSS Writer OverlappingCompression [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston closed issue #2714: [Bug] Jvm OOM caused by RSS Writer OverlappingCompression
URL: https://github.com/apache/uniffle/issues/2714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org