You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "daijy (via GitHub)" <gi...@apache.org> on 2025/02/11 18:57:08 UTC

Re: [I] [Bug] spark executor task error when reading shuffle data when using java open jdk11 [incubator-uniffle]

daijy commented on issue #2082:
URL: https://github.com/apache/incubator-uniffle/issues/2082#issuecomment-2651779511

   I also hit the same issue. In my case, readBuffer is an instance of DirectByteBuffer and readBuffer#cleaner is null and readBuffer#attachment is not null. Since readBuffer#cleaner is null, PlatformDependent.freeDirectBuffer which eventually invokes cleaner won't do anything regardless, plus before reaching cleaner, it's already [error out when checking attachment](https://github.com/openjdk/jdk/blob/jdk-11-ga/src/jdk.unsupported/share/classes/sun/misc/Unsafe.java#L1238). The readBuffer we get is from [DirectByteBuffer.duplicate](https://github.com/ydq/OpenJDK-11/blob/master/src/java/nio/DirectByteBuffer.java#L168), which sets cleaner to null intentionally. It seems readBuffer is just a pointer to the base buffer, which would [cleanup anyway](https://github.com/apache/incubator-uniffle/blob/master/common/src/main/java/org/apache/uniffle/common/netty/TransportFrameDecoder.java#L202). It seems we can simply don't release readBuffer in ShuffleReadClientImpl. I tested my application wi
 th that and seems run successfully and didn't see memory issue as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org