[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #806: [#596] feat(netty): Use off heap memory to read HDFS data

["kaijchen (via GitHub)" <gi...@apache.org>]

kaijchen commented on code in PR #806:
URL: https://github.com/apache/incubator-uniffle/pull/806#discussion_r1161542828


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/reader/RssShuffleDataIterator.java:
##########
@@ -73,9 +73,11 @@ public RssShuffleDataIterator(
     this.codec = compress ? Codec.newInstance(rssConf) : null;
   }
 
-  public Iterator<Tuple2<Object, Object>> createKVIterator(ByteBuffer data, int size) {
+  public Iterator<Tuple2<Object, Object>> createKVIterator(ByteBuffer data) {
     clearDeserializationStream();
-    byteBufInputStream = new ByteBufInputStream(Unpooled.wrappedBuffer(data.array(), data.position(), size), true);
+    // Uncompressed data is released in this class, Compressed data is release in the class ShuffleReadClientImpl
+    // So if codec is null, we don't release the data when the stream is closed
+    byteBufInputStream = new ByteBufInputStream(Unpooled.wrappedBuffer(data), codec != null);

Review Comment:
   Is it possible to unify where the buffer is released?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org