You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/03/31 03:43:06 UTC
[GitHub] [iceberg] openinx edited a comment on pull request #2343: Core: Fix the kryo serialization issue in BaseFile.
openinx edited a comment on pull request #2343:
URL: https://github.com/apache/iceberg/pull/2343#issuecomment-810735680
@aokolnychyi I tried to run the unit tests by switching to use `DirectByteBuffer`, both spark/flink `TestDataFileSerialization` tests works fine (See following patch). So did I miss some thing in you last [comment](https://github.com/apache/iceberg/pull/2343#issuecomment-810685833) ? I guess kryo will use the Java serializer to serialize & deserialize the `SerializableByteBufferMap`, but I need to confirm this by checking the code.
```diff
diff --git a/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java b/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java
index ad67b181..4b43718d 100644
--- a/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java
+++ b/flink/src/test/java/org/apache/iceberg/flink/TestDataFileSerialization.java
@@ -196,6 +196,6 @@ public class TestDataFileSerialization {
}
private static ByteBuffer longToBuffer(long value) {
- return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value);
+ return ByteBuffer.allocateDirect(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value);
}
}
diff --git a/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java b/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java
index f5c2990b..eb22873d 100644
--- a/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java
+++ b/spark/src/test/java/org/apache/iceberg/TestDataFileSerialization.java
@@ -164,6 +164,6 @@ public class TestDataFileSerialization {
}
private static ByteBuffer longToBuffer(long value) {
- return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value);
+ return ByteBuffer.allocateDirect(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, value);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org