[GitHub] [iceberg] liubo1022126 edited a comment on issue #2219: RewriteDataFiles & DataStream streaming read error when row level delete by equal

[GitBox <gi...@apache.org>]

liubo1022126 edited a comment on issue #2219:
URL: https://github.com/apache/iceberg/issues/2219#issuecomment-775921861


   > I think it may be a flink serialization problem. Variables in flink operators (such as map and flatmap etc) need to be serializabled, and some variables do not meet the serialization requirements of flink.
   > 
   > https://ci.apache.org/projects/flink/flink-docs-stable/dev/types_serialization.html
   
   thanks zhangjun0x01
   
   yes, it seems that there are something wrong with kryo serialization, it use flink default serialization(kryo), I think https://github.com/EsotericSoftware/kryo/issues/693 is useful but not sure
   
   for test, if i declare the var with transient, it is ok, but I am not sure if var is required for the next operation in flink DAG
   
   **transient var like**
   
     private transient Map<Integer, Long> columnSizes = null;
     private transient Map<Integer, Long> valueCounts = null;
     private transient Map<Integer, Long> nullValueCounts = null;
     private transient Map<Integer, Long> nanValueCounts = null;
     private transient Map<Integer, ByteBuffer> lowerBounds = null;
     private transient Map<Integer, ByteBuffer> upperBounds = null;
   
   in https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/BaseFile.java
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org