You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/07/25 11:15:55 UTC

[GitHub] [iceberg] Reo-LEI commented on pull request #2862: Flink: Enrich flink actions

Reo-LEI commented on pull request #2862:
URL: https://github.com/apache/iceberg/pull/2862#issuecomment-886186358


   Now, the `expireSnapshots` coding is finish, but I'm blocking by a Kyro Serialization Error when I testing.
   
   Kyro raise NEP when serialize `hadoopConf` of `HadoopFileIO` when flink create `CombinedScanTask` source. the full stack as follow:
   ```
   Exception in thread "main" java.lang.RuntimeException: Serializing the source elements failed: java.lang.NullPointerException
   Serialization trace:
   hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO)
   io (org.apache.iceberg.AllManifestsTable$ManifestListReadTask)
   tasks (org.apache.iceberg.BaseCombinedScanTask)
   	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:1046)
   	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:1021)
   	at org.apache.iceberg.flink.source.MetadataTableSource$Builder.build(MetadataTableSource.java:92)
   	at org.apache.iceberg.flink.actions.BaseFlinkAction.loadMetadataTable(BaseFlinkAction.java:117)
   	at org.apache.iceberg.flink.actions.BaseFlinkAction.buildValidDataFileTable(BaseFlinkAction.java:92)
   	at org.apache.iceberg.flink.actions.ExpireSnapshotsAction.buildValidFileTable(ExpireSnapshotsAction.java:122)
   	at org.apache.iceberg.flink.actions.ExpireSnapshotsAction.expire(ExpireSnapshotsAction.java:139)
   	at org.apache.iceberg.flink.actions.ExpireSnapshotsAction.execute(ExpireSnapshotsAction.java:117)
   	at org.apache.iceberg.flink.actions.ExpireSnapshotsAction.execute(ExpireSnapshotsAction.java:44)
   	at cn.walden.stream.IcebergFlinkActions.main(IcebergFlinkActions.java:28)
   Caused by: java.io.IOException: Serializing the source elements failed: java.lang.NullPointerException
   Serialization trace:
   hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO)
   io (org.apache.iceberg.AllManifestsTable$ManifestListReadTask)
   tasks (org.apache.iceberg.BaseCombinedScanTask)
   	at org.apache.flink.streaming.api.functions.source.FromElementsFunction.<init>(FromElementsFunction.java:93)
   	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:1044)
   	... 9 more
   Caused by: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
   Serialization trace:
   hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO)
   io (org.apache.iceberg.AllManifestsTable$ManifestListReadTask)
   tasks (org.apache.iceberg.BaseCombinedScanTask)
   	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82)
   	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
   	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
   	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
   	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
   	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
   	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
   	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
   	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
   	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
   	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
   	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
   	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:316)
   	at org.apache.flink.streaming.api.functions.source.FromElementsFunction.<init>(FromElementsFunction.java:89)
   	... 10 more
   Caused by: java.lang.NullPointerException
   	at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:80)
   	at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
   	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:57)
   	... 23 more
   ```
   
   I'm trying to figure out the reason of that, but I have been blocking for this for a while and I don't have a clue. Could someone give me a little help? @rdblue @aokolnychyi @openinx 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org