[GitHub] [iceberg] kbendick commented on a change in pull request #1185: Flink: Add the iceberg files committer to collect data files and commit to iceberg table.

[GitBox <gi...@apache.org>]

kbendick commented on a change in pull request #1185:
URL: https://github.com/apache/iceberg/pull/1185#discussion_r460361491



##########
File path: build.gradle
##########
@@ -253,6 +255,9 @@ project(':iceberg-flink') {
     compileOnly("org.apache.hadoop:hadoop-minicluster") {
       exclude group: 'org.apache.avro', module: 'avro'
     }
+    compileOnly("de.javakaffee:kryo-serializers:0.45") {
+      exclude group: 'com.esotericsoftware', module: 'kryo'
+    }

Review comment:
       As discussed in https://github.com/apache/iceberg/pull/1145/files#diff-22e0fba47887f8c06bc56a654fd3ee9fL363, we were considering avoiding the usage of `de.javakaffee` for additional kryo serializers.
   
   Could we create custom Flink serializers, or possibly use POJOs or Flink's built in support for Avro serializers. There's a recently posted blog post on the discussion of serializer choices in Flink here: https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
   
   I think Avro makes the most sense given that it's been native to Flink for a long time and because Avro is so widely used in Iceberg already. 
   
   As an added bonus, Avro supports full schema evolution of state (such that changes to the Avro schema of types stored in stateful operators don't cause said operators to lose their state on upgrade to the new schema, provided they meet certain specifications which are relatively normal rules for type migrations, especially with Avro).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org