Custom GenericRecord Serializer with Tuple?

["Hailu, Andreas" <An...@gs.com>]

Hello!

I have a custom Avro GenericRecord serializer that supports reading and writing records without having to pass along the schema with every record by using a centralized registry. I've registered it with the execution environment as:

                environment.addDefaultKryoSerializer(GenericRecord.class, CustomGenericRecordSerializer.class);

We read from a source that provides a Tuple2<Void, GenericRecord>, and then map to GenericRecord to process and then sink. If I disable Kryo generic types given I don't want GenericRecord to fall back to the plain Kryo serializer, I encounter the following exception:

Caused by: java.lang.UnsupportedOperationException: Generic types have been disabled in the ExecutionConfig and type org.apache.avro.generic.GenericRecord is treated as a generic type.
        at org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:87)
        at org.apache.flink.api.java.typeutils.TupleTypeInfo.createSerializer(TupleTypeInfo.java:104)
        at org.apache.flink.api.java.typeutils.TupleTypeInfo.createSerializer(TupleTypeInfo.java:49)
        at org.apache.flink.optimizer.postpass.JavaApiPostPass.createSerializer(JavaApiPostPass.java:310)
        at org.apache.flink.optimizer.postpass.JavaApiPostPass.traverseChannel(JavaApiPostPass.java:270)
        at org.apache.flink.optimizer.postpass.JavaApiPostPass.traverse(JavaApiPostPass.java:96)
        at org.apache.flink.optimizer.postpass.JavaApiPostPass.postPass(JavaApiPostPass.java:81)
        at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:543)
        at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:404)

It seems the GenericRecord in the Tuple2 gets interpreted as a GenericTypeInfo. I expected it should use the CustomGenericRecordSerializer that is registered. Have I misunderstood how the serializer registration works? Is there an extra step when using these types as part of a Tuple?

best,
ah


________________________________

Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>