You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yun Tang (JIRA)" <ji...@apache.org> on 2019/01/29 16:38:00 UTC
[jira] [Commented] (FLINK-11333) First-class support for Protobuf
types with evolvable schema
[ https://issues.apache.org/jira/browse/FLINK-11333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755185#comment-16755185 ]
Yun Tang commented on FLINK-11333:
----------------------------------
I planed to use avro-protobuf to extract Protobuf message's schema to verify whether schema evolvable in avro's view. However, from [the comment|https://github.com/confluentinc/schema-registry/pull/672#issuecomment-393348920] described, avro defines different rules for compatibility. For example, Long cannot be treated as compatibly with Integer for avro, while protobuf could. And I write a simple program to verify this, there really exists difference.
I'm afraid the idea to use avro to judge whether two protobuf message compatible is not correct. As far as I could see, check protobuf message compatibility seems a non-trivial work, we might have to leave schema evolution check of protobuf during job runtime.
> First-class support for Protobuf types with evolvable schema
> ------------------------------------------------------------
>
> Key: FLINK-11333
> URL: https://issues.apache.org/jira/browse/FLINK-11333
> Project: Flink
> Issue Type: Sub-task
> Components: Type Serialization System
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Yun Tang
> Priority: Major
>
> I think we have more and more users who are thinking about using Protobuf for their state types.
> Right now, Protobuf isn't supported directly in Flink. The only way to use Protobuf for a type is to register it via Kryo: https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/custom_serializers.html.
> Likewise for Avro types, we should be able to natively support Protobuf, having a {{ProtobufSerializer}} that handles serialization of Protobuf types. The serializer should also write necessary information in its snapshot, to enable schema evolution for it in the future. For Protobuf, this should almost work out-of-the-box.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)