You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Yevhen Cherkes (Jira)" <ji...@apache.org> on 2022/08/28 08:11:00 UTC
[jira] [Comment Edited] (AVRO-3027) .net consumer Backward compatibility issue.
[ https://issues.apache.org/jira/browse/AVRO-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585722#comment-17585722 ]
Yevhen Cherkes edited comment on AVRO-3027 at 8/28/22 8:10 AM:
---------------------------------------------------------------
[~kniemitalo],
The problem occurs in a heterogeneous environment. The problem is in the cross-language consistency of consumers. So, I think, the best solution is to fix the java-based schema-compatibility validator so it indicates the schema incompatibility. But if it's impossible because of backward compatibility - I think it must be included in specifications and implemented in every language.
Also, if you take a look closer at the problem - the replacement of the type of field can be decomposed as 2 actions - delete the optional field and add the optional field, [it's FULL_TRANSITIVE compatibility type due specs|https://docs.confluent.io/platform/current/schema-registry/avro.html#:~:text=Any%20order-,FULL_TRANSITIVE,-Add%20optional%20fields].
I've checked out that the built-in schema validator doesn't indicate any problems. You can try it, for example, in the [Confluent cp-all-in-one docker image|https://github.com/confluentinc/cp-all-in-one].
I just would like to give some context, on what's happened:
We have N teams and a centralized schema repository.
Team #1 works with the Java stack. It creates the Schema v1, adds this v1 to the repository, and produces messages in Kafka.
Then they are changing the Schema to v2, checking the backward compatibility with built-in java based tools. Nothing indicates incompatibility. They started producing the messages with Schema v2. All works fine for Java-based consumers.
Team #2 works with the .NET stack. It consumes the messages with the Shema1, but when the message with Shema v2 appears in Kafka, they run into runtime trouble.
Of course, Team2 can create an MR/PR to add the aliases to the existing Shema, but why?
Team #1 is the creator and owner of the Schema.
was (Author: ycherkes):
[~kniemitalo],
The problem occurs in a heterogeneous environment. The problem is in the cross-language consistency of consumers. So, I think, the best solution is to remove an undocumented match algorithm from the java-based consumer. But if it's impossible because of backward compatibility - I think it must be included in specifications and implemented in every language.
I just would like to give some context, on what's happened:
We have N teams and a centralized schema repository.
Team #1 works with the Java stack. It creates the Schema v1, adds this v1 to the repository, and produces messages in Kafka.
Then they are changing the Schema to v2, checking the backward compatibility with built-in java based tools. Nothing indicates incompatibility. They started producing the messages with Schema v2. All works fine for Java-based consumers.
Team #2 works with the .NET stack. It consumes the messages with the Shema1, but when the message with Shema v2 appears in Kafka, they run into runtime trouble.
Of course, Team2 can create an MR/PR to add the aliases to the existing Shema, but why?
Team #1 is the creator and owner of the Schema.
> .net consumer Backward compatibility issue.
> --------------------------------------------
>
> Key: AVRO-3027
> URL: https://issues.apache.org/jira/browse/AVRO-3027
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp, java
> Affects Versions: 1.10.1
> Environment: .Net core 3.1
> Reporter: Yevhen Cherkes
> Priority: Critical
> Labels: pull-request-available
> Attachments: CompatibilityTests.zip
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We are facing the following exception on the .net core consumer's side:
> (*Java*-based consumers work with no issues for this particular case.)
> {code:java}
> No matching schema for {"type":"record","name":"InnerType1","namespace":"com.mycorp.mynamespace","fields":[
> {"name":"value","type":"string"}
> ]} in {"type":["null",{"type":"record","name":"InnerType2","namespace":"com.mycorp.mynamespace","fields":[
> {"name":"value","type":"string"}
> ]}]}
> at Avro.Generic.DefaultReader.findBranch(UnionSchema us, Schema s)
> at Avro.Generic.DefaultReader.ReadUnion(Object reuse, UnionSchema writerSchema, Schema readerSchema, Decoder d)
> at Avro.Generic.DefaultReader.Read(Object reuse, Schema writerSchema, Schema readerSchema, Decoder d)
> at Avro.Specific.SpecificDefaultReader.ReadRecord(Object reuse, RecordSchema writerSchema, Schema readerSchema, Decoder dec)
> {code}
> We have 2 versions of the schema on a topic:
> Schema v1:
> {code:json}
> {
> "fields": [
> {
> "name": "InnerComplexValue",
> "type": [
> "null",
> {
> "fields": [
> {
> "name": "value",
> "type": "string"
> }
> ],
> "name": "InnerType1",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> ]
> }
> ],
> "name": "RootType",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> {code}
> Schema v2:
> {code:json}
> {
> "fields": [
> {
> "name": "InnerComplexValue",
> "type": [
> "null",
> {
> "fields": [
> {
> "name": "value",
> "type": "string"
> }
> ],
> "name": "InnerType2",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> ]
> }
> ],
> "name": "RootType",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> {code}
> InnerType1 -> InnerType2 is the only change.
> The schema for a topic is configured with *Backward* compatibility.
> An updated version of a Schema is saved with no errors.
> Then we generated a specific record RootType with avrogen tool.
> Reproduction code:
>
> {code:c#}
> var base64 = "AhR0ZXN0IHZhbHVl";
> using var stream = new MemoryStream(Convert.FromBase64String(base64));
> var schema_v1 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
> "[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType1\"" +
> ",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
> var schema_v2 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
> "[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType2\"" +
> ",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
> var specificRecord = Deserialize<RootType>(stream, schema_v1, schema_v2);
> {code}
> Full sources see in attachments.
> Method Deserialize was taken from the [SpecificTests.cs#L408-L417|https://github.com/apache/avro/blob/master/lang/csharp/src/apache/test/Specific/SpecificTests.cs#L408-L417]
>
> Probably, the reason is in a different Schema resolution code:
> java: [https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L647-L659]
> .net: [https://github.com/apache/avro/blob/master/lang/csharp/src/apache/main/Schema/RecordSchema.cs#L310-L312]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)