You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Yevhen Cherkes (Jira)" <ji...@apache.org> on 2021/01/19 14:56:00 UTC
[jira] [Updated] (AVRO-3027) .net consumer Backward compatibility
issue.
[ https://issues.apache.org/jira/browse/AVRO-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yevhen Cherkes updated AVRO-3027:
---------------------------------
Description:
We are facing the following exception during the consuming messages:
{code}
No matching schema for {"type":"record","name":"InnerType1","namespace":"com.mycorp.mynamespace","fields":[
{"name":"value","type":"string"}
]} in {"type":["null",{"type":"record","name":"InnerType2","namespace":"com.mycorp.mynamespace","fields":[
{"name":"value","type":"string"}
]}]}
at Avro.Generic.DefaultReader.findBranch(UnionSchema us, Schema s)
at Avro.Generic.DefaultReader.ReadUnion(Object reuse, UnionSchema writerSchema, Schema readerSchema, Decoder d)
at Avro.Generic.DefaultReader.Read(Object reuse, Schema writerSchema, Schema readerSchema, Decoder d)
at Avro.Specific.SpecificDefaultReader.ReadRecord(Object reuse, RecordSchema writerSchema, Schema readerSchema, Decoder dec)
{code}
We have 2 versions of the schema on a topic:
Schema v1:
{code:json}
{
"fields": [
{
"name": "InnerComplexValue",
"type": [
"null",
{
"fields": [
{
"name": "value",
"type": "string"
}
],
"name": "InnerType1",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
]
}
],
"name": "RootType",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
{code}
Schema v2:
{code:json}
{
"fields": [
{
"name": "InnerComplexValue",
"type": [
"null",
{
"fields": [
{
"name": "value",
"type": "string"
}
],
"name": "InnerType2",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
]
}
],
"name": "RootType",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
{code}
InnerType1 -> InnerType2 is the only change.
The schema for a topic is configured with Backward compatibility.
An updated version of a Schema saved with no errors.
Then we generated a specific record RootType with avrogen tool.
Reproduction code:
{code:c#}
var base64 = "AhR0ZXN0IHZhbHVl";
using var stream = new MemoryStream(Convert.FromBase64String(base64));
var schema_v1 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
"[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType1\"" +
",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
var schema_v2 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
"[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType2\"" +
",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
var specificRecord = Deserialize<RootType>(stream, schema_v1, schema_v2);
{code}
Method Deserialize was taken from the [SpecificTests.cs#L408-L417|https://github.com/apache/avro/blob/master/lang/csharp/src/apache/test/Specific/SpecificTests.cs#L408-L417]
*Java*-based consumers work with no issues for this particular case.
Probably, the reason is in a different name resolution code:
java: https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L647-L659
.net: https://github.com/apache/avro/blob/master/lang/csharp/src/apache/main/Schema/RecordSchema.cs#L310-L312
was:
We are facing the following exception during the consuming messages:
{code}
No matching schema for {"type":"record","name":"InnerType1","namespace":"com.mycorp.mynamespace","fields":[
{"name":"value","type":"string"}
]} in {"type":["null",{"type":"record","name":"InnerType2","namespace":"com.mycorp.mynamespace","fields":[
{"name":"value","type":"string"}
]}]}
at Avro.Generic.DefaultReader.findBranch(UnionSchema us, Schema s)
at Avro.Generic.DefaultReader.ReadUnion(Object reuse, UnionSchema writerSchema, Schema readerSchema, Decoder d)
at Avro.Generic.DefaultReader.Read(Object reuse, Schema writerSchema, Schema readerSchema, Decoder d)
at Avro.Specific.SpecificDefaultReader.ReadRecord(Object reuse, RecordSchema writerSchema, Schema readerSchema, Decoder dec)
{code}
We have 2 versions of the schema on a topic:
Schema v1:
{code:json}
{
"fields": [
{
"name": "InnerComplexValue",
"type": [
"null",
{
"fields": [
{
"name": "value",
"type": "string"
}
],
"name": "InnerType1",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
]
}
],
"name": "RootType",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
{code}
Schema v2:
{code:json}
{
"fields": [
{
"name": "InnerComplexValue",
"type": [
"null",
{
"fields": [
{
"name": "value",
"type": "string"
}
],
"name": "InnerType2",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
]
}
],
"name": "RootType",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
{code}
InnerType1 -> InnerType2 is the only change.
The schema for a topic is configured with Backward compatibility.
An updated version of a Schema saved with no errors.
Then we generated a specific record RootType with avrogen tool.
Reproduction code:
{code:c#}
var base64 = "AhR0ZXN0IHZhbHVl";
using var stream = new MemoryStream(Convert.FromBase64String(base64));
var schema_v1 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
"[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType1\"" +
",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
var schema_v2 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
"[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType2\"" +
",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
var specificRecord = Deserialize<RootType>(stream, schema_v1, schema_v2);
{code}
Method Deserialize was taken from the [SpecificTests.cs#L408-L417|https://github.com/apache/avro/blob/master/lang/csharp/src/apache/test/Specific/SpecificTests.cs#L408-L417]
*Java*-based consumers work with no issues for this particular case.
> .net consumer Backward compatibility issue.
> --------------------------------------------
>
> Key: AVRO-3027
> URL: https://issues.apache.org/jira/browse/AVRO-3027
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
> Affects Versions: 1.10.1
> Environment: .Net core 3.1
> Reporter: Yevhen Cherkes
> Priority: Major
> Attachments: CompatibilityTests.zip
>
>
> We are facing the following exception during the consuming messages:
> {code}
> No matching schema for {"type":"record","name":"InnerType1","namespace":"com.mycorp.mynamespace","fields":[
> {"name":"value","type":"string"}
> ]} in {"type":["null",{"type":"record","name":"InnerType2","namespace":"com.mycorp.mynamespace","fields":[
> {"name":"value","type":"string"}
> ]}]}
> at Avro.Generic.DefaultReader.findBranch(UnionSchema us, Schema s)
> at Avro.Generic.DefaultReader.ReadUnion(Object reuse, UnionSchema writerSchema, Schema readerSchema, Decoder d)
> at Avro.Generic.DefaultReader.Read(Object reuse, Schema writerSchema, Schema readerSchema, Decoder d)
> at Avro.Specific.SpecificDefaultReader.ReadRecord(Object reuse, RecordSchema writerSchema, Schema readerSchema, Decoder dec)
> {code}
> We have 2 versions of the schema on a topic:
> Schema v1:
> {code:json}
> {
> "fields": [
> {
> "name": "InnerComplexValue",
> "type": [
> "null",
> {
> "fields": [
> {
> "name": "value",
> "type": "string"
> }
> ],
> "name": "InnerType1",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> ]
> }
> ],
> "name": "RootType",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> {code}
> Schema v2:
> {code:json}
> {
> "fields": [
> {
> "name": "InnerComplexValue",
> "type": [
> "null",
> {
> "fields": [
> {
> "name": "value",
> "type": "string"
> }
> ],
> "name": "InnerType2",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> ]
> }
> ],
> "name": "RootType",
> "namespace": "com.mycorp.mynamespace",
> "type": "record"
> }
> {code}
> InnerType1 -> InnerType2 is the only change.
> The schema for a topic is configured with Backward compatibility.
> An updated version of a Schema saved with no errors.
> Then we generated a specific record RootType with avrogen tool.
> Reproduction code:
>
> {code:c#}
> var base64 = "AhR0ZXN0IHZhbHVl";
> using var stream = new MemoryStream(Convert.FromBase64String(base64));
> var schema_v1 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
> "[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType1\"" +
> ",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
> var schema_v2 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
> "[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType2\"" +
> ",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");
> var specificRecord = Deserialize<RootType>(stream, schema_v1, schema_v2);
> {code}
>
> Method Deserialize was taken from the [SpecificTests.cs#L408-L417|https://github.com/apache/avro/blob/master/lang/csharp/src/apache/test/Specific/SpecificTests.cs#L408-L417]
> *Java*-based consumers work with no issues for this particular case.
> Probably, the reason is in a different name resolution code:
> java: https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L647-L659
> .net: https://github.com/apache/avro/blob/master/lang/csharp/src/apache/main/Schema/RecordSchema.cs#L310-L312
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)