You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by KV 59 <kv...@gmail.com> on 2023/05/05 02:47:18 UTC
enum schema evolution
Hi,
I see that that java Avro compatibility check doesn't work as per
specification for enum schema evolution. I have a the following schema
{
> "type" : "record",
> "name" : "TestEnumRec",
> "namespace" : "com.five9.avro.enum.test",
> "fields" : [ {
> "name" : "enumType",
> "type" : {
> "type" : "enum",
> "name" : "EnumType",
> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6" ]
> }
> } ]
> }
And another version of the same schema
{
> "type" : "record",
> "name" : "TestEnumRec",
> "namespace" : "com.five9.avro.enum.test",
> "fields" : [ {
> "name" : "enumType",
> "type" : {
> "type" : "enum",
> "name" : "EnumType",
> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6", "S7" ]
> }
> } ]
> }
These schemas show as compatible for mutual read
This is not in line with what the specification says
>
> -
>
> if both are enums: if the writer’s symbol is not present in the
> reader’s enum and the reader has a default value, then that value is used,
> otherwise an error is signalled.
>
>
I have tried this in Avro 1.9.1 and 1.11.1. Is this a bug? If not, what am
I doing wrong?
Appreciate reponses
Regards,
Kishore
Re: enum schema evolution
Posted by Brennan Vincent <br...@umanwizard.com>.
https://umanwizard.com/avro.rst
Read the section called "Is it always possible to tell, given two schemas,
whether they are compatible?". The example in that section uses unions, but your
example is similar. The schemas are "compatible" in the sense that it is
possible to resolve one against the other; however, _at runtime_ an error might
be signaled if the writer actually happens to write the enum value that is
missing in the reader.
This is not strictly a "bug", but I think Avro is missing some nuance in how
schema compatibility is described. Rather than just "compatible" vs. "not
compatible", it should actually distinguish "definitely not compatible",
"compatible depending on what data is observed at runtime", and "definitely
compatible".
On 2023-05-05 12:54, KV 59 wrote:
> It works both ways in compatibility Validation. I have used the Avro library for
> compatibility check
> SchemaValidator validator = builder.mutualReadStrategy().validateAll();
>
>
> The SchemaValidator says it is valid with the first schema as reader and second
> schema as writer and vice versa is valid as well.
>
> Obviously this is not the case. since symbol "S7" is not present in schema1 and
> also it has no default value defined,
>
>
>
> On Fri, May 5, 2023 at 7:37 AM Brennan Vincent <brennan@umanwizard.com
> <ma...@umanwizard.com>> wrote:
>
> Which one is the writer and which is the reader?
>
> Sent from my iPhone
>
>> On May 4, 2023, at 22:47, KV 59 <kvajjala59@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>
>> Hi,
>>
>> I see that that java Avro compatibility check doesn't work as per
>> specification for enum schema evolution. I have a the following schema
>>
>> {
>> "type" : "record",
>> "name" : "TestEnumRec",
>> "namespace" : "com.five9.avro.enum.test",
>> "fields" : [ {
>> "name" : "enumType",
>> "type" : {
>> "type" : "enum",
>> "name" : "EnumType",
>> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6" ]
>> }
>> } ]
>> }
>>
>>
>> And another version of the same schema
>>
>> {
>> "type" : "record",
>> "name" : "TestEnumRec",
>> "namespace" : "com.five9.avro.enum.test",
>> "fields" : [ {
>> "name" : "enumType",
>> "type" : {
>> "type" : "enum",
>> "name" : "EnumType",
>> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6", "S7" ]
>> }
>> } ]
>> }
>>
>>
>> These schemas show as compatible for mutual read
>>
>> This is not in line with what the specification says
>>
>> *
>>
>> if both are enums: if the writer’s symbol is not present in the
>> reader’s enum and the reader has a default value, then that value
>> is used, otherwise an error is signalled.
>>
>>
>> I have tried this in Avro 1.9.1 and 1.11.1. Is this a bug? If not, what
>> am I doing wrong?
>>
>> Appreciate reponses
>>
>> Regards,
>> Kishore
>
Re: enum schema evolution
Posted by KV 59 <kv...@gmail.com>.
It works both ways in compatibility Validation. I have used the Avro
library for compatibility check
SchemaValidator validator = builder.mutualReadStrategy().validateAll();
The SchemaValidator says it is valid with the first schema as reader and
second schema as writer and vice versa is valid as well.
Obviously this is not the case. since symbol "S7" is not present in schema1
and also it has no default value defined,
On Fri, May 5, 2023 at 7:37 AM Brennan Vincent <br...@umanwizard.com>
wrote:
> Which one is the writer and which is the reader?
>
> Sent from my iPhone
>
> On May 4, 2023, at 22:47, KV 59 <kv...@gmail.com> wrote:
>
>
> Hi,
>
> I see that that java Avro compatibility check doesn't work as per
> specification for enum schema evolution. I have a the following schema
>
> {
>> "type" : "record",
>> "name" : "TestEnumRec",
>> "namespace" : "com.five9.avro.enum.test",
>> "fields" : [ {
>> "name" : "enumType",
>> "type" : {
>> "type" : "enum",
>> "name" : "EnumType",
>> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6" ]
>> }
>> } ]
>> }
>
>
> And another version of the same schema
>
> {
>> "type" : "record",
>> "name" : "TestEnumRec",
>> "namespace" : "com.five9.avro.enum.test",
>> "fields" : [ {
>> "name" : "enumType",
>> "type" : {
>> "type" : "enum",
>> "name" : "EnumType",
>> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6", "S7" ]
>> }
>> } ]
>> }
>
>
> These schemas show as compatible for mutual read
>
> This is not in line with what the specification says
>
>>
>> -
>>
>> if both are enums: if the writer’s symbol is not present in the
>> reader’s enum and the reader has a default value, then that value is used,
>> otherwise an error is signalled.
>>
>>
> I have tried this in Avro 1.9.1 and 1.11.1. Is this a bug? If not, what
> am I doing wrong?
>
> Appreciate reponses
>
> Regards,
> Kishore
>
>
Re: enum schema evolution
Posted by Brennan Vincent <br...@umanwizard.com>.
Which one is the writer and which is the reader?
Sent from my iPhone
> On May 4, 2023, at 22:47, KV 59 <kv...@gmail.com> wrote:
>
>
> Hi,
>
> I see that that java Avro compatibility check doesn't work as per specification for enum schema evolution. I have a the following schema
>> {
>> "type" : "record",
>> "name" : "TestEnumRec",
>> "namespace" : "com.five9.avro.enum.test",
>> "fields" : [ {
>> "name" : "enumType",
>> "type" : {
>> "type" : "enum",
>> "name" : "EnumType",
>> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6" ]
>> }
>> } ]
>> }
>
> And another version of the same schema
>
>> {
>> "type" : "record",
>> "name" : "TestEnumRec",
>> "namespace" : "com.five9.avro.enum.test",
>> "fields" : [ {
>> "name" : "enumType",
>> "type" : {
>> "type" : "enum",
>> "name" : "EnumType",
>> "symbols" : [ "S1", "S2", "S3", "S4", "S5", "S6", "S7" ]
>> }
>> } ]
>> }
>
> These schemas show as compatible for mutual read
>
> This is not in line with what the specification says
>> if both are enums: if the writer’s symbol is not present in the reader’s enum and the reader has a default value, then that value is used, otherwise an error is signalled.
>>
>
> I have tried this in Avro 1.9.1 and 1.11.1. Is this a bug? If not, what am I doing wrong?
>
> Appreciate reponses
>
> Regards,
> Kishore