You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Vinicius Carvalho <vi...@gmail.com> on 2016/05/04 15:02:14 UTC
Schema resolution question
Hi there, I'm using Avro 1.8.0, and I'm not sure why one of my
compatibility tests is failing.
I have two schemas:
V1:
{
"namespace" : "io.igx.android",
"type" : "record",
"name" : "Sensor",
"fields" : [
{"name":"id","type":"string"},
{"name":"temperature", "type":"float"},
{"name":"acceleration", "type":"float","default":0.0},
{"name":"velocity","type":"float","default":0.0},
{"name":"accelerometer","type":[
"null",{
"type":"array",
"items":"float"
}
]},
{"name":"magneticField","type":[
"null",{
"type":"array",
"items":"float"
}
]},
{"name":"orientation","type":[
"null",{
"type":"array",
"items":"float"
}
],"default":"null"}
]
}
V2:
{
"namespace" : "io.igx.android",
"type" : "record",
"name" : "Sensor",
"fields" : [
{"name":"id","type":"string"},
{"name":"temperature", "type":"float", "default":0.0},
{"name":"acceleration", "type":"float","default":0.0},
{"name":"velocity","type":"float","default":0.0},
{"name":"accelerometer","type":[
"null",{
"type":"array",
"items":"float"
}
]},
{"name":"magneticField","type":[
"null",{
"type":"array",
"items":"float"
}
]}
]
}
As you can see the only difference between those two is that one of the
arrays is removed from V2, and I've added a default value to a property on
V2.
If I run:
SchemaCompatibility.checkReaderWriterCompatibility(load("schemas/sensor_v2.avsc"),load("schemas/sensor_v1.avsc")).getType()
It prints out that data written in v2 should be able to be read by v1,
which seems logical
But when I try to deserialize data from v2 using v1 reader I get an error:
Sensor v1Fromv2 = deserialize(v2Bytes,v1.getSchema(),v2.getSchema());
private <T> T deserialize(byte[] bytes, Schema reader, Schema writer)
throws Exception{
DatumReader datumReader = new SpecificDatumReader<>(writer,reader);
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes,null);
return (T) datumReader.read(null,decoder);
}
org.apache.avro.AvroTypeException: Non-null default value for null type:
"null"
Not sure why is the deserialization not working if the compatibility check
passes
Any ideas?
Regards
Re: Schema resolution question
Posted by Vinicius Carvalho <vi...@gmail.com>.
Spot on! Thank you Tim
On Wed, May 4, 2016 at 11:14 AM, Tim Perkins <tp...@salsify.com> wrote:
> In the V1 schema, I think the default for the orientation array should be
> null (without the quotes), not the string "null".
>
> On Wed, May 4, 2016 at 11:02 AM, Vinicius Carvalho <
> viniciusccarvalho@gmail.com> wrote:
>
>> Hi there, I'm using Avro 1.8.0, and I'm not sure why one of my
>> compatibility tests is failing.
>>
>> I have two schemas:
>>
>> V1:
>>
>> {
>> "namespace" : "io.igx.android",
>> "type" : "record",
>> "name" : "Sensor",
>> "fields" : [
>> {"name":"id","type":"string"},
>> {"name":"temperature", "type":"float"},
>> {"name":"acceleration", "type":"float","default":0.0},
>> {"name":"velocity","type":"float","default":0.0},
>> {"name":"accelerometer","type":[
>> "null",{
>> "type":"array",
>> "items":"float"
>> }
>> ]},
>> {"name":"magneticField","type":[
>> "null",{
>> "type":"array",
>> "items":"float"
>> }
>> ]},
>> {"name":"orientation","type":[
>> "null",{
>> "type":"array",
>> "items":"float"
>> }
>> ],"default":"null"}
>>
>> ]
>>
>> }
>>
>> V2:
>> {
>> "namespace" : "io.igx.android",
>> "type" : "record",
>> "name" : "Sensor",
>> "fields" : [
>> {"name":"id","type":"string"},
>> {"name":"temperature", "type":"float", "default":0.0},
>> {"name":"acceleration", "type":"float","default":0.0},
>> {"name":"velocity","type":"float","default":0.0},
>> {"name":"accelerometer","type":[
>> "null",{
>> "type":"array",
>> "items":"float"
>> }
>> ]},
>> {"name":"magneticField","type":[
>> "null",{
>> "type":"array",
>> "items":"float"
>> }
>> ]}
>>
>> ]
>>
>> }
>>
>> As you can see the only difference between those two is that one of the
>> arrays is removed from V2, and I've added a default value to a property on
>> V2.
>>
>> If I run:
>>
>> SchemaCompatibility.checkReaderWriterCompatibility(load("schemas/sensor_v2.avsc"),load("schemas/sensor_v1.avsc")).getType()
>>
>> It prints out that data written in v2 should be able to be read by v1,
>> which seems logical
>>
>> But when I try to deserialize data from v2 using v1 reader I get an error:
>>
>> Sensor v1Fromv2 = deserialize(v2Bytes,v1.getSchema(),v2.getSchema());
>>
>> private <T> T deserialize(byte[] bytes, Schema reader, Schema writer)
>> throws Exception{
>> DatumReader datumReader = new SpecificDatumReader<>(writer,reader);
>> Decoder decoder = DecoderFactory.get().binaryDecoder(bytes,null);
>> return (T) datumReader.read(null,decoder);
>> }
>> org.apache.avro.AvroTypeException: Non-null default value for null type:
>> "null"
>>
>> Not sure why is the deserialization not working if the compatibility
>> check passes
>>
>> Any ideas?
>>
>> Regards
>>
>
>
>
> --
>
> *Tim Perkins*
> *Software Architect, Salsify*
> *a:* 1 Winthrop Square • Boston, MA 02110
> *t:* 617-669-7835
> *e:* tperkins@salsify.com
> *w:* www.salsify.com
>
> See how customers are rating Salsify
> <https://www.g2crowd.com/products/salsify/reviews>
>
Re: Schema resolution question
Posted by Tim Perkins <tp...@salsify.com>.
In the V1 schema, I think the default for the orientation array should be
null (without the quotes), not the string "null".
On Wed, May 4, 2016 at 11:02 AM, Vinicius Carvalho <
viniciusccarvalho@gmail.com> wrote:
> Hi there, I'm using Avro 1.8.0, and I'm not sure why one of my
> compatibility tests is failing.
>
> I have two schemas:
>
> V1:
>
> {
> "namespace" : "io.igx.android",
> "type" : "record",
> "name" : "Sensor",
> "fields" : [
> {"name":"id","type":"string"},
> {"name":"temperature", "type":"float"},
> {"name":"acceleration", "type":"float","default":0.0},
> {"name":"velocity","type":"float","default":0.0},
> {"name":"accelerometer","type":[
> "null",{
> "type":"array",
> "items":"float"
> }
> ]},
> {"name":"magneticField","type":[
> "null",{
> "type":"array",
> "items":"float"
> }
> ]},
> {"name":"orientation","type":[
> "null",{
> "type":"array",
> "items":"float"
> }
> ],"default":"null"}
>
> ]
>
> }
>
> V2:
> {
> "namespace" : "io.igx.android",
> "type" : "record",
> "name" : "Sensor",
> "fields" : [
> {"name":"id","type":"string"},
> {"name":"temperature", "type":"float", "default":0.0},
> {"name":"acceleration", "type":"float","default":0.0},
> {"name":"velocity","type":"float","default":0.0},
> {"name":"accelerometer","type":[
> "null",{
> "type":"array",
> "items":"float"
> }
> ]},
> {"name":"magneticField","type":[
> "null",{
> "type":"array",
> "items":"float"
> }
> ]}
>
> ]
>
> }
>
> As you can see the only difference between those two is that one of the
> arrays is removed from V2, and I've added a default value to a property on
> V2.
>
> If I run:
>
> SchemaCompatibility.checkReaderWriterCompatibility(load("schemas/sensor_v2.avsc"),load("schemas/sensor_v1.avsc")).getType()
>
> It prints out that data written in v2 should be able to be read by v1,
> which seems logical
>
> But when I try to deserialize data from v2 using v1 reader I get an error:
>
> Sensor v1Fromv2 = deserialize(v2Bytes,v1.getSchema(),v2.getSchema());
>
> private <T> T deserialize(byte[] bytes, Schema reader, Schema writer)
> throws Exception{
> DatumReader datumReader = new SpecificDatumReader<>(writer,reader);
> Decoder decoder = DecoderFactory.get().binaryDecoder(bytes,null);
> return (T) datumReader.read(null,decoder);
> }
> org.apache.avro.AvroTypeException: Non-null default value for null type:
> "null"
>
> Not sure why is the deserialization not working if the compatibility check
> passes
>
> Any ideas?
>
> Regards
>
--
*Tim Perkins*
*Software Architect, Salsify*
*a:* 1 Winthrop Square • Boston, MA 02110
*t:* 617-669-7835
*e:* tperkins@salsify.com
*w:* www.salsify.com
See how customers are rating Salsify
<https://www.g2crowd.com/products/salsify/reviews>