You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Francois Forster <fr...@bazaarvoice.com> on 2011/12/07 21:53:15 UTC
AvroTypeException thrown with version change on optional record
Hi,
I'm trying to test versioning in Avro data serialization and I'm seeing an AvroTypeException thrown when I add a new field to an optional record.
Here's the V1 schema:
@namespace("v1")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
V2 (added isFeatured to Result):
@namespace("v2")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
boolean isFeatured;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
Here's the code:
import org.apache.avro.Protocol;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.BinaryEncoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
public class AvroServiceMain {
public static void main(String[] args)
throws Exception {
Protocol protocol_v2 = Protocol.parse(new File("TestCase_v2.avdr"));
Protocol protocol_v1 = Protocol.parse(new File("TestCase_v1.avdr"));
SpecificDatumWriter< v2.Response> responseSerializer = new SpecificDatumWriter< v2.Response>(protocol_v2.getType("Response"));
SpecificDatumReader< v1.Response> responseDeserializer = new SpecificDatumReader< v1.Response>(protocol_v2.getType("Response"),protocol_v1.getType("Response"));
v2.Response responseV2 = new v2.Response();
responseV2.setVersion(2);
responseV2.setSuccess(true);
v2.Result result = new v2.Result();
result.setId("1234");
result.setIsRecommended(false);
result.setIsFeatured(true);
result.setText("Some text...");
responseV2.setResults(result);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(baos, null);
responseSerializer.write(responseV2, encoder);
byte[] bytes = baos.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
BinaryDecoder decoder = DecoderFactory.get().directBinaryDecoder(bais, null);
v1.Response responseV1 = new v1.Response();
responseDeserializer.read(responseV1, decoder);
System.out.println(responseV1);
}
}
Here's the exception thrown:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Result",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
}, {
"name" : "isFeatured",
"type" : "boolean"
} ]
}, expecting [ "null", {
"type" : "record",
"name" : "Result",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:135)
at TestCaseMain.main(TestCaseMain.java:44)
The exception doesn't occur when the field is not Optional.
Am I missing something or does this look like a bug?
Thanks,
Francois.
Multiple records in a single schema in C++
Posted by Francois Forster <fr...@bazaarvoice.com>.
Hi there,
I'm trying to define a record with a sub-record and keep getting an error.
I modified the example cpx.json as follows:
[
{
"type": "record",
"name": "cpy",
"fields" : [
{"name": "re", "type": "double"},
{"name": "im", "type" : "double"}
]
}
,
{
"type": "record",
"name": "cpx",
"fields" : [
{"name": "re", "type": "double"},
{"name": "im", "type" : "double"},
{"name": "opt", "type" : union ["null","cpy"]}
]
}
]
And get the following error:
terminate called after throwing an instance of 'avro::Exception'
what(): Invalid operation. Expected: Double got Union
union Aborted
Is the schema supposed to be formatted differently?
Thanks,
Francois.
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Thanks.
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 5:08 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
On 12/07/2011 02:55 PM, Francois Forster wrote:
> I'm trying to test the case where a server returns a newer version of a response to make sure an older client can read it.
> I think I can accomplish that by removing the version out of the namespace altogether in my test case.
Yes, for bi-directional compatibility you should not change the
namespace or the name of your records. Also it's best to specify a
default value for fields. If you add a new field these will be used
when reading old data. If you remove a field and read newer data using
the old schema then the default would be used.
Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
On 12/07/2011 02:55 PM, Francois Forster wrote:
> I'm trying to test the case where a server returns a newer version of a response to make sure an older client can read it.
> I think I can accomplish that by removing the version out of the namespace altogether in my test case.
Yes, for bi-directional compatibility you should not change the
namespace or the name of your records. Also it's best to specify a
default value for fields. If you add a new field these will be used
when reading old data. If you remove a field and read newer data using
the old schema then the default would be used.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
I'm trying to test the case where a server returns a newer version of a response to make sure an older client can read it.
I think I can accomplish that by removing the version out of the namespace altogether in my test case.
Francois.
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 4:48 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
I think you have the parameters reversed to SpecificDatumWriter and
SpecificDatumReader. If v2 is the new version, then you should
construct a SpecificDatumWriter<v1.Response> to write the old version
and a SpecificDatumReader<v2.Response> to read the new one. The
parameters to SpecificDatumReader's constructor are old-then-new, so
you'll need to reverse those too.
Doug
On 12/07/2011 02:40 PM, Francois Forster wrote:
> It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
>
> Francois.
>
> -----Original Message-----
> From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
> Sent: Wednesday, December 07, 2011 4:34 PM
> To: user@avro.apache.org
> Subject: RE: AvroTypeException thrown with version change on optional record
>
> Interesting. I tried it but get:
>
> Exception in thread "main" org.apache.avro.AvroTypeException: Found {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v2",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ],
> "aliases" : [ "v1.Review" ]
> }, expecting [ "null", {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v1",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ]
> } ]
>
>
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org]
> Sent: Wednesday, December 07, 2011 4:15 PM
> To: user@avro.apache.org
> Subject: Re: AvroTypeException thrown with version change on optional record
>
> On 12/07/2011 01:41 PM, Francois Forster wrote:
>> Actually, it happens even if I don't add isFeatured. Is there something
>> incompatible due to the different namespace?
>
> Changing the namespace is probably why this is failing.
>
> If you need to change the namespace then you can use aliases:
>
> @namespace("v2")
> protocol Service {
> @aliases(["v1.Result"])
> record Result { ... }
> ...
> }
>
> This will make v2 be able to read v1.
>
> Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
I think you have the parameters reversed to SpecificDatumWriter and
SpecificDatumReader. If v2 is the new version, then you should
construct a SpecificDatumWriter<v1.Response> to write the old version
and a SpecificDatumReader<v2.Response> to read the new one. The
parameters to SpecificDatumReader's constructor are old-then-new, so
you'll need to reverse those too.
Doug
On 12/07/2011 02:40 PM, Francois Forster wrote:
> It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
>
> Francois.
>
> -----Original Message-----
> From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
> Sent: Wednesday, December 07, 2011 4:34 PM
> To: user@avro.apache.org
> Subject: RE: AvroTypeException thrown with version change on optional record
>
> Interesting. I tried it but get:
>
> Exception in thread "main" org.apache.avro.AvroTypeException: Found {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v2",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ],
> "aliases" : [ "v1.Review" ]
> }, expecting [ "null", {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v1",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ]
> } ]
>
>
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org]
> Sent: Wednesday, December 07, 2011 4:15 PM
> To: user@avro.apache.org
> Subject: Re: AvroTypeException thrown with version change on optional record
>
> On 12/07/2011 01:41 PM, Francois Forster wrote:
>> Actually, it happens even if I don't add isFeatured. Is there something
>> incompatible due to the different namespace?
>
> Changing the namespace is probably why this is failing.
>
> If you need to change the namespace then you can use aliases:
>
> @namespace("v2")
> protocol Service {
> @aliases(["v1.Result"])
> record Result { ... }
> ...
> }
>
> This will make v2 be able to read v1.
>
> Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
On 12/07/2011 02:40 PM, Francois Forster wrote:
> It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
Note that you could programmatically add the alias, e.g.:
v2Response.addAlias(v1Response.getFullName());
v2Result.addAlias(v1Result.getFullName());
So you don't actually have to change the stored data.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
Francois.
-----Original Message-----
From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
Sent: Wednesday, December 07, 2011 4:34 PM
To: user@avro.apache.org
Subject: RE: AvroTypeException thrown with version change on optional record
Interesting. I tried it but get:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Review",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ],
"aliases" : [ "v1.Review" ]
}, expecting [ "null", {
"type" : "record",
"name" : "Review",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 4:15 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
On 12/07/2011 01:41 PM, Francois Forster wrote:
> Actually, it happens even if I don't add isFeatured. Is there something
> incompatible due to the different namespace?
Changing the namespace is probably why this is failing.
If you need to change the namespace then you can use aliases:
@namespace("v2")
protocol Service {
@aliases(["v1.Result"])
record Result { ... }
...
}
This will make v2 be able to read v1.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Interesting. I tried it but get:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Review",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ],
"aliases" : [ "v1.Review" ]
}, expecting [ "null", {
"type" : "record",
"name" : "Review",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 4:15 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
On 12/07/2011 01:41 PM, Francois Forster wrote:
> Actually, it happens even if I don't add isFeatured. Is there something
> incompatible due to the different namespace?
Changing the namespace is probably why this is failing.
If you need to change the namespace then you can use aliases:
@namespace("v2")
protocol Service {
@aliases(["v1.Result"])
record Result { ... }
...
}
This will make v2 be able to read v1.
Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
On 12/07/2011 01:41 PM, Francois Forster wrote:
> Actually, it happens even if I don’t add isFeatured. Is there something
> incompatible due to the different namespace?
Changing the namespace is probably why this is failing.
If you need to change the namespace then you can use aliases:
@namespace("v2")
protocol Service {
@aliases(["v1.Result"])
record Result { ... }
...
}
This will make v2 be able to read v1.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Actually, it happens even if I don't add isFeatured. Is there something incompatible due to the different namespace?
From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
Sent: Wednesday, December 07, 2011 2:53 PM
To: user@avro.apache.org
Subject: AvroTypeException thrown with version change on optional record
Hi,
I'm trying to test versioning in Avro data serialization and I'm seeing an AvroTypeException thrown when I add a new field to an optional record.
Here's the V1 schema:
@namespace("v1")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
V2 (added isFeatured to Result):
@namespace("v2")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
boolean isFeatured;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
Here's the code:
import org.apache.avro.Protocol;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.BinaryEncoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
public class AvroServiceMain {
public static void main(String[] args)
throws Exception {
Protocol protocol_v2 = Protocol.parse(new File("TestCase_v2.avdr"));
Protocol protocol_v1 = Protocol.parse(new File("TestCase_v1.avdr"));
SpecificDatumWriter< v2.Response> responseSerializer = new SpecificDatumWriter< v2.Response>(protocol_v2.getType("Response"));
SpecificDatumReader< v1.Response> responseDeserializer = new SpecificDatumReader< v1.Response>(protocol_v2.getType("Response"),protocol_v1.getType("Response"));
v2.Response responseV2 = new v2.Response();
responseV2.setVersion(2);
responseV2.setSuccess(true);
v2.Result result = new v2.Result();
result.setId("1234");
result.setIsRecommended(false);
result.setIsFeatured(true);
result.setText("Some text...");
responseV2.setResults(result);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(baos, null);
responseSerializer.write(responseV2, encoder);
byte[] bytes = baos.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
BinaryDecoder decoder = DecoderFactory.get().directBinaryDecoder(bais, null);
v1.Response responseV1 = new v1.Response();
responseDeserializer.read(responseV1, decoder);
System.out.println(responseV1);
}
}
Here's the exception thrown:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Result",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
}, {
"name" : "isFeatured",
"type" : "boolean"
} ]
}, expecting [ "null", {
"type" : "record",
"name" : "Result",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:135)
at TestCaseMain.main(TestCaseMain.java:44)
The exception doesn't occur when the field is not Optional.
Am I missing something or does this look like a bug?
Thanks,
Francois.