You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by "Chau, Victor" <vi...@x.com> on 2011/12/07 19:13:36 UTC
Importing in avdl from classpath of project
Hello,
I am trying to address a shortcoming of the way that the import feature works in IDL. Currently, it looks like the only option is to place the file being imported inside the same directory as that of the importing avdl.
In our setup, we have avdl's that are spread among several maven projects that are owned by different teams. I would like to be able to just create a dependency on another jar that contains the avdl I am want to import and have Avro be smart enough to look for it in the classpath of the project containing the avdl.
The main problem is to make all of this work with the avro-maven-plugin. The plugin's runtime classpath is not the same as that of the maven project's classpath. Through the magic of Stackoverflow, I figured out how to get the project's classpath and construct a new classloader and pass it to the Idl compiler for it to lookup the file if it is not available in the local directory.
Is this a feature that people think would be useful? Essentially, the IDL syntax would not change but the behavior is:
1. If imported file is available locally (in the current input path), use it
2. Else look for it on the project's classpath.
If so, I have a working patch that needs some cleanup but I can submit it as a feature request in JIRA.
Multiple records in a single schema in C++
Posted by Francois Forster <fr...@bazaarvoice.com>.
Hi there,
I'm trying to define a record with a sub-record and keep getting an error.
I modified the example cpx.json as follows:
[
{
"type": "record",
"name": "cpy",
"fields" : [
{"name": "re", "type": "double"},
{"name": "im", "type" : "double"}
]
}
,
{
"type": "record",
"name": "cpx",
"fields" : [
{"name": "re", "type": "double"},
{"name": "im", "type" : "double"},
{"name": "opt", "type" : union ["null","cpy"]}
]
}
]
And get the following error:
terminate called after throwing an instance of 'avro::Exception'
what(): Invalid operation. Expected: Double got Union
union Aborted
Is the schema supposed to be formatted differently?
Thanks,
Francois.
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Thanks.
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 5:08 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
On 12/07/2011 02:55 PM, Francois Forster wrote:
> I'm trying to test the case where a server returns a newer version of a response to make sure an older client can read it.
> I think I can accomplish that by removing the version out of the namespace altogether in my test case.
Yes, for bi-directional compatibility you should not change the
namespace or the name of your records. Also it's best to specify a
default value for fields. If you add a new field these will be used
when reading old data. If you remove a field and read newer data using
the old schema then the default would be used.
Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
On 12/07/2011 02:55 PM, Francois Forster wrote:
> I'm trying to test the case where a server returns a newer version of a response to make sure an older client can read it.
> I think I can accomplish that by removing the version out of the namespace altogether in my test case.
Yes, for bi-directional compatibility you should not change the
namespace or the name of your records. Also it's best to specify a
default value for fields. If you add a new field these will be used
when reading old data. If you remove a field and read newer data using
the old schema then the default would be used.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
I'm trying to test the case where a server returns a newer version of a response to make sure an older client can read it.
I think I can accomplish that by removing the version out of the namespace altogether in my test case.
Francois.
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 4:48 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
I think you have the parameters reversed to SpecificDatumWriter and
SpecificDatumReader. If v2 is the new version, then you should
construct a SpecificDatumWriter<v1.Response> to write the old version
and a SpecificDatumReader<v2.Response> to read the new one. The
parameters to SpecificDatumReader's constructor are old-then-new, so
you'll need to reverse those too.
Doug
On 12/07/2011 02:40 PM, Francois Forster wrote:
> It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
>
> Francois.
>
> -----Original Message-----
> From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
> Sent: Wednesday, December 07, 2011 4:34 PM
> To: user@avro.apache.org
> Subject: RE: AvroTypeException thrown with version change on optional record
>
> Interesting. I tried it but get:
>
> Exception in thread "main" org.apache.avro.AvroTypeException: Found {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v2",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ],
> "aliases" : [ "v1.Review" ]
> }, expecting [ "null", {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v1",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ]
> } ]
>
>
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org]
> Sent: Wednesday, December 07, 2011 4:15 PM
> To: user@avro.apache.org
> Subject: Re: AvroTypeException thrown with version change on optional record
>
> On 12/07/2011 01:41 PM, Francois Forster wrote:
>> Actually, it happens even if I don't add isFeatured. Is there something
>> incompatible due to the different namespace?
>
> Changing the namespace is probably why this is failing.
>
> If you need to change the namespace then you can use aliases:
>
> @namespace("v2")
> protocol Service {
> @aliases(["v1.Result"])
> record Result { ... }
> ...
> }
>
> This will make v2 be able to read v1.
>
> Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
I think you have the parameters reversed to SpecificDatumWriter and
SpecificDatumReader. If v2 is the new version, then you should
construct a SpecificDatumWriter<v1.Response> to write the old version
and a SpecificDatumReader<v2.Response> to read the new one. The
parameters to SpecificDatumReader's constructor are old-then-new, so
you'll need to reverse those too.
Doug
On 12/07/2011 02:40 PM, Francois Forster wrote:
> It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
>
> Francois.
>
> -----Original Message-----
> From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
> Sent: Wednesday, December 07, 2011 4:34 PM
> To: user@avro.apache.org
> Subject: RE: AvroTypeException thrown with version change on optional record
>
> Interesting. I tried it but get:
>
> Exception in thread "main" org.apache.avro.AvroTypeException: Found {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v2",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ],
> "aliases" : [ "v1.Review" ]
> }, expecting [ "null", {
> "type" : "record",
> "name" : "Review",
> "namespace" : "v1",
> "fields" : [ {
> "name" : "id",
> "type" : "string"
> }, {
> "name" : "text",
> "type" : "string"
> }, {
> "name" : "isRecommended",
> "type" : "boolean"
> } ]
> } ]
>
>
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org]
> Sent: Wednesday, December 07, 2011 4:15 PM
> To: user@avro.apache.org
> Subject: Re: AvroTypeException thrown with version change on optional record
>
> On 12/07/2011 01:41 PM, Francois Forster wrote:
>> Actually, it happens even if I don't add isFeatured. Is there something
>> incompatible due to the different namespace?
>
> Changing the namespace is probably why this is failing.
>
> If you need to change the namespace then you can use aliases:
>
> @namespace("v2")
> protocol Service {
> @aliases(["v1.Result"])
> record Result { ... }
> ...
> }
>
> This will make v2 be able to read v1.
>
> Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
On 12/07/2011 02:40 PM, Francois Forster wrote:
> It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
Note that you could programmatically add the alias, e.g.:
v2Response.addAlias(v1Response.getFullName());
v2Result.addAlias(v1Result.getFullName());
So you don't actually have to change the stored data.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
It seems to work when I do the reverse (add @aliases(["v2.Result"]) to the v1 schema), but that's not useful when dealing with versioning.
Francois.
-----Original Message-----
From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
Sent: Wednesday, December 07, 2011 4:34 PM
To: user@avro.apache.org
Subject: RE: AvroTypeException thrown with version change on optional record
Interesting. I tried it but get:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Review",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ],
"aliases" : [ "v1.Review" ]
}, expecting [ "null", {
"type" : "record",
"name" : "Review",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 4:15 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
On 12/07/2011 01:41 PM, Francois Forster wrote:
> Actually, it happens even if I don't add isFeatured. Is there something
> incompatible due to the different namespace?
Changing the namespace is probably why this is failing.
If you need to change the namespace then you can use aliases:
@namespace("v2")
protocol Service {
@aliases(["v1.Result"])
record Result { ... }
...
}
This will make v2 be able to read v1.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Interesting. I tried it but get:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Review",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ],
"aliases" : [ "v1.Review" ]
}, expecting [ "null", {
"type" : "record",
"name" : "Review",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, December 07, 2011 4:15 PM
To: user@avro.apache.org
Subject: Re: AvroTypeException thrown with version change on optional record
On 12/07/2011 01:41 PM, Francois Forster wrote:
> Actually, it happens even if I don't add isFeatured. Is there something
> incompatible due to the different namespace?
Changing the namespace is probably why this is failing.
If you need to change the namespace then you can use aliases:
@namespace("v2")
protocol Service {
@aliases(["v1.Result"])
record Result { ... }
...
}
This will make v2 be able to read v1.
Doug
Re: AvroTypeException thrown with version change on optional record
Posted by Doug Cutting <cu...@apache.org>.
On 12/07/2011 01:41 PM, Francois Forster wrote:
> Actually, it happens even if I don’t add isFeatured. Is there something
> incompatible due to the different namespace?
Changing the namespace is probably why this is failing.
If you need to change the namespace then you can use aliases:
@namespace("v2")
protocol Service {
@aliases(["v1.Result"])
record Result { ... }
...
}
This will make v2 be able to read v1.
Doug
RE: AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Actually, it happens even if I don't add isFeatured. Is there something incompatible due to the different namespace?
From: Francois Forster [mailto:francois.forster@bazaarvoice.com]
Sent: Wednesday, December 07, 2011 2:53 PM
To: user@avro.apache.org
Subject: AvroTypeException thrown with version change on optional record
Hi,
I'm trying to test versioning in Avro data serialization and I'm seeing an AvroTypeException thrown when I add a new field to an optional record.
Here's the V1 schema:
@namespace("v1")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
V2 (added isFeatured to Result):
@namespace("v2")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
boolean isFeatured;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
Here's the code:
import org.apache.avro.Protocol;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.BinaryEncoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
public class AvroServiceMain {
public static void main(String[] args)
throws Exception {
Protocol protocol_v2 = Protocol.parse(new File("TestCase_v2.avdr"));
Protocol protocol_v1 = Protocol.parse(new File("TestCase_v1.avdr"));
SpecificDatumWriter< v2.Response> responseSerializer = new SpecificDatumWriter< v2.Response>(protocol_v2.getType("Response"));
SpecificDatumReader< v1.Response> responseDeserializer = new SpecificDatumReader< v1.Response>(protocol_v2.getType("Response"),protocol_v1.getType("Response"));
v2.Response responseV2 = new v2.Response();
responseV2.setVersion(2);
responseV2.setSuccess(true);
v2.Result result = new v2.Result();
result.setId("1234");
result.setIsRecommended(false);
result.setIsFeatured(true);
result.setText("Some text...");
responseV2.setResults(result);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(baos, null);
responseSerializer.write(responseV2, encoder);
byte[] bytes = baos.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
BinaryDecoder decoder = DecoderFactory.get().directBinaryDecoder(bais, null);
v1.Response responseV1 = new v1.Response();
responseDeserializer.read(responseV1, decoder);
System.out.println(responseV1);
}
}
Here's the exception thrown:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Result",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
}, {
"name" : "isFeatured",
"type" : "boolean"
} ]
}, expecting [ "null", {
"type" : "record",
"name" : "Result",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:135)
at TestCaseMain.main(TestCaseMain.java:44)
The exception doesn't occur when the field is not Optional.
Am I missing something or does this look like a bug?
Thanks,
Francois.
AvroTypeException thrown with version change on optional record
Posted by Francois Forster <fr...@bazaarvoice.com>.
Hi,
I'm trying to test versioning in Avro data serialization and I'm seeing an AvroTypeException thrown when I add a new field to an optional record.
Here's the V1 schema:
@namespace("v1")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
V2 (added isFeatured to Result):
@namespace("v2")
protocol Service {
record Result {
string id;
string text;
boolean isRecommended;
boolean isFeatured;
}
record Response {
int version;
boolean success;
union {null, Result} results;
}
}
Here's the code:
import org.apache.avro.Protocol;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.BinaryEncoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
public class AvroServiceMain {
public static void main(String[] args)
throws Exception {
Protocol protocol_v2 = Protocol.parse(new File("TestCase_v2.avdr"));
Protocol protocol_v1 = Protocol.parse(new File("TestCase_v1.avdr"));
SpecificDatumWriter< v2.Response> responseSerializer = new SpecificDatumWriter< v2.Response>(protocol_v2.getType("Response"));
SpecificDatumReader< v1.Response> responseDeserializer = new SpecificDatumReader< v1.Response>(protocol_v2.getType("Response"),protocol_v1.getType("Response"));
v2.Response responseV2 = new v2.Response();
responseV2.setVersion(2);
responseV2.setSuccess(true);
v2.Result result = new v2.Result();
result.setId("1234");
result.setIsRecommended(false);
result.setIsFeatured(true);
result.setText("Some text...");
responseV2.setResults(result);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(baos, null);
responseSerializer.write(responseV2, encoder);
byte[] bytes = baos.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
BinaryDecoder decoder = DecoderFactory.get().directBinaryDecoder(bais, null);
v1.Response responseV1 = new v1.Response();
responseDeserializer.read(responseV1, decoder);
System.out.println(responseV1);
}
}
Here's the exception thrown:
Exception in thread "main" org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "Result",
"namespace" : "v2",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
}, {
"name" : "isFeatured",
"type" : "boolean"
} ]
}, expecting [ "null", {
"type" : "record",
"name" : "Result",
"namespace" : "v1",
"fields" : [ {
"name" : "id",
"type" : "string"
}, {
"name" : "text",
"type" : "string"
}, {
"name" : "isRecommended",
"type" : "boolean"
} ]
} ]
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:135)
at TestCaseMain.main(TestCaseMain.java:44)
The exception doesn't occur when the field is not Optional.
Am I missing something or does this look like a bug?
Thanks,
Francois.
RE: Importing in avdl from classpath of project
Posted by "Chau, Victor" <vi...@x.com>.
Hi Scott,
I created JIRA issue AVRO-971.
From: Scott Carey [mailto:scott@richrelevance.com] On Behalf Of Scott Carey
Sent: Wednesday, December 07, 2011 10:50 AM
To: user@avro.apache.org
Subject: Re: Importing in avdl from classpath of project
I think that at minimum, it would be useful to have an option to 'also look in the classpath' in the maven plugin, and have the option to do so in general with the IDL compiler. I would gladly review the patch in a JIRA.
-Scott
On 12/7/11 10:13 AM, "Chau, Victor" <vi...@x.com>> wrote:
Hello,
I am trying to address a shortcoming of the way that the import feature works in IDL. Currently, it looks like the only option is to place the file being imported inside the same directory as that of the importing avdl.
In our setup, we have avdl's that are spread among several maven projects that are owned by different teams. I would like to be able to just create a dependency on another jar that contains the avdl I am want to import and have Avro be smart enough to look for it in the classpath of the project containing the avdl.
The main problem is to make all of this work with the avro-maven-plugin. The plugin's runtime classpath is not the same as that of the maven project's classpath. Through the magic of Stackoverflow, I figured out how to get the project's classpath and construct a new classloader and pass it to the Idl compiler for it to lookup the file if it is not available in the local directory.
Is this a feature that people think would be useful? Essentially, the IDL syntax would not change but the behavior is:
1. If imported file is available locally (in the current input path), use it
2. Else look for it on the project's classpath.
If so, I have a working patch that needs some cleanup but I can submit it as a feature request in JIRA.
Re: Importing in avdl from classpath of project
Posted by Scott Carey <sc...@apache.org>.
I think that at minimum, it would be useful to have an option to 'also look
in the classpath' in the maven plugin, and have the option to do so in
general with the IDL compiler. I would gladly review the patch in a JIRA.
-Scott
On 12/7/11 10:13 AM, "Chau, Victor" <vi...@x.com> wrote:
> Hello,
>
> I am trying to address a shortcoming of the way that the import feature works
> in IDL. Currently, it looks like the only option is to place the file being
> imported inside the same directory as that of the importing avdl.
>
> In our setup, we have avdl¹s that are spread among several maven projects that
> are owned by different teams. I would like to be able to just create a
> dependency on another jar that contains the avdl I am want to import and have
> Avro be smart enough to look for it in the classpath of the project containing
> the avdl.
>
> The main problem is to make all of this work with the avro-maven-plugin. The
> plugin¹s runtime classpath is not the same as that of the maven project¹s
> classpath. Through the magic of Stackoverflow, I figured out how to get the
> project¹s classpath and construct a new classloader and pass it to the Idl
> compiler for it to lookup the file if it is not available in the local
> directory.
>
> Is this a feature that people think would be useful? Essentially, the IDL
> syntax would not change but the behavior is:
>
> 1. If imported file is available locally (in the current input path),
> use it
>
> 2. Else look for it on the project¹s classpath.
>
>
> If so, I have a working patch that needs some cleanup but I can submit it as a
> feature request in JIRA.