You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Yang <te...@gmail.com> on 2011/05/24 20:57:27 UTC
schema backward compatibility in RPC?
I remember that Avro can handle schema changes, for example: when I
parse an incoming avro message, I can supply the known schema
associated with this message,
and the desired schema to parse it into.
but I wonder if this ability is automatically available in RPC too?
for example , originally I define a remote call echo() to be
echo ( message {
date: integer,
content: string,
}
)
now the client still uses the old code, but server changes to
echo ( message {
content: string
}
)
would that still work? if so, I wonder where the server gets the
incoming schema?
Thanks a lot
Yang
Re: schema backward compatibility in RPC?
Posted by Yang <te...@gmail.com>.
got it, thanks
On Tue, May 31, 2011 at 8:58 AM, Scott Carey <sc...@richrelevance.com>wrote:
> Null is not a valid default value for string.
>
> It is a valid default value for a union of "null" and "string" however.
>
> "" is a valid default value for a String, which is what I typically use
> when I need a default string value and "" doesn't have any meaning as a
> value.
>
>
> On 5/31/11 8:53 AM, "Yang" <te...@gmail.com> wrote:
>
> Doug:
>
> thanks a lot for the fast fix.
>
>
> it works now, but there is a minor issue, I'm not sure if this is the
> desired behavior:
>
>
> in the client schema, I set the string field "extra_field" with a default
> value of null, on the server side, the server
> probably tries to use that default value, and gave the following error:
>
>
> Tests in error:
> testClientDiffReturnType(org.apache.avro.TestSimpleClient): Non-string
> default value for string: null
>
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
>
>
>
> is a NULL default value for string valid? if so, then it's all fine
>
>
> Thanks
> Yang
>
> On Tue, May 31, 2011 at 3:17 AM, Doug Cutting <cu...@apache.org> wrote:
>
>> This is a bug. I've filed an issue and attached a patch that fixes it:
>>
>> https://issues.apache.org/jira/browse/AVRO-832
>>
>> Please tell me if this fixes things for you.
>>
>> Doug
>>
>> On 05/30/2011 08:06 PM, Yang wrote:
>> > Thanks Doug.
>> >
>> > regarding "adding or removing fields from a returned record type should
>> > work",
>> > this is actually what triggered my investigation at first, and my tests
>> > seem to show that
>> > it's not the case:
>> >
>> > please see the attached schema.tar , and you can see 2 dirs, "1/" is
>> > for server, "2/" is for client.
>> > they are created by slightly modifying the standard 1.5.1 distribution.
>> > to see the demo, you need
>> > to do (please change the "path_to....." params below accordingly )
>> >
>> > mkdir demo
>> > cd demo
>> > tar xzf path_to_avro-1.5.1-src.tar.gz
>> > mv avro-src-1.5.1 1
>> > tar xzf path_to_avro-1.5.1-src.tar.gz
>> > mv avro-src-1.5.1 2
>> > ## the following overrides some files in standard distro
>> > tar xf path_to_schema.tar
>> > cd 1/lang/java/ipc
>> > mvn compile exec:java -Dexec.mainClass=org.apache.avro.SimpleServer
>> &>log &
>> > cd -
>> > cd 2/lang/java/ipc
>> > mvn -Dtest=org.apache.avro.TestSimpleClient test
>> >
>> >
>> > you will see that the test testClientDiffReturnType() fails with
>> > following exception:
>> >
>> -------------------------------------------------------------------------------
>> > Test set: org.apache.avro.TestSimpleClient
>> >
>> -------------------------------------------------------------------------------
>> > Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.392
>> > sec <<< FAILURE!
>> > testClientDiffReturnType(org.apache.avro.TestSimpleClient) Time
>> > elapsed: 0.005 sec <<< ERROR!
>> > java.lang.ClassCastException: org.apache.avro.test.evolution.Kind cannot
>> > be cast to java.lang.CharSequence
>> > at org.apache.avro.test.evolution.TestRecord.put(TestRecord.java:28)
>> > at
>> org.apache.avro.generic.GenericData.setField(GenericData.java:463)
>> > at
>> >
>> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
>> > at
>> >
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
>> > at
>> >
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
>> > at
>> >
>> org.apache.avro.ipc.specific.SpecificRequestor.readResponse(SpecificRequestor.java:77)
>> > at org.apache.avro.ipc.Requestor.request(Requestor.java:153)
>> > at
>> >
>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
>> > at $Proxy8.echo(Unknown Source)
>> > at
>> >
>> org.apache.avro.TestSimpleClient.testClientDiffReturnType(TestSimpleClient.java:111)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:616)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>> > at
>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>> > at
>> >
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
>> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
>> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>> >
>> >
>> >
>> > the above error shows that avro on the client result parsing step tries
>> > to convert the old TestRecord.kind (second field) to the new
>> > TestRecord.extra_field, which is also the second field. so we get an
>> > error. so that's why I say that it seems not to be doing return schema
>> > conversion for me. maybe I need to do something to enable that?
>> >
>> > Thanks a lot
>> > Yang
>> >
>> > On Mon, May 30, 2011 at 12:36 AM, Doug Cutting <cutting@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > On 05/26/2011 10:10 PM, Yang wrote:
>> > > I guess the schema resolution is done only on input params, and
>> not on
>> > > return value?
>> >
>> > The same resolution should be done on return values. For example,
>> > changing the return value from one record to another would not work
>> > (unless you used aliases) but adding or removing fields from a
>> returned
>> > record type should work.
>> >
>> > > incidentally, I found that if I keep the name of an enum type ,
>> but
>> > > add new enum values to it (at the end of the value list), and then
>> use
>> > > the enum type as the return
>> > > type of my RPC call, it still works, although strictly speaking,
>> the
>> > > enum type has already changed. but this is probably quite a
>> dangerous
>> > > hack.
>> >
>> > That should not be dangerous. Avro translates enum values so that
>> > adding & removing symbols is permitted. As long as the reader's
>> schema
>> > still has the symbol that the writer sends things should work.
>> >
>> > Doug
>> >
>> >
>>
>
>
Re: schema backward compatibility in RPC?
Posted by Scott Carey <sc...@richrelevance.com>.
Null is not a valid default value for string.
It is a valid default value for a union of "null" and "string" however.
"" is a valid default value for a String, which is what I typically use when I need a default string value and "" doesn't have any meaning as a value.
On 5/31/11 8:53 AM, "Yang" <te...@gmail.com>> wrote:
Doug:
thanks a lot for the fast fix.
it works now, but there is a minor issue, I'm not sure if this is the desired behavior:
in the client schema, I set the string field "extra_field" with a default value of null, on the server side, the server
probably tries to use that default value, and gave the following error:
Tests in error:
testClientDiffReturnType(org.apache.avro.TestSimpleClient): Non-string default value for string: null
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
is a NULL default value for string valid? if so, then it's all fine
Thanks
Yang
On Tue, May 31, 2011 at 3:17 AM, Doug Cutting <cu...@apache.org>> wrote:
This is a bug. I've filed an issue and attached a patch that fixes it:
https://issues.apache.org/jira/browse/AVRO-832
Please tell me if this fixes things for you.
Doug
On 05/30/2011 08:06 PM, Yang wrote:
> Thanks Doug.
>
> regarding "adding or removing fields from a returned record type should
> work",
> this is actually what triggered my investigation at first, and my tests
> seem to show that
> it's not the case:
>
> please see the attached schema.tar , and you can see 2 dirs, "1/" is
> for server, "2/" is for client.
> they are created by slightly modifying the standard 1.5.1 distribution.
> to see the demo, you need
> to do (please change the "path_to....." params below accordingly )
>
> mkdir demo
> cd demo
> tar xzf path_to_avro-1.5.1-src.tar.gz
> mv avro-src-1.5.1 1
> tar xzf path_to_avro-1.5.1-src.tar.gz
> mv avro-src-1.5.1 2
> ## the following overrides some files in standard distro
> tar xf path_to_schema.tar
> cd 1/lang/java/ipc
> mvn compile exec:java -Dexec.mainClass=org.apache.avro.SimpleServer &>log &
> cd -
> cd 2/lang/java/ipc
> mvn -Dtest=org.apache.avro.TestSimpleClient test
>
>
> you will see that the test testClientDiffReturnType() fails with
> following exception:
> -------------------------------------------------------------------------------
> Test set: org.apache.avro.TestSimpleClient
> -------------------------------------------------------------------------------
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.392
> sec <<< FAILURE!
> testClientDiffReturnType(org.apache.avro.TestSimpleClient) Time
> elapsed: 0.005 sec <<< ERROR!
> java.lang.ClassCastException: org.apache.avro.test.evolution.Kind cannot
> be cast to java.lang.CharSequence
> at org.apache.avro.test.evolution.TestRecord.put(TestRecord.java:28)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at
> org.apache.avro.ipc.specific.SpecificRequestor.readResponse(SpecificRequestor.java:77)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:153)
> at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
> at $Proxy8.echo(Unknown Source)
> at
> org.apache.avro.TestSimpleClient.testClientDiffReturnType(TestSimpleClient.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>
>
>
> the above error shows that avro on the client result parsing step tries
> to convert the old TestRecord.kind (second field) to the new
> TestRecord.extra_field, which is also the second field. so we get an
> error. so that's why I say that it seems not to be doing return schema
> conversion for me. maybe I need to do something to enable that?
>
> Thanks a lot
> Yang
>
> On Mon, May 30, 2011 at 12:36 AM, Doug Cutting <cu...@apache.org>
> <ma...@apache.org>>> wrote:
>
> On 05/26/2011 10:10 PM, Yang wrote:
> > I guess the schema resolution is done only on input params, and not on
> > return value?
>
> The same resolution should be done on return values. For example,
> changing the return value from one record to another would not work
> (unless you used aliases) but adding or removing fields from a returned
> record type should work.
>
> > incidentally, I found that if I keep the name of an enum type , but
> > add new enum values to it (at the end of the value list), and then use
> > the enum type as the return
> > type of my RPC call, it still works, although strictly speaking, the
> > enum type has already changed. but this is probably quite a dangerous
> > hack.
>
> That should not be dangerous. Avro translates enum values so that
> adding & removing symbols is permitted. As long as the reader's schema
> still has the symbol that the writer sends things should work.
>
> Doug
>
>
Re: schema backward compatibility in RPC?
Posted by Yang <te...@gmail.com>.
Doug:
thanks a lot for the fast fix.
it works now, but there is a minor issue, I'm not sure if this is the
desired behavior:
in the client schema, I set the string field "extra_field" with a default
value of null, on the server side, the server
probably tries to use that default value, and gave the following error:
Tests in error:
testClientDiffReturnType(org.apache.avro.TestSimpleClient): Non-string
default value for string: null
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
is a NULL default value for string valid? if so, then it's all fine
Thanks
Yang
On Tue, May 31, 2011 at 3:17 AM, Doug Cutting <cu...@apache.org> wrote:
> This is a bug. I've filed an issue and attached a patch that fixes it:
>
> https://issues.apache.org/jira/browse/AVRO-832
>
> Please tell me if this fixes things for you.
>
> Doug
>
> On 05/30/2011 08:06 PM, Yang wrote:
> > Thanks Doug.
> >
> > regarding "adding or removing fields from a returned record type should
> > work",
> > this is actually what triggered my investigation at first, and my tests
> > seem to show that
> > it's not the case:
> >
> > please see the attached schema.tar , and you can see 2 dirs, "1/" is
> > for server, "2/" is for client.
> > they are created by slightly modifying the standard 1.5.1 distribution.
> > to see the demo, you need
> > to do (please change the "path_to....." params below accordingly )
> >
> > mkdir demo
> > cd demo
> > tar xzf path_to_avro-1.5.1-src.tar.gz
> > mv avro-src-1.5.1 1
> > tar xzf path_to_avro-1.5.1-src.tar.gz
> > mv avro-src-1.5.1 2
> > ## the following overrides some files in standard distro
> > tar xf path_to_schema.tar
> > cd 1/lang/java/ipc
> > mvn compile exec:java -Dexec.mainClass=org.apache.avro.SimpleServer &>log
> &
> > cd -
> > cd 2/lang/java/ipc
> > mvn -Dtest=org.apache.avro.TestSimpleClient test
> >
> >
> > you will see that the test testClientDiffReturnType() fails with
> > following exception:
> >
> -------------------------------------------------------------------------------
> > Test set: org.apache.avro.TestSimpleClient
> >
> -------------------------------------------------------------------------------
> > Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.392
> > sec <<< FAILURE!
> > testClientDiffReturnType(org.apache.avro.TestSimpleClient) Time
> > elapsed: 0.005 sec <<< ERROR!
> > java.lang.ClassCastException: org.apache.avro.test.evolution.Kind cannot
> > be cast to java.lang.CharSequence
> > at org.apache.avro.test.evolution.TestRecord.put(TestRecord.java:28)
> > at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> > at
> >
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> > at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> > at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> > at
> >
> org.apache.avro.ipc.specific.SpecificRequestor.readResponse(SpecificRequestor.java:77)
> > at org.apache.avro.ipc.Requestor.request(Requestor.java:153)
> > at
> >
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
> > at $Proxy8.echo(Unknown Source)
> > at
> >
> org.apache.avro.TestSimpleClient.testClientDiffReturnType(TestSimpleClient.java:111)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> > at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> >
> >
> >
> > the above error shows that avro on the client result parsing step tries
> > to convert the old TestRecord.kind (second field) to the new
> > TestRecord.extra_field, which is also the second field. so we get an
> > error. so that's why I say that it seems not to be doing return schema
> > conversion for me. maybe I need to do something to enable that?
> >
> > Thanks a lot
> > Yang
> >
> > On Mon, May 30, 2011 at 12:36 AM, Doug Cutting <cutting@apache.org
> > <ma...@apache.org>> wrote:
> >
> > On 05/26/2011 10:10 PM, Yang wrote:
> > > I guess the schema resolution is done only on input params, and not
> on
> > > return value?
> >
> > The same resolution should be done on return values. For example,
> > changing the return value from one record to another would not work
> > (unless you used aliases) but adding or removing fields from a
> returned
> > record type should work.
> >
> > > incidentally, I found that if I keep the name of an enum type ,
> but
> > > add new enum values to it (at the end of the value list), and then
> use
> > > the enum type as the return
> > > type of my RPC call, it still works, although strictly speaking,
> the
> > > enum type has already changed. but this is probably quite a
> dangerous
> > > hack.
> >
> > That should not be dangerous. Avro translates enum values so that
> > adding & removing symbols is permitted. As long as the reader's
> schema
> > still has the symbol that the writer sends things should work.
> >
> > Doug
> >
> >
>
Re: schema backward compatibility in RPC?
Posted by Doug Cutting <cu...@apache.org>.
This is a bug. I've filed an issue and attached a patch that fixes it:
https://issues.apache.org/jira/browse/AVRO-832
Please tell me if this fixes things for you.
Doug
On 05/30/2011 08:06 PM, Yang wrote:
> Thanks Doug.
>
> regarding "adding or removing fields from a returned record type should
> work",
> this is actually what triggered my investigation at first, and my tests
> seem to show that
> it's not the case:
>
> please see the attached schema.tar , and you can see 2 dirs, "1/" is
> for server, "2/" is for client.
> they are created by slightly modifying the standard 1.5.1 distribution.
> to see the demo, you need
> to do (please change the "path_to....." params below accordingly )
>
> mkdir demo
> cd demo
> tar xzf path_to_avro-1.5.1-src.tar.gz
> mv avro-src-1.5.1 1
> tar xzf path_to_avro-1.5.1-src.tar.gz
> mv avro-src-1.5.1 2
> ## the following overrides some files in standard distro
> tar xf path_to_schema.tar
> cd 1/lang/java/ipc
> mvn compile exec:java -Dexec.mainClass=org.apache.avro.SimpleServer &>log &
> cd -
> cd 2/lang/java/ipc
> mvn -Dtest=org.apache.avro.TestSimpleClient test
>
>
> you will see that the test testClientDiffReturnType() fails with
> following exception:
> -------------------------------------------------------------------------------
> Test set: org.apache.avro.TestSimpleClient
> -------------------------------------------------------------------------------
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.392
> sec <<< FAILURE!
> testClientDiffReturnType(org.apache.avro.TestSimpleClient) Time
> elapsed: 0.005 sec <<< ERROR!
> java.lang.ClassCastException: org.apache.avro.test.evolution.Kind cannot
> be cast to java.lang.CharSequence
> at org.apache.avro.test.evolution.TestRecord.put(TestRecord.java:28)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at
> org.apache.avro.ipc.specific.SpecificRequestor.readResponse(SpecificRequestor.java:77)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:153)
> at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
> at $Proxy8.echo(Unknown Source)
> at
> org.apache.avro.TestSimpleClient.testClientDiffReturnType(TestSimpleClient.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>
>
>
> the above error shows that avro on the client result parsing step tries
> to convert the old TestRecord.kind (second field) to the new
> TestRecord.extra_field, which is also the second field. so we get an
> error. so that's why I say that it seems not to be doing return schema
> conversion for me. maybe I need to do something to enable that?
>
> Thanks a lot
> Yang
>
> On Mon, May 30, 2011 at 12:36 AM, Doug Cutting <cutting@apache.org
> <ma...@apache.org>> wrote:
>
> On 05/26/2011 10:10 PM, Yang wrote:
> > I guess the schema resolution is done only on input params, and not on
> > return value?
>
> The same resolution should be done on return values. For example,
> changing the return value from one record to another would not work
> (unless you used aliases) but adding or removing fields from a returned
> record type should work.
>
> > incidentally, I found that if I keep the name of an enum type , but
> > add new enum values to it (at the end of the value list), and then use
> > the enum type as the return
> > type of my RPC call, it still works, although strictly speaking, the
> > enum type has already changed. but this is probably quite a dangerous
> > hack.
>
> That should not be dangerous. Avro translates enum values so that
> adding & removing symbols is permitted. As long as the reader's schema
> still has the symbol that the writer sends things should work.
>
> Doug
>
>
Re: schema backward compatibility in RPC?
Posted by Yang <te...@gmail.com>.
Thanks Doug.
regarding "adding or removing fields from a returned record type should
work",
this is actually what triggered my investigation at first, and my tests seem
to show that
it's not the case:
please see the attached schema.tar , and you can see 2 dirs, "1/" is for
server, "2/" is for client.
they are created by slightly modifying the standard 1.5.1 distribution. to
see the demo, you need
to do (please change the "path_to....." params below accordingly )
mkdir demo
cd demo
tar xzf path_to_avro-1.5.1-src.tar.gz
mv avro-src-1.5.1 1
tar xzf path_to_avro-1.5.1-src.tar.gz
mv avro-src-1.5.1 2
## the following overrides some files in standard distro
tar xf path_to_schema.tar
cd 1/lang/java/ipc
mvn compile exec:java -Dexec.mainClass=org.apache.avro.SimpleServer &>log &
cd -
cd 2/lang/java/ipc
mvn -Dtest=org.apache.avro.TestSimpleClient test
you will see that the test testClientDiffReturnType() fails with following
exception:
-------------------------------------------------------------------------------
Test set: org.apache.avro.TestSimpleClient
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.392 sec
<<< FAILURE!
testClientDiffReturnType(org.apache.avro.TestSimpleClient) Time elapsed:
0.005 sec <<< ERROR!
java.lang.ClassCastException: org.apache.avro.test.evolution.Kind cannot be
cast to java.lang.CharSequence
at org.apache.avro.test.evolution.TestRecord.put(TestRecord.java:28)
at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at
org.apache.avro.ipc.specific.SpecificRequestor.readResponse(SpecificRequestor.java:77)
at org.apache.avro.ipc.Requestor.request(Requestor.java:153)
at
org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
at $Proxy8.echo(Unknown Source)
at
org.apache.avro.TestSimpleClient.testClientDiffReturnType(TestSimpleClient.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
the above error shows that avro on the client result parsing step tries to
convert the old TestRecord.kind (second field) to the new
TestRecord.extra_field, which is also the second field. so we get an error.
so that's why I say that it seems not to be doing return schema conversion
for me. maybe I need to do something to enable that?
Thanks a lot
Yang
On Mon, May 30, 2011 at 12:36 AM, Doug Cutting <cu...@apache.org> wrote:
> On 05/26/2011 10:10 PM, Yang wrote:
> > I guess the schema resolution is done only on input params, and not on
> > return value?
>
> The same resolution should be done on return values. For example,
> changing the return value from one record to another would not work
> (unless you used aliases) but adding or removing fields from a returned
> record type should work.
>
> > incidentally, I found that if I keep the name of an enum type , but
> > add new enum values to it (at the end of the value list), and then use
> > the enum type as the return
> > type of my RPC call, it still works, although strictly speaking, the
> > enum type has already changed. but this is probably quite a dangerous
> > hack.
>
> That should not be dangerous. Avro translates enum values so that
> adding & removing symbols is permitted. As long as the reader's schema
> still has the symbol that the writer sends things should work.
>
> Doug
>
Re: schema backward compatibility in RPC?
Posted by Doug Cutting <cu...@apache.org>.
On 05/26/2011 10:10 PM, Yang wrote:
> I guess the schema resolution is done only on input params, and not on
> return value?
The same resolution should be done on return values. For example,
changing the return value from one record to another would not work
(unless you used aliases) but adding or removing fields from a returned
record type should work.
> incidentally, I found that if I keep the name of an enum type , but
> add new enum values to it (at the end of the value list), and then use
> the enum type as the return
> type of my RPC call, it still works, although strictly speaking, the
> enum type has already changed. but this is probably quite a dangerous
> hack.
That should not be dangerous. Avro translates enum values so that
adding & removing symbols is permitted. As long as the reader's schema
still has the symbol that the writer sends things should work.
Doug
Re: schema backward compatibility in RPC?
Posted by Yang <te...@gmail.com>.
Doug:
thanks a lot.
I tried it again, yes it does work. The reason I thought it doesn't
work last time was, I actually tried the following code (modified from
the simple.avpr in source code):
{
type: record, name:TestRecord, fields: [
{name : msg, type: string },
{name: date, type:int}
]
}
messages: {
echo : {
request: [ { name: var1 , type: TestRecord } ]
response: "TestRecord"
<=================================================== note here....
complex response type
}
}
so the "echo" RPC call returns a TestRecord response, I actually got
errors on the return response parsing, if I change the protocol of the
client to change the "date" param to another one: {name: signature ,
type: string }, then on the return value parsing, avro tries to insert
the value of "date" into the "signature" field, and gets a cast error.
I guess the schema resolution is done only on input params, and not on
return value?
if that's the case, then I think it probably makes sense, since the
return type is part of the method signature, and supposedly should not
change. and most likely in production, our use cases are that we
return only simple status codes, which can be encoded in int or string
types. but I think there are other cases where the return value is a
complex structure, and in those cases, it probably is quite a trouble
if you have to create a new method name, simply because you added a
new field to your output param.
incidentally, I found that if I keep the name of an enum type , but
add new enum values to it (at the end of the value list), and then use
the enum type as the return
type of my RPC call, it still works, although strictly speaking, the
enum type has already changed. but this is probably quite a dangerous
hack.
Thanks
Yang
On Thu, May 26, 2011 at 2:18 AM, Doug Cutting <cu...@apache.org> wrote:
> Yes, this is intended to work. The client and server perform a
> handshake that ensures that each has the other's protocol. Message
> parameters are treated as a record. So, in your example, the server
> would ignore the date parameter. If the server adds a parameter then it
> should supply a default value to its version of the protocol to allow
> for clients that do not supply this parameter.
>
> Doug
>
> On 05/24/2011 08:57 PM, Yang wrote:
>> I remember that Avro can handle schema changes, for example: when I
>> parse an incoming avro message, I can supply the known schema
>> associated with this message,
>> and the desired schema to parse it into.
>>
>>
>> but I wonder if this ability is automatically available in RPC too?
>>
>> for example , originally I define a remote call echo() to be
>>
>> echo ( message {
>> date: integer,
>> content: string,
>> }
>> )
>>
>>
>> now the client still uses the old code, but server changes to
>>
>> echo ( message {
>> content: string
>> }
>> )
>>
>>
>> would that still work? if so, I wonder where the server gets the
>> incoming schema?
>>
>> Thanks a lot
>> Yang
>
Re: schema backward compatibility in RPC?
Posted by Doug Cutting <cu...@apache.org>.
Yes, this is intended to work. The client and server perform a
handshake that ensures that each has the other's protocol. Message
parameters are treated as a record. So, in your example, the server
would ignore the date parameter. If the server adds a parameter then it
should supply a default value to its version of the protocol to allow
for clients that do not supply this parameter.
Doug
On 05/24/2011 08:57 PM, Yang wrote:
> I remember that Avro can handle schema changes, for example: when I
> parse an incoming avro message, I can supply the known schema
> associated with this message,
> and the desired schema to parse it into.
>
>
> but I wonder if this ability is automatically available in RPC too?
>
> for example , originally I define a remote call echo() to be
>
> echo ( message {
> date: integer,
> content: string,
> }
> )
>
>
> now the client still uses the old code, but server changes to
>
> echo ( message {
> content: string
> }
> )
>
>
> would that still work? if so, I wonder where the server gets the
> incoming schema?
>
> Thanks a lot
> Yang