how to handle schema changes in RPC? (RPC version support)

[Yang <te...@gmail.com>]

generally schema/method signature changes are unavoidable in a real
production system.

so once I setup an avro RPC server, what is the best strategy to
handle such cases?


one way I thought about is  to put all the params of my RPC into a
single record , and encode that record into
a byte array, and the nominal signature of the RPC is just MyMethod(
byteArray param, int version );

on the receiving side, I lookup the schema from the "version" field,
and parse out the record using expected versions and
incoming version. the incoming version can be and old schema, so we
achieve backward-compatibility.

of course all the above can be built into RPC directly, so that we do
not have to do this manual encoding and decoding.
but there is one small problem left: if the old schema and new schema
both contain a record field, but the record field schema also
changed, currently it can not resolve. but I think this can be modified




the other way I guess is, when you update your schema, and publish new
API, you keep the old server running, and setup the new server
on a different port, or host; then you have to tell all your users to
migrate to the new server.


I'm sure you have encountered the schema changes problem in
production, how do you handle this?

thanks
Yang

Re: how to handle schema changes in RPC? (RPC version support)

[Doug Cutting <cu...@apache.org>]

I believe this is already supported by Avro, as described in my previous
message.

Doug

On 05/25/2011 08:15 PM, Yang wrote:
> generally schema/method signature changes are unavoidable in a real
> production system.
> 
> so once I setup an avro RPC server, what is the best strategy to
> handle such cases?
> 
> 
> one way I thought about is  to put all the params of my RPC into a
> single record , and encode that record into
> a byte array, and the nominal signature of the RPC is just MyMethod(
> byteArray param, int version );
> 
> on the receiving side, I lookup the schema from the "version" field,
> and parse out the record using expected versions and
> incoming version. the incoming version can be and old schema, so we
> achieve backward-compatibility.
> 
> of course all the above can be built into RPC directly, so that we do
> not have to do this manual encoding and decoding.
> but there is one small problem left: if the old schema and new schema
> both contain a record field, but the record field schema also
> changed, currently it can not resolve. but I think this can be modified
> 
> 
> 
> 
> the other way I guess is, when you update your schema, and publish new
> API, you keep the old server running, and setup the new server
> on a different port, or host; then you have to tell all your users to
> migrate to the new server.
> 
> 
> I'm sure you have encountered the schema changes problem in
> production, how do you handle this?
> 
> thanks
> Yang

Re: how to handle schema changes in RPC? (RPC version support)

[Scott Carey <sc...@richrelevance.com>]

My understanding is that you are supposed to be able to migrate schemas
using Avro RPC.  I am not familiar enough with the API to be 100% certain
or show you how, however.

Generally speaking, configuring a ResolvingDecoder on the reader side is
all that is required by Avro in java to handle schema migration.

On 5/25/11 11:15 AM, "Yang" <te...@gmail.com> wrote:

>generally schema/method signature changes are unavoidable in a real
>production system.
>
>so once I setup an avro RPC server, what is the best strategy to
>handle such cases?
>
>
>one way I thought about is  to put all the params of my RPC into a
>single record , and encode that record into
>a byte array, and the nominal signature of the RPC is just MyMethod(
>byteArray param, int version );
>
>on the receiving side, I lookup the schema from the "version" field,
>and parse out the record using expected versions and
>incoming version. the incoming version can be and old schema, so we
>achieve backward-compatibility.
>
>of course all the above can be built into RPC directly, so that we do
>not have to do this manual encoding and decoding.
>but there is one small problem left: if the old schema and new schema
>both contain a record field, but the record field schema also
>changed, currently it can not resolve. but I think this can be modified
>
>
>
>
>the other way I guess is, when you update your schema, and publish new
>API, you keep the old server running, and setup the new server
>on a different port, or host; then you have to tell all your users to
>migrate to the new server.
>
>
>I'm sure you have encountered the schema changes problem in
>production, how do you handle this?
>
>thanks
>Yang