Avro cross-version serialization format compatibility

[Scott Arthur <sa...@salesforce.com>]

Hi,

I have a question about serialization compatibility across Avro versions.  I'm looking to deploy a product across my organization that uses Avro to package messages.  Existing products in the organization already have a customized Avro library a few versions old.  I've been trying to determine if, for example, a message serialized with Avro 1.3.0 will be deserializeable with version 1.7.4 if they use the same schema definition.  I found the Change Log<https://issues.apache.org/jira/browse/AVRO?report=com.atlassian.jira.plugin.system.project:changelog-panel#selectedTab=com.atlassian.jira.plugin.system.project:changelog-panel&allVersions=true> page, and it looks like the last version where there were "incompatible changes" outside the API was version 1.3.0.  However, I didn't find any resource on the Avro pages or the net that specifically went through changes to the serialization format for different Avro releases.  So, is what I'm asking possible?  Is there any resource where I can find information about breaking changes between Avro versions?

Thanks in advance,
Scott Arthur

RE: Avro cross-version serialization format compatibility

[Scott Arthur <sa...@salesforce.com>]

Fantastic. Thank you!

-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org] 
Sent: Wednesday, February 27, 2013 12:47 PM
To: user@avro.apache.org
Subject: Re: Avro cross-version serialization format compatibility

I believe there have been no incompatible changes in the base object serialization format or schema language since Avro 1.0.  There was one incompatible change to the file format, from release 1.2 to 1.3.  The
1.2 file format is still readable by post-1.2 releases, but the 1.3+ file format is not readable by pre-1.3 releases.

The format of Avro release numbers are major.minor.bugfix.  Data formats should only change in major releases.  Minor-releases can have incompatible API changes but no format changes.  Bugfix releases may include back-compatible API changes but no incompatible API changes and no format changes.  In particular, new features may be introduced in bugfix releases.

The change to the file format in 1.3 was an exception to the above policy, the only exception to date.

So, to answer your question: Yes, an object serialized with 1.3 will be readable by 1.7.4 and vice-versa.

Doug

On Wed, Feb 27, 2013 at 8:26 AM, Scott Arthur <sa...@salesforce.com> wrote:
> Hi,
>
>
>
> I have a question about serialization compatibility across Avro versions.
> I’m looking to deploy a product across my organization that uses Avro 
> to package messages.  Existing products in the organization already 
> have a customized Avro library a few versions old.  I’ve been trying 
> to determine if, for example, a message serialized with Avro 1.3.0 
> will be deserializeable with version 1.7.4 if they use the same schema definition.
> I found the Change Log page, and it looks like the last version where 
> there were “incompatible changes” outside the API was version 1.3.0.  
> However, I didn’t find any resource on the Avro pages or the net that 
> specifically went through changes to the serialization format for different Avro releases.
> So, is what I’m asking possible?  Is there any resource where I can 
> find information about breaking changes between Avro versions?
>
>
>
> Thanks in advance,
>
> Scott Arthur

Re: Avro cross-version serialization format compatibility

[Doug Cutting <cu...@apache.org>]

I believe there have been no incompatible changes in the base object
serialization format or schema language since Avro 1.0.  There was one
incompatible change to the file format, from release 1.2 to 1.3.  The
1.2 file format is still readable by post-1.2 releases, but the 1.3+
file format is not readable by pre-1.3 releases.

The format of Avro release numbers are major.minor.bugfix.  Data
formats should only change in major releases.  Minor-releases can have
incompatible API changes but no format changes.  Bugfix releases may
include back-compatible API changes but no incompatible API changes
and no format changes.  In particular, new features may be introduced
in bugfix releases.

The change to the file format in 1.3 was an exception to the above
policy, the only exception to date.

So, to answer your question: Yes, an object serialized with 1.3 will
be readable by 1.7.4 and vice-versa.

Doug

On Wed, Feb 27, 2013 at 8:26 AM, Scott Arthur <sa...@salesforce.com> wrote:
> Hi,
>
>
>
> I have a question about serialization compatibility across Avro versions.
> I’m looking to deploy a product across my organization that uses Avro to
> package messages.  Existing products in the organization already have a
> customized Avro library a few versions old.  I’ve been trying to determine
> if, for example, a message serialized with Avro 1.3.0 will be
> deserializeable with version 1.7.4 if they use the same schema definition.
> I found the Change Log page, and it looks like the last version where there
> were “incompatible changes” outside the API was version 1.3.0.  However, I
> didn’t find any resource on the Avro pages or the net that specifically went
> through changes to the serialization format for different Avro releases.
> So, is what I’m asking possible?  Is there any resource where I can find
> information about breaking changes between Avro versions?
>
>
>
> Thanks in advance,
>
> Scott Arthur