Passively Converting Null Map to be valid

[Micah Whitacre <mk...@gmail.com>]

We've recently upgraded our Avro dependency to 1.7.7 from 1.7.5.  The avdl
for Avro used to look like this:

record Event{

         long creationTime;
         /**
         * The optional payload of the event.
         */
         union{null, bytes} value = null;
         /**
         * Optional properties of the event.
         */
         map<string> properties = null;
}

When we upgraded we started seeing warnings like this:

[WARNING] Avro: Invalid default for field properties: null not a
{"type":"map","values":"string"}

So we converted the file to be this:

record Event{

         long creationTime;
         /**
         * The optional payload of the event.
         */
         union{null, bytes} value = null;
         /**
         * Optional properties of the event.
         */
         map<string> properties = {};
}

We also had the need to add a new field to the record and thought we could
do so passively like so:

record Event{

         long creationTime;
         /**
         * The optional payload of the event.
         */
         union{null, bytes} value = null;
         /**
         * Optional properties of the event.
         */
         map<string> properties = {};
         /**
         * Type of operation
         */
         union{null, string} operation = null;
}

However when we then read data that was written with the very first schema
we get an EOFException.

Caused by: java.io.EOFException
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)

We are reading with a BufferedBinaryDecoder and using the new schema as
both the written and reader schema because the written schema is not
preserved with the payload so it is not easy to retrieve.

My questions are:
1. Is the change we made to add a new defaulted union truly non-passive?
2. Is there a workaround so I can continue to evolve my schema?

Thanks for the help,
Micah

Re: Avro and serializable interface

[Doug Cutting <cu...@apache.org>]

On Thu, Aug 28, 2014 at 4:22 AM, Casadio Phaedra
<Ph...@datamanagementpa.it> wrote:
> Now the problem is that avro beans does not extend serializable interface.
>
> How can i solve the problem? if it’s possible…

There is a request to change this.  I will upload a patch there soon.

https://issues.apache.org/jira/browse/AVRO-1502

> Also, trough avro tools in command line version, how can i pass
> fieldvisibility param ?

Please file a request to add this in Jira.

Thanks,

Doug

Avro and serializable interface

[Casadio Phaedra <Ph...@datamanagementpa.it>]

Hi, i’m in the need to pass an avro generated bean trough idl trough an ejb.

Now the problem is that avro beans does not extend serializable interface.

How can i solve the problem? if it’s possible…

Also, trough avro tools in command line version, how can i pass fieldvisibility param ?

Thanks Phaedra.

Re: Passively Converting Null Map to be valid

[Micah Whitacre <mk...@gmail.com>]

Thanks for the help.  In my situation I'm reading Avro payloads out of
Kafka so I can't use the Avro data file to detect the written schema.  I'll
look into tracking the written schema in an alternate manner.




On Wed, Aug 27, 2014 at 5:50 PM, Doug Cutting <cu...@apache.org> wrote:

> On Wed, Aug 27, 2014 at 11:08 AM, Micah Whitacre <mk...@gmail.com>
> wrote:
> > We are reading with a BufferedBinaryDecoder and using the new schema as
> both
> > the written and reader schema because the written schema is not preserved
> > with the payload so it is not easy to retrieve.
> >
> > My questions are:
> > 1. Is the change we made to add a new defaulted union truly non-passive?
>
> Adding the new union field changes the format for how instances are
> written, even those that have a null value for this new field.  (A
> union is written as an int indicating the selected schema, then an
> instance of that schema.)
>
> > 2. Is there a workaround so I can continue to evolve my schema?
>
> To permit schema evolution, you must pass the schema used to write
> when reading.  Avro's data file format and RPC take care of this.  If
> you're writing Avro data to some other container then you need to do
> this yourself.
>
> Doug
>

Re: Passively Converting Null Map to be valid

[Doug Cutting <cu...@apache.org>]

On Wed, Aug 27, 2014 at 11:08 AM, Micah Whitacre <mk...@gmail.com> wrote:
> We are reading with a BufferedBinaryDecoder and using the new schema as both
> the written and reader schema because the written schema is not preserved
> with the payload so it is not easy to retrieve.
>
> My questions are:
> 1. Is the change we made to add a new defaulted union truly non-passive?

Adding the new union field changes the format for how instances are
written, even those that have a null value for this new field.  (A
union is written as an int indicating the selected schema, then an
instance of that schema.)

> 2. Is there a workaround so I can continue to evolve my schema?

To permit schema evolution, you must pass the schema used to write
when reading.  Avro's data file format and RPC take care of this.  If
you're writing Avro data to some other container then you need to do
this yourself.

Doug