You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by "kulkarni.swarnim@gmail.com" <ku...@gmail.com> on 2013/10/16 00:43:34 UTC

GenericRecord and passivity

Do we know if a GenericRecord is robust to schema evolution? I am currently
seeing cases where I get an exception like the following if I try to
deserialize an older record with a newer schema.

 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 8

at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)

at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)

at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)

at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)

at org.apache.avro.generic.GenericDatumReader.read(
GenericDatumReader.java:152)

at org.apache.avro.generic.GenericDatumReader.readRecord(
GenericDatumReader.java:177)

at org.apache.avro.generic.GenericDatumReader.read(
GenericDatumReader.java:148)

at org.apache.avro.generic.GenericDatumReader.read(
GenericDatumReader.java:139)


The newer schema just has a few more fields added to it and no names from
the older schema were changed.

Is this a known issue? Should a SpecificRecord be always considered when
passivity is important?


Thanks,

-- 
Swarnim

Re: GenericRecord and passivity

Posted by Doug Cutting <cu...@apache.org>.
On Tue, Oct 15, 2013 at 4:49 PM, Eric Wasserman <ew...@247-inc.com> wrote:
> Change this line:
> DatumReader<GenericRecord> reader = new
> GenericDatumReader<GenericRecord>(schema_11);
>
> to this:
> DatumReader<GenericRecord> reader = new
> GenericDatumReader<GenericRecord>(schema_10,  schema_11);

Yes, that is the fix.

Or, better yet, use DataFileWriter & DataFileReader, which handle this for you.

Doug

Re: GenericRecord and passivity

Posted by Eric Wasserman <ew...@247-inc.com>.
Change this line:
DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema_11);

to this:
DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema_10,  schema_11);


On Oct 15, 2013, at 4:01 PM, kulkarni.swarnim@gmail.com<ma...@gmail.com> wrote:

Old Schema: http://pastebin.com/x5CtmKWK
New Schema: http://pastebin.com/c8ZSKMaG
Code to deserialize: http://pastebin.com/RbBWm1VN


On Tue, Oct 15, 2013 at 5:46 PM, Doug Cutting <cu...@apache.org>> wrote:
GenericRecord should work well in this context.  Can you provide a
complete example that fails?

Doug

On Tue, Oct 15, 2013 at 3:43 PM, kulkarni.swarnim@gmail.com<ma...@gmail.com>
<ku...@gmail.com>> wrote:
> Do we know if a GenericRecord is robust to schema evolution? I am currently
> seeing cases where I get an exception like the following if I try to
> deserialize an older record with a newer schema.
>
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 8
>
> at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
>
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
>
>
> The newer schema just has a few more fields added to it and no names from
> the older schema were changed.
>
> Is this a known issue? Should a SpecificRecord be always considered when
> passivity is important?
>
>
> Thanks,
>
>
> --
> Swarnim



--
Swarnim

Re: GenericRecord and passivity

Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.
Old Schema: http://pastebin.com/x5CtmKWK
New Schema: http://pastebin.com/c8ZSKMaG
Code to deserialize: http://pastebin.com/RbBWm1VN


On Tue, Oct 15, 2013 at 5:46 PM, Doug Cutting <cu...@apache.org> wrote:

> GenericRecord should work well in this context.  Can you provide a
> complete example that fails?
>
> Doug
>
> On Tue, Oct 15, 2013 at 3:43 PM, kulkarni.swarnim@gmail.com
> <ku...@gmail.com> wrote:
> > Do we know if a GenericRecord is robust to schema evolution? I am
> currently
> > seeing cases where I get an exception like the following if I try to
> > deserialize an older record with a newer schema.
> >
> > Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 8
> >
> > at
> org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
> >
> > at
> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
> >
> > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> >
> > at
> org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
> >
> > at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
> >
> > at
> >
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
> >
> > at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
> >
> > at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
> >
> >
> > The newer schema just has a few more fields added to it and no names from
> > the older schema were changed.
> >
> > Is this a known issue? Should a SpecificRecord be always considered when
> > passivity is important?
> >
> >
> > Thanks,
> >
> >
> > --
> > Swarnim
>



-- 
Swarnim

Re: GenericRecord and passivity

Posted by Doug Cutting <cu...@apache.org>.
GenericRecord should work well in this context.  Can you provide a
complete example that fails?

Doug

On Tue, Oct 15, 2013 at 3:43 PM, kulkarni.swarnim@gmail.com
<ku...@gmail.com> wrote:
> Do we know if a GenericRecord is robust to schema evolution? I am currently
> seeing cases where I get an exception like the following if I try to
> deserialize an older record with a newer schema.
>
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 8
>
> at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
>
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
>
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
>
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
>
>
> The newer schema just has a few more fields added to it and no names from
> the older schema were changed.
>
> Is this a known issue? Should a SpecificRecord be always considered when
> passivity is important?
>
>
> Thanks,
>
>
> --
> Swarnim