You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@avro.apache.org by Sripad Sriram <sr...@path.com> on 2013/05/13 23:58:52 UTC

ArrayIndexOutOfBoundsException in Symbol.getSymbol in map reduce job

Hi all,

A java hadoop job that's previously executed without issue began erroring
with the following stack trace - have any of you seen this before?

java.lang.ArrayIndexOutOfBoundsException: 14
        at
org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
        at
org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
        at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
        at
org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
        at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
        at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
        at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
        at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
        at
org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:83)
        at
org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:65)
        at
org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1262)
        at
org.apache.hadoop.mapred.Task$ValuesIterator.nextKey(Task.java:1233)
        at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:533)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

Re: ArrayIndexOutOfBoundsException in Symbol.getSymbol in map reduce job

Posted by Sripad Sriram <sr...@path.com>.

I'm using Avro 1.7.4, and the error is coming between a map step and reduce
step. I'm unsure how an Avro record could get corrupted between a mapper
and reducer, but it has, how would I be able to handle that exception?


On Mon, May 13, 2013 at 3:40 PM, Harsh J <ha...@cloudera.com> wrote:

> Its difficult to tell what the error means without context and other
> info (such as version). If I had to guess, I think there may be a
> corruption on the file being processed here. Does running the file
> through avro-tools' tojson sub-command end up in a successful read?
>
> On Tue, May 14, 2013 at 3:28 AM, Sripad Sriram <sr...@path.com> wrote:
> > Hi all,
> >
> > A java hadoop job that's previously executed without issue began erroring
> > with the following stack trace - have any of you seen this before?
> >
> > java.lang.ArrayIndexOutOfBoundsException: 14
> >         at
> > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
> >         at
> > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
> >         at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> >         at
> > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
> >         at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> >         at
> >
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> >         at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> >         at
> >
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> >         at
> >
> org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:83)
> >         at
> >
> org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:65)
> >         at
> > org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1262)
> >         at
> > org.apache.hadoop.mapred.Task$ValuesIterator.nextKey(Task.java:1233)
> >         at
> > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:533)
> >         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429)
> >         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at javax.security.auth.Subject.doAs(Subject.java:396)
> >         at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
> >         at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
>
> --
> Harsh J
>

Re: ArrayIndexOutOfBoundsException in Symbol.getSymbol in map reduce job

Posted by Harsh J <ha...@cloudera.com>.

Its difficult to tell what the error means without context and other
info (such as version). If I had to guess, I think there may be a
corruption on the file being processed here. Does running the file
through avro-tools' tojson sub-command end up in a successful read?

On Tue, May 14, 2013 at 3:28 AM, Sripad Sriram <sr...@path.com> wrote:
> Hi all,
>
> A java hadoop job that's previously executed without issue began erroring
> with the following stack trace - have any of you seen this before?
>
> java.lang.ArrayIndexOutOfBoundsException: 14
>         at
> org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
>         at
> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>         at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>         at
> org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>         at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
>         at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
>         at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
>         at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
>         at
> org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:83)
>         at
> org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:65)
>         at
> org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1262)
>         at
> org.apache.hadoop.mapred.Task$ValuesIterator.nextKey(Task.java:1233)
>         at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:533)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)



-- 
Harsh J