You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by David Ortiz <do...@videologygroup.com> on 2015/11/23 21:22:57 UTC
Strange ClassCastException emitting avro record
Hello,
Anyone have any idea what might cause this stack trace? I am trying to emit a descendent of SpecificRecordBase that was generated using the maven avro plugin with an avdl file. It is all simple fields, no nested records or anything. The last line in my code it references is specifically the emitter.emit() call on my record instance.
Error: org.apache.crunch.CrunchRuntimeException: java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:45) at org.apache.crunch.MapFn.process(MapFn.java:34) at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56) at org.apache.crunch.MapFn.process(MapFn.java:34) at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56) at com.videologygroup.crunch.ClickstreamWarehouse$3.process(ClickstreamWarehouse.java:528) at com.videologygroup.crunch.ClickstreamWarehouse$3.process(ClickstreamWarehouse.java:509) at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56) at org.apache.crunch.MapFn.process(MapFn.java:34) at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56) at org.apache.crunch.MapFn.process(MapFn.java:34) at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at org.apache.crunch.impl.mr.run.RTNode.processIterable(RTNode.java:113) at org.apache.crunch.impl.mr.run.CrunchReducer.reduce(CrunchReducer.java:57) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord at org.apache.avro.generic.GenericData.getField(GenericData.java:583) at org.apache.avro.generic.GenericData.getField(GenericData.java:598) at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:112) at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58) at org.apache.crunch.types.avro.SafeAvroSerialization$AvroWrapperSerializer.serialize(SafeAvroSerialization.java:128) at org.apache.crunch.types.avro.SafeAvroSerialization$AvroWrapperSerializer.serialize(SafeAvroSerialization.java:113) at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1517) at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83) at org.apache.crunch.io.CrunchOutputs$OutputState.write(CrunchOutputs.java:296) at org.apache.crunch.io.CrunchOutputs.write(CrunchOutputs.java:175) at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:41) ... 25 more
Thanks,
Dave
This email is intended only for the use of the individual(s) to whom it is addressed. If you have received this communication in error, please immediately notify the sender and delete the original email.
Re: Strange ClassCastException emitting avro record
Posted by David Ortiz <dp...@gmail.com>.
Disregard. I forgot to specify Writables.strings() as my output type when
writing to sequence file.
On Mon, Nov 23, 2015 at 3:23 PM David Ortiz <do...@videologygroup.com>
wrote:
> Hello,
>
>
>
> Anyone have any idea what might cause this stack trace? I am trying
> to emit a descendent of SpecificRecordBase that was generated using the
> maven avro plugin with an avdl file. It is all simple fields, no nested
> records or anything. The last line in my code it references is
> specifically the emitter.emit() call on my record instance.
>
>
>
> Error: org.apache.crunch.CrunchRuntimeException:
> java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
> org.apache.avro.generic.IndexedRecord at
> org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:45)
> at org.apache.crunch.MapFn.process(MapFn.java:34) at
> org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
> at org.apache.crunch.MapFn.process(MapFn.java:34) at
> org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
> at
> com.videologygroup.crunch.ClickstreamWarehouse$3.process(ClickstreamWarehouse.java:528)
> at
> com.videologygroup.crunch.ClickstreamWarehouse$3.process(ClickstreamWarehouse.java:509)
> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
> at org.apache.crunch.MapFn.process(MapFn.java:34) at
> org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
> at org.apache.crunch.MapFn.process(MapFn.java:34) at
> org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
> org.apache.crunch.impl.mr.run.RTNode.processIterable(RTNode.java:113) at
> org.apache.crunch.impl.mr.run.CrunchReducer.reduce(CrunchReducer.java:57)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
> org.apache.avro.generic.IndexedRecord at
> org.apache.avro.generic.GenericData.getField(GenericData.java:583) at
> org.apache.avro.generic.GenericData.getField(GenericData.java:598) at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:112)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at
> org.apache.crunch.types.avro.SafeAvroSerialization$AvroWrapperSerializer.serialize(SafeAvroSerialization.java:128)
> at
> org.apache.crunch.types.avro.SafeAvroSerialization$AvroWrapperSerializer.serialize(SafeAvroSerialization.java:113)
> at
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1517)
> at
> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83)
> at
> org.apache.crunch.io.CrunchOutputs$OutputState.write(CrunchOutputs.java:296)
> at org.apache.crunch.io.CrunchOutputs.write(CrunchOutputs.java:175) at
> org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:41)
> ... 25 more
>
>
>
> Thanks,
>
> Dave
> *This email is intended only for the use of the individual(s) to whom it
> is addressed. If you have received this communication in error, please
> immediately notify the sender and delete the original email.*
>