You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by "Felix.徐" <yg...@gmail.com> on 2011/04/13 13:29:42 UTC

When I use GenericRecord as value in mapreduce,serilization problem occurs.

Hi,all. I try to use GenericRecord as value in mapreduce:

public void map(GenericRecord record, AvroCollector<Pair<Utf8,GenericRecord>>
collector,  Reporter reporter)
....
collector.collect(new Pair<Utf8,GenericRecord>(new Utf8(key),value));

here, object value is a GenericRecord. Then I run this job, exception comes
out:

11/04/13 02:23:05 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record
incompatible with java.lang.CharSequence
    at
org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:189)
    at
org.apache.avro.reflect.ReflectDatumWriter.writeString(ReflectDatumWriter.java:85)
    at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:77)
    at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:102)
    at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
    at
org.apache.avro.mapred.AvroSerialization$AvroWrapperSerializer.serialize(AvroSerialization.java:131)
    at
org.apache.avro.mapred.AvroSerialization$AvroWrapperSerializer.serialize(AvroSerialization.java:1)
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:900)
    at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
    at
org.apache.avro.mapred.HadoopMapper$MapCollector.collect(HadoopMapper.java:69)
    at com.ibm.crl.manager.MREngine$AggregationMapper.map(MREngine.java:73)
    at com.ibm.crl.manager.MREngine$AggregationMapper.map(MREngine.java:1)
    at org.apache.avro.mapred.HadoopMapper.map(HadoopMapper.java:80)
    at org.apache.avro.mapred.HadoopMapper.map(HadoopMapper.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

It seems that the GenericDatumWriter writes my record by using writeString()
instead of writeRecord()..

Re: When I use GenericRecord as value in mapreduce,serilization problem occurs.

Posted by Doug Cutting <cu...@apache.org>.
What output schema did you specify for the job?

Doug

On 04/13/2011 04:29 AM, Felix.徐 wrote:
> Hi,all. I try to use GenericRecord as value in mapreduce:
> 
> public void map(GenericRecord record,
> AvroCollector<Pair<Utf8,GenericRecord>> collector,  Reporter reporter)
> ....
> collector.collect(new Pair<Utf8,GenericRecord>(new Utf8(key),value));
> 
> here, object value is a GenericRecord. Then I run this job, exception
> comes out:
> 
> 11/04/13 02:23:05 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record
> incompatible with java.lang.CharSequence
>     at
> org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:189)
>     at
> org.apache.avro.reflect.ReflectDatumWriter.writeString(ReflectDatumWriter.java:85)
>     at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:77)
>     at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:102)
>     at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
>     at
> org.apache.avro.mapred.AvroSerialization$AvroWrapperSerializer.serialize(AvroSerialization.java:131)
>     at
> org.apache.avro.mapred.AvroSerialization$AvroWrapperSerializer.serialize(AvroSerialization.java:1)
>     at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:900)
>     at
> org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
>     at
> org.apache.avro.mapred.HadoopMapper$MapCollector.collect(HadoopMapper.java:69)
>     at com.ibm.crl.manager.MREngine$AggregationMapper.map(MREngine.java:73)
>     at com.ibm.crl.manager.MREngine$AggregationMapper.map(MREngine.java:1)
>     at org.apache.avro.mapred.HadoopMapper.map(HadoopMapper.java:80)
>     at org.apache.avro.mapred.HadoopMapper.map(HadoopMapper.java:1)
>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>     at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> 
> It seems that the GenericDatumWriter writes my record by using
> writeString() instead of writeRecord()..