You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Hrishikesh P <hr...@gmail.com> on 2015/08/06 17:46:11 UTC
ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord
I'm trying to write a PTable<String, Map<String, String>> to a
TextFileSourceTarget but I am running into a ClassCastException.
{code}
PTable<String, Map<String, String>> table = ...;
table.write(new
TextFileSourceTarget<>(Path.mergePaths(fs.getHomeDirectory(), path),
Avros.tableOf(Avros.strings(), Avros.maps(Avros.strings()))));
{code}
The exception is:
WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot
be cast to org.apache.avro.generic.IndexedRecord
at org.apache.crunch.types.avro.AvroTableType$IndexedRecordToPair.map(AvroTableType.java:88)
at org.apache.crunch.MapFn.process(MapFn.java:34)
at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Does someone have an idea why the exception might be occurring? The version
of avro is 1.7.6. I have checked for dependency conflicts but there doesn't
seem to be a problem with it although it looks like it.
Thanks.
Re: ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord
Posted by Gabriel Reid <ga...@gmail.com>.
Hi Hrishikesh,
This should work (or at least, I don't know of a reason why it shouldn't
work). I'm having a hard time replicating your problem though -- in all the
various setups where I've tried to replicate this, it works fine for me.
Which version of Crunch are you using? Could you put together a small (but
complete) test class that demonstrates this problem?
- Gabriel
On Thu, Aug 6, 2015 at 5:51 PM David Ortiz <dp...@gmail.com> wrote:
> Believe the output type for a TextFileOutput has to be a String, not a
> Table containing a String key with an Avro Map. With the complex object I
> think you have to write it out as an AvroFile.
>
> On Thu, Aug 6, 2015 at 11:47 AM Hrishikesh P <hr...@gmail.com>
> wrote:
>
>> I'm trying to write a PTable<String, Map<String, String>> to a
>> TextFileSourceTarget but I am running into a ClassCastException.
>>
>> {code}
>> PTable<String, Map<String, String>> table = ...;
>>
>> table.write(new
>> TextFileSourceTarget<>(Path.mergePaths(fs.getHomeDirectory(), path),
>> Avros.tableOf(Avros.strings(), Avros.maps(Avros.strings()))));
>> {code}
>>
>> The exception is:
>>
>> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord
>> at org.apache.crunch.types.avro.AvroTableType$IndexedRecordToPair.map(AvroTableType.java:88)
>> at org.apache.crunch.MapFn.process(MapFn.java:34)
>> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
>> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
>> at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>>
>>
>> Does someone have an idea why the exception might be occurring? The
>> version of avro is 1.7.6. I have checked for dependency conflicts but there
>> doesn't seem to be a problem with it although it looks like it.
>>
>> Thanks.
>>
>
Re: ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord
Posted by David Ortiz <dp...@gmail.com>.
Believe the output type for a TextFileOutput has to be a String, not a
Table containing a String key with an Avro Map. With the complex object I
think you have to write it out as an AvroFile.
On Thu, Aug 6, 2015 at 11:47 AM Hrishikesh P <hr...@gmail.com>
wrote:
> I'm trying to write a PTable<String, Map<String, String>> to a
> TextFileSourceTarget but I am running into a ClassCastException.
>
> {code}
> PTable<String, Map<String, String>> table = ...;
>
> table.write(new
> TextFileSourceTarget<>(Path.mergePaths(fs.getHomeDirectory(), path),
> Avros.tableOf(Avros.strings(), Avros.maps(Avros.strings()))));
> {code}
>
> The exception is:
>
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord
> at org.apache.crunch.types.avro.AvroTableType$IndexedRecordToPair.map(AvroTableType.java:88)
> at org.apache.crunch.MapFn.process(MapFn.java:34)
> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
> at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
> Does someone have an idea why the exception might be occurring? The
> version of avro is 1.7.6. I have checked for dependency conflicts but there
> doesn't seem to be a problem with it although it looks like it.
>
> Thanks.
>