You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Максим Стаценко <m....@corp.mail.ru> on 2016/08/24 15:38:00 UTC

Problems with writing avro using map reduce.

Hello!
Thank you for your software! I have one problem with my attempts to write avrofile using mapreduce. Can you help me, please. I asked my friends and stackoverflow, but nobody can find an answer.
On the input I have file with data for some hive tables. And on the output of the MapReduce I want to have some groups of avro files with tables schemes for each table. I'm writing data to Generic record and after that I'm trying to write it with AvroMultipleOutputs.

But I get a very strange error from reducer part :
>16/08/23 16:08:08 INFO mapreduce.Job: Task Id : attempt_1471450637179_19439_r_000000_0, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 16/08/23 16:08:15 INFO mapreduce.Job: Task Id : attempt_1471450637179_19439_r_000000_1, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 16/08/23 16:08:23 INFO mapreduce.Job: Task Id : attempt_1471450637179_19439_r_000000_2, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
I simplified my code in attach

My input file:
1234567
12345566
0987654321
 
I add link to stackoverflow, because there is a good highlighting in it.
http://stackoverflow.com/questions/39102348/writing-genericrecord-to-multiple-avro-output  

Thank you!
Max Statsenko
----------------------------------------------------------------------