You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by John Smith <le...@gmail.com> on 2016/02/01 13:46:51 UTC
AvroStorage issues during write into HDFS
Dear,
I created the simple pig script that reads two avro files, merges the
two relations and stores it into the output avro file.
I tried to store output relation into avro file using:
store outputSet into 'avrostorage' using AvroStorage();
Some workaround was required because pig has problems to process
schema with :: (maybe another bug?)
Added code below the result 'avrostorage' file was generated.
outputSet = foreach outputSet generate $0 as (name:chararray) , $1 as
(customerId:chararray), $2 as (VIN:chararray) , $3 as
(Birthdate:chararray), $4 as (Mileage:chararray) ,$5 as
(Fuel_Consumption:chararray);
When I tried to store avro file with the schema definition using code below,
strange error is occurring https://bpaste.net/show/ccf0cbef06a9 (Full log).
...
10.0.1.47:8050 2016-01-29 17:24:39,600 [main] ERROR
org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce
job(s) failed!
...
STORE outputSet INTO '/avro-dest/Test-20160129-1401822'
USING org.apache.pig.piggybank.storage.avro.AvroStorage('no_schema_check',
'schema', '....')
Sample data and pig script:
https://drive.google.com/file/d/0B6RZ_9vVuTEcd01aWm9zczNUUWc/view
I think these might be two important issues, could you please investigate?
Was also reported at: https://issues.apache.org/jira/browse/PIG-4793
Thank you