You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Jaikit Savla <ja...@yahoo.com> on 2015/03/26 23:50:21 UTC

AvroStorage taking long time to load and iterate over records

Folks,
I am noticing weird behavior where loading and iterating avro records via AvroStorage takes long time as compared to  iterating via MapReduce job.  Any known issues or any clue as to why AvroStorage would take such long time ?
Example:Schema which I am using:
{  "type": "record",  "name": "Timber",  "namespace": "com.timber.avro",  "fields": [    {      "name": "identifier",      "type": "string",      "doc": "Identifier. NonNull."    },    {      "name": "reservation",      "type": [        "null",        {          "type": "array",          "items": {            "name": "Reservation",            "type": "record",            "fields": [              {                "name": "bookingDate",                "type": "long",                "doc": "Timestamp in UTC. NonNull"              },              {                "name": "code",                "type": [                  "null",                  "string"                ],                "doc": "Code.",                "default": null              },                          ]          }        }      ],      "default": null,      "doc": "array of segment id which this urn belongs."    }  ]}
---> Pig
using Pig AvroStorage, it takes more than 30 minutes to simple iterate. I have been adding more optional fields (like code) in above Reservation record. Does that affect how I am using AvroStorage ?
register /json-simple-1.1.jarregister /piggybank.jar
records = LOAD '/data/*/one.avro'          USING org.apache.pig.piggybank.storage.avro.AvroStorage('no_schema_check')
reservation = FOREACH records {            selectHotelAtt = FOREACH reservation GENERATE bookingDate;            GENERATE FLATTEN(selectHotelAtt.bookingDate) as bookingDate;                };DUMP reservation;

--> MapReduceWhen I use MapReduce job to iterate through all the records it completes in less than 2 minutes for about million records
Mapper interface        @Override        public void map(final AvroKey<Timber> key, final NullWritable value, final Context context) throws IOException, InterruptedException {
Thanks,Jaikit