Kafka Avro Based Consumer

[Murtaza Doctor <mu...@richrelevance.com>]

Hello Folks,

We are trying to write an Avro based HDFS Consumer which is capable of reading offsets from zookeeper. Currently we do have a producer which writes avro messages. We are trying to utilize the avro map-reduce API and running into challenges. Additionally there is a big schema management piece here as you can imagine. Ideally there should a schema service which could be utilized by the producer and consumer.

 Any thoughts or ideas on the avro based consumer will be greatly appreciated. If there is sample avro based consumer code or anything else we could look at as well to debug the issue. Finally there were talks around LinkedIn open sourcing the avro schema management piece has this already happened or in progress?

Exception Stacktrace when we run the Map Only Job (Just for context)

Additional Context:
org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: in com.rr.avro.ViewEvent in union null of union in field viewGuid of com.rr.avro.ViewEvent
    at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:261)
    at org.apache.avro.mapred.AvroOutputFormat$1.write(AvroOutputFormat.java:122)
    at org.apache.avro.mapred.AvroOutputFormat$1.write(AvroOutputFormat.java:119)
    at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.collect(MapTask.java:706)
    at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:499)
    at com.rr.hadoop.consumer.HadoopConsumer$KafkaAvroMapper.map(HadoopConsumer.java:92)
    at com.rr.hadoop.consumer.HadoopConsumer$KafkaAvroMapper.map(HadoopConsumer.java:68)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessC
attempt_201207041152_56118_m_000000_0: KafkaContext

Thanks,
murtaza