Reducer Failing with java.io.IOException: Invalid int encoding

[Akshay Aggarwal <ak...@flipkart.com>]

Hey,

I need help debugging the following exception, I tried googling but all the
questions are still open. The code had been running in prod since one
month, and this happened.

2016-08-25 09:21:20,130 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.lang.RuntimeException: next value iterator failed
	at org.apache.hadoop.mapreduce.task.ReduceContextImpl$ValueIterator.next(ReduceContextImpl.java:242)
	at com.flipkart.dpg.unified.session.reducer.SessionReducer.reduce(SessionReducer.java:54)
	at com.flipkart.dpg.unified.session.reducer.SessionReducer.reduce(SessionReducer.java:24)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: Invalid int encoding
	at org.apache.avro.io.DirectBinaryDecoder.readInt(DirectBinaryDecoder.java:113)
	at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:259)
	at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:430)
	at org.apache.avro.reflect.ReflectDatumReader.readString(ReflectDatumReader.java:229)
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)

        ...

        ...
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
	at org.apache.avro.hadoop.io.AvroDeserializer.deserialize(AvroDeserializer.java:122)
	at org.apache.avro.hadoop.io.AvroDeserializer.deserialize(AvroDeserializer.java:48)
	at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146)
	at org.apache.hadoop.mapreduce.task.ReduceContextImpl$ValueIterator.next(ReduceContextImpl.java:239)
	... 10 more

2016-08-25 09:21:20,132 INFO [main] org.apache.hadoop.mapred.Task:
Runnning cleanup for the task


Any pointers on how I can debug / fix this would be much appreciated.

Thanks,
Akshay Aggarwal