BufferedBinaryEncoder OOM from mapred

[Michael Burr <mi...@engisys.com>]

Hello,

We are starting up a project using map/reduce to produce avro files. In short, our job produces avro records which can contain very large arrays. In effect, we really can't practically predict how large some of them can get. 

When we hit one of these "very large" records, the BufferedBinaryEncoder seems to blow out the heap when calling org.apache.avro.mapred.AvroMultipleOutputs$1.collect() from a reducer (see stack trace below).

Browsing through the avro code and the Jira's, it seems that AVRO-105  could be part of the solution here, as I believe we would probably want to be able to use the BlockingBinaryEncoder (or perhaps even the DirectBinaryEncoder?? ) to be able to write these large arrays in a memory-efficient manner. 

Am I on the right track here? If so, it also seems that we would  need an additional feature to be able to configure/enable this from mapred via the  JobConf etc.. 

Since I'm as-of-yet not that familiar with the internals of avro, I would appreciate it if anyone could give me a sanity check, and/or potentially offer other suggestions as to how we may be able to work around this problem.

Thanks in advance for your help,
-Mike


Error running child : java.lang.OutOfMemoryError: Java heap space
         at java.util.Arrays.copyOf(Arrays.java:2786)
         at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
         at org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerWrite(BufferedBinaryEncoder.java:216)
         at org.apache.avro.io.BufferedBinaryEncoder.flushBuffer(BufferedBinaryEncoder.java:93)
         at org.apache.avro.io.BufferedBinaryEncoder.ensureBounds(BufferedBinaryEncoder.java:108)
         at org.apache.avro.io.BufferedBinaryEncoder.writeFixed(BufferedBinaryEncoder.java:153)
         at org.apache.avro.io.Encoder.writeFixed(Encoder.java:174)
         at org.apache.avro.io.BufferedBinaryEncoder.writeFixed(BufferedBinaryEncoder.java:164)
         at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:65)
         at org.apache.avro.generic.GenericDatumWriter.writeBytes(GenericDatumWriter.java:212)
         at org.apache.avro.reflect.ReflectDatumWriter.writeBytes(ReflectDatumWriter.java:93)
         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:77)
         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
         at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:106)
         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
         at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:131)
         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68)
         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
         at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:106)
         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
         at org.apache.avro.mapred.AvroOutputFormat$1.write(AvroOutputFormat.java:160)
         at org.apache.avro.mapred.AvroOutputFormat$1.write(AvroOutputFormat.java:157)
         at org.apache.avro.mapred.AvroMultipleOutputs$RecordWriterWithCounter.write(AvroMultipleOutputs.java:436)
         at org.apache.avro.mapred.AvroMultipleOutputs$1.collect(AvroMultipleOutputs.java:499)

>