You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Markus Weimer <we...@yahoo-inc.com> on 2010/06/08 20:10:40 UTC

How to store a rather large double[]

Hi,

I have a schema to store linear models in machine learning whose  
relevant subpart looks like this:

{
             "type": "record",
             "name": "LinearModel",
             "fields": [{"name": "weights", "type":{"type":"array",  
"items":"double"}}]
}

I understand that the actual serialized form of this should be rather  
efficient. What worries me is how the Java (specific) API for the  
weights plays out:

public class LinearModel{...
   public GenericArray<Double> weights;
...}

This means that I have to wrap each and every double in my double[]  
into a Double object and add it to the GenericArray, right?

The trouble is that the double[] I intend to store may very well be  
choosen in size to max out the available memory of the machine, so I  
don't really have room for a more-than-lifesize copy of the data.

Is there a way to "stream" the doubles into the output without holding  
a copy in memory? Or is there another way to encode a double[] in a  
schema?

Thanks for any pointers,

Markus

Re: How to store a rather large double[]

Posted by Doug Cutting <cu...@apache.org>.
On 06/08/2010 11:10 AM, Markus Weimer wrote:
> Is there a way to "stream" the doubles into the output without holding a
> copy in memory? Or is there another way to encode a double[] in a schema?

Avro arrays and maps are written in a blocked representation, so the 
binary encoding does support arbitrarily large arrays.  But Java's 
specific API does not currently take advantage of this.

The BlockingBinaryEncoder will break large arrays into blocks on write. 
  Note that it's clever, since arrays may contain nested objects and 
arrays, yet BlockingBinaryEncoder only starts a new block when the 
specified buffer size is exceeded.  The assumption is simply that no 
primitive leaf value exceed the buffer size.

BlockingBinaryEncoder can be used with ValidatingEncoder and 
ValidatingDecoder to safely write code that streams instances of a 
schema.  For example, for your schema:

{"type": "record", "name": "LinearModel", "fields": [
    {"name": "weights", "type": {"type":"array", "items":"double"}}
]}

You could write instances with something like:

public writeLinearModel(Encoder out,
                         Iteratable<List<Double>> buffers) {
   out.writeArrayStart();
   for (List<Double> buffer : buffers) {
     out.setItemCount(buffer.size());
     for (double d : buffer)
       out.writeDouble(d);
   }
   out.writeArrayEnd();
}

This would re-buffer, writing a block of doubles only when the 
BlockingBinaryEncoder's buffer is filled.  A ValidatingDecoder could 
ensure that the sequence of calls conforms to the declared schema.  One 
could structure the control flow alternately, if Iterable<List<Double>> 
is not the natural way in which doubles are produced.  So for example, 
one could instead do something like:

public writeLinearModel(Encoder out, Iteratable<Double> dubs) {
   out.writeArrayStart();
   for (d : dubs) {
     out.setItemCount(1);
     out.writeDouble(d);
   }
   out.writeArrayEnd();
}

And still rely on BlockingBinaryEncoder to only generate blocks when the 
buffer's filled, e.g., every 64kB.  Note that there are no per-record 
encoder/decoder calls, only per-value.  The validator infers the record 
from the other calls.

Similarly, one could write a reader something like:

public Iterable<Double> readLinearModel(final Decoder in);

See 
http://avro.apache.org/docs/current/api/java/org/apache/avro/io/Decoder.html#readArrayStart() 
for the calls this should make.  ValidatingDecoder could ensure that 
calls conform to the schema written.

These classes were implemented precisely to support this use case.

Doug