You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Markus Weimer <we...@yahoo-inc.com> on 2010/06/08 20:10:40 UTC
How to store a rather large double[]
Hi,
I have a schema to store linear models in machine learning whose
relevant subpart looks like this:
{
"type": "record",
"name": "LinearModel",
"fields": [{"name": "weights", "type":{"type":"array",
"items":"double"}}]
}
I understand that the actual serialized form of this should be rather
efficient. What worries me is how the Java (specific) API for the
weights plays out:
public class LinearModel{...
public GenericArray<Double> weights;
...}
This means that I have to wrap each and every double in my double[]
into a Double object and add it to the GenericArray, right?
The trouble is that the double[] I intend to store may very well be
choosen in size to max out the available memory of the machine, so I
don't really have room for a more-than-lifesize copy of the data.
Is there a way to "stream" the doubles into the output without holding
a copy in memory? Or is there another way to encode a double[] in a
schema?
Thanks for any pointers,
Markus
Re: How to store a rather large double[]
Posted by Doug Cutting <cu...@apache.org>.
On 06/08/2010 11:10 AM, Markus Weimer wrote:
> Is there a way to "stream" the doubles into the output without holding a
> copy in memory? Or is there another way to encode a double[] in a schema?
Avro arrays and maps are written in a blocked representation, so the
binary encoding does support arbitrarily large arrays. But Java's
specific API does not currently take advantage of this.
The BlockingBinaryEncoder will break large arrays into blocks on write.
Note that it's clever, since arrays may contain nested objects and
arrays, yet BlockingBinaryEncoder only starts a new block when the
specified buffer size is exceeded. The assumption is simply that no
primitive leaf value exceed the buffer size.
BlockingBinaryEncoder can be used with ValidatingEncoder and
ValidatingDecoder to safely write code that streams instances of a
schema. For example, for your schema:
{"type": "record", "name": "LinearModel", "fields": [
{"name": "weights", "type": {"type":"array", "items":"double"}}
]}
You could write instances with something like:
public writeLinearModel(Encoder out,
Iteratable<List<Double>> buffers) {
out.writeArrayStart();
for (List<Double> buffer : buffers) {
out.setItemCount(buffer.size());
for (double d : buffer)
out.writeDouble(d);
}
out.writeArrayEnd();
}
This would re-buffer, writing a block of doubles only when the
BlockingBinaryEncoder's buffer is filled. A ValidatingDecoder could
ensure that the sequence of calls conforms to the declared schema. One
could structure the control flow alternately, if Iterable<List<Double>>
is not the natural way in which doubles are produced. So for example,
one could instead do something like:
public writeLinearModel(Encoder out, Iteratable<Double> dubs) {
out.writeArrayStart();
for (d : dubs) {
out.setItemCount(1);
out.writeDouble(d);
}
out.writeArrayEnd();
}
And still rely on BlockingBinaryEncoder to only generate blocks when the
buffer's filled, e.g., every 64kB. Note that there are no per-record
encoder/decoder calls, only per-value. The validator infers the record
from the other calls.
Similarly, one could write a reader something like:
public Iterable<Double> readLinearModel(final Decoder in);
See
http://avro.apache.org/docs/current/api/java/org/apache/avro/io/Decoder.html#readArrayStart()
for the calls this should make. ValidatingDecoder could ensure that
calls conform to the schema written.
These classes were implemented precisely to support this use case.
Doug