Re: Decoding a single field in Avro data

[Doug Cutting <cu...@apache.org>]

Steinar Knutsen wrote:
> Avro supports skip information, 
> but it
> is somewhat inefficient to skip across a block of an array, a record or a
> map, if any of these contain a variable length object. The headers only
> contain the number of objects contained, not the length in bytes.

Arrays and maps can optionally encode their length in bytes.  If the 
item count is a negative number, then -count is the actual count and 
immediately following the count is the size in bytes.  Java's 
BlockingBinaryEncoder implements this.  All decoders must implement it.

http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/io/BlockingBinaryEncoder.html

It does not emit a size for every array and map, but only for arrays and 
maps whose contained size exceeds a threshold, so the overhead of adding 
the size is limited.  It also splits arrays and maps that are larger 
than can be buffered as a whole into a sequence of blocks.

Doug

Re: Decoding a single field in Avro data

[Steinar Knutsen <st...@yahoo-inc.com>]

On Tue, 1 Dec 2009, Doug Cutting wrote:

> Arrays and maps can optionally encode their length in bytes.  If the
> item count is a negative number, then -count is the actual count and
> immediately following the count is the size in bytes.  Java's
> BlockingBinaryEncoder implements this.  All decoders must implement it.
>
> http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/io/BlockingBinaryEncoder.html
>
> It does not emit a size for every array and map, but only for arrays and
> maps whose contained size exceeds a threshold, so the overhead of adding
> the size is limited.  It also splits arrays and maps that are larger
> than can be buffered as a whole into a sequence of blocks.

Interesting, thanks for the reference. Sounds like pretty much what I was
looking for.
-- 
Steinar Knutsen <st...@yahoo-inc.com>
Yahoo! Technologies Norway AS