Compressing values before inserting them

[Jean-Daniel Cryans <jd...@apache.org>]

Hi users,

I just want to share a useful tip when storing very fat values into
HBase, we were able to get some of our MR jobs an order of magnitude
faster by simply using Java's Deflater and then passing the byte[] to
Put (and the equivalent when retrieving the values with Inflator). We
also use LZO compression on top of that.

I created this jira in order to ingrate that into HBase's client layer
as a configuration option for families:
https://issues.apache.org/jira/browse/HBASE-3732

Hope that can be useful to someone,

J-D

Re: Compressing values before inserting them

[Ferdy Galema <fe...@kalooga.com>]

Thanks. This seems very useful. Just to add, in terms of 
compression/decompression speed, we're having very good performance with 
the lzf codec. It is Apache licensed and pure Java code with no external 
dependancies. See https://github.com/ning/compress/

Ferdy

On 04/05/2011 12:55 AM, Jean-Daniel Cryans wrote:
> Hi users,
>
> I just want to share a useful tip when storing very fat values into
> HBase, we were able to get some of our MR jobs an order of magnitude
> faster by simply using Java's Deflater and then passing the byte[] to
> Put (and the equivalent when retrieving the values with Inflator). We
> also use LZO compression on top of that.
>
> I created this jira in order to ingrate that into HBase's client layer
> as a configuration option for families:
> https://issues.apache.org/jira/browse/HBASE-3732
>
> Hope that can be useful to someone,
>
> J-D