You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Weishung Chung <we...@gmail.com> on 2011/03/23 17:52:46 UTC
google snappy
Hey my fellow hadoop/hbase developers,
I just came across this google compression/decompression package yesterday,
could we make a good use of this compression scheme in hadoop? It's written
in C++ though.
http://code.google.com/p/snappy/
<http://code.google.com/p/snappy/>I haven't looked close into this snappy
package yet but i would love to know about the differences compared to LZO.
Thank you,
Wei Shung
Re: google snappy
Posted by Weishung Chung <we...@gmail.com>.
Great to know that hadoop/hbase are integrating it :D
On Wed, Mar 23, 2011 at 12:11 PM, Tim Wintle <ti...@teamrubber.com>wrote:
> On Wed, 2011-03-23 at 10:03 -0700, Jean-Daniel Cryans wrote:
> > Somebody obviously needs to publish some benchmarks, but knowing
> > Snappy's origin I can believe that claim.
>
> There were some benchmarks in the original Bigtable presentation
>
> Results from compressing bigtable blocks:
>
> Algorithm % remaining Encoding Decoding
> Gzip 13.4% 21MB/s 118MB/s
> LZO 20.5% 135MB/s 410MB/s
> Zippy 22.2% 172MB/s 409MB/s
>
> (Zippy is apparently what's now been renamed snappy)
>
> Tim Wintle
>
>
Re: google snappy
Posted by Weishung Chung <we...@gmail.com>.
Great to know that hadoop/hbase are integrating it :D
On Wed, Mar 23, 2011 at 12:11 PM, Tim Wintle <ti...@teamrubber.com>wrote:
> On Wed, 2011-03-23 at 10:03 -0700, Jean-Daniel Cryans wrote:
> > Somebody obviously needs to publish some benchmarks, but knowing
> > Snappy's origin I can believe that claim.
>
> There were some benchmarks in the original Bigtable presentation
>
> Results from compressing bigtable blocks:
>
> Algorithm % remaining Encoding Decoding
> Gzip 13.4% 21MB/s 118MB/s
> LZO 20.5% 135MB/s 410MB/s
> Zippy 22.2% 172MB/s 409MB/s
>
> (Zippy is apparently what's now been renamed snappy)
>
> Tim Wintle
>
>
Re: google snappy
Posted by Tim Wintle <ti...@teamrubber.com>.
On Wed, 2011-03-23 at 10:03 -0700, Jean-Daniel Cryans wrote:
> Somebody obviously needs to publish some benchmarks, but knowing
> Snappy's origin I can believe that claim.
There were some benchmarks in the original Bigtable presentation
Results from compressing bigtable blocks:
Algorithm % remaining Encoding Decoding
Gzip 13.4% 21MB/s 118MB/s
LZO 20.5% 135MB/s 410MB/s
Zippy 22.2% 172MB/s 409MB/s
(Zippy is apparently what's now been renamed snappy)
Tim Wintle
Re: google snappy
Posted by Tim Wintle <ti...@teamrubber.com>.
On Wed, 2011-03-23 at 10:03 -0700, Jean-Daniel Cryans wrote:
> Somebody obviously needs to publish some benchmarks, but knowing
> Snappy's origin I can believe that claim.
There were some benchmarks in the original Bigtable presentation
Results from compressing bigtable blocks:
Algorithm % remaining Encoding Decoding
Gzip 13.4% 21MB/s 118MB/s
LZO 20.5% 135MB/s 410MB/s
Zippy 22.2% 172MB/s 409MB/s
(Zippy is apparently what's now been renamed snappy)
Tim Wintle
Re: google snappy
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(Please don't cross-post like that, it only adds confusion. I put
everything in bcc and posted to general instead)
Their README says the following:
Snappy usually is faster than algorithms in the same class (e.g. LZO,
LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression
ratios.
Somebody obviously needs to publish some benchmarks, but knowing
Snappy's origin I can believe that claim.
Relevant jiras:
HADOOP-7206 Integrate Snappy compression
HBASE-3691 Add compressor support for 'snappy', google's compressor
J-D
On Wed, Mar 23, 2011 at 9:52 AM, Weishung Chung <we...@gmail.com> wrote:
> Hey my fellow hadoop/hbase developers,
>
> I just came across this google compression/decompression package yesterday,
> could we make a good use of this compression scheme in hadoop? It's written
> in C++ though.
>
> http://code.google.com/p/snappy/
>
> <http://code.google.com/p/snappy/>I haven't looked close into this snappy
> package yet but i would love to know about the differences compared to LZO.
>
> Thank you,
> Wei Shung
>
Re: google snappy
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(Please don't cross-post like that, it only adds confusion. I put
everything in bcc and posted to general instead)
Their README says the following:
Snappy usually is faster than algorithms in the same class (e.g. LZO,
LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression
ratios.
Somebody obviously needs to publish some benchmarks, but knowing
Snappy's origin I can believe that claim.
Relevant jiras:
HADOOP-7206 Integrate Snappy compression
HBASE-3691 Add compressor support for 'snappy', google's compressor
J-D
On Wed, Mar 23, 2011 at 9:52 AM, Weishung Chung <we...@gmail.com> wrote:
> Hey my fellow hadoop/hbase developers,
>
> I just came across this google compression/decompression package yesterday,
> could we make a good use of this compression scheme in hadoop? It's written
> in C++ though.
>
> http://code.google.com/p/snappy/
>
> <http://code.google.com/p/snappy/>I haven't looked close into this snappy
> package yet but i would love to know about the differences compared to LZO.
>
> Thank you,
> Wei Shung
>
Re: google snappy
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(Please don't cross-post like that, it only adds confusion. I put
everything in bcc and posted to general instead)
Their README says the following:
Snappy usually is faster than algorithms in the same class (e.g. LZO,
LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression
ratios.
Somebody obviously needs to publish some benchmarks, but knowing
Snappy's origin I can believe that claim.
Relevant jiras:
HADOOP-7206 Integrate Snappy compression
HBASE-3691 Add compressor support for 'snappy', google's compressor
J-D
On Wed, Mar 23, 2011 at 9:52 AM, Weishung Chung <we...@gmail.com> wrote:
> Hey my fellow hadoop/hbase developers,
>
> I just came across this google compression/decompression package yesterday,
> could we make a good use of this compression scheme in hadoop? It's written
> in C++ though.
>
> http://code.google.com/p/snappy/
>
> <http://code.google.com/p/snappy/>I haven't looked close into this snappy
> package yet but i would love to know about the differences compared to LZO.
>
> Thank you,
> Wei Shung
>
Re: google snappy
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(Please don't cross-post like that, it only adds confusion. I put
everything in bcc and posted to general instead)
Their README says the following:
Snappy usually is faster than algorithms in the same class (e.g. LZO,
LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression
ratios.
Somebody obviously needs to publish some benchmarks, but knowing
Snappy's origin I can believe that claim.
Relevant jiras:
HADOOP-7206 Integrate Snappy compression
HBASE-3691 Add compressor support for 'snappy', google's compressor
J-D
On Wed, Mar 23, 2011 at 9:52 AM, Weishung Chung <we...@gmail.com> wrote:
> Hey my fellow hadoop/hbase developers,
>
> I just came across this google compression/decompression package yesterday,
> could we make a good use of this compression scheme in hadoop? It's written
> in C++ though.
>
> http://code.google.com/p/snappy/
>
> <http://code.google.com/p/snappy/>I haven't looked close into this snappy
> package yet but i would love to know about the differences compared to LZO.
>
> Thank you,
> Wei Shung
>