You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by David Koch <og...@googlemail.com> on 2012/11/03 15:57:37 UTC
Scan performance on compressed column families
Hello,
Are scans faster when compression is activated? The HBase book by Lars
George seems to suggest so (p424, Section on "Compression" in chapter
"Performance Tuning").
"... compression usually will yield overall better performance, because the
overhead of the CPU performing the compression and de- compression is less
than what is required to read more data from disk."
I searched around for a bit and found this:
http://gbif.blogspot.fr/2012/02/performance-evaluation-of-hbase.html. The
author conducted a series of scan performance tests on tables of up to
200million rows and found that compression actually slowed down read
performance slightly - albeit at lower CPU load.
Thank you,
/David
Re: Scan performance on compressed column families
Posted by David Koch <og...@googlemail.com>.
Hello Oliver,
Thank you for the clarification. As Kevin also pointed out, I guess we will
just have to test compression in our environment.
Regards,
/David
On Fri, Nov 9, 2012 at 8:46 PM, Oliver Meyn (GBIF) <om...@gbif.org> wrote:
> Hi David,
>
> I wrote that blog post and I know that Lars George has much more
> experience than me with tuning HBase, especially in different environments,
> so weight our opinions accordingly. As he says, it will "usually" help,
> and the unusual cases of lower spec'd hardware (that I did those tests on)
> are where it might hurt scans, but obviously still helps with disk and
> network use. So take my post with a grain of salt, and as Kevin says, try
> it out on your data and your cluster and see what works best for you.
>
> Cheers,
> Oliver
>
> On 2012-11-03, at 3:57 PM, David Koch wrote:
>
> > Hello,
> >
> > Are scans faster when compression is activated? The HBase book by Lars
> > George seems to suggest so (p424, Section on "Compression" in chapter
> > "Performance Tuning").
> >
> > "... compression usually will yield overall better performance, because
> the
> > overhead of the CPU performing the compression and de- compression is
> less
> > than what is required to read more data from disk."
> >
> > I searched around for a bit and found this:
> > http://gbif.blogspot.fr/2012/02/performance-evaluation-of-hbase.html.
> The
> > author conducted a series of scan performance tests on tables of up to
> > 200million rows and found that compression actually slowed down read
> > performance slightly - albeit at lower CPU load.
> >
> > Thank you,
> >
> > /David
>
>
> --
> Oliver Meyn
> Software Developer
> Global Biodiversity Information Facility (GBIF)
> +45 35 32 15 12
> http://www.gbif.org
>
>
Re: Scan performance on compressed column families
Posted by "Oliver Meyn (GBIF)" <om...@gbif.org>.
Hi David,
I wrote that blog post and I know that Lars George has much more experience than me with tuning HBase, especially in different environments, so weight our opinions accordingly. As he says, it will "usually" help, and the unusual cases of lower spec'd hardware (that I did those tests on) are where it might hurt scans, but obviously still helps with disk and network use. So take my post with a grain of salt, and as Kevin says, try it out on your data and your cluster and see what works best for you.
Cheers,
Oliver
On 2012-11-03, at 3:57 PM, David Koch wrote:
> Hello,
>
> Are scans faster when compression is activated? The HBase book by Lars
> George seems to suggest so (p424, Section on "Compression" in chapter
> "Performance Tuning").
>
> "... compression usually will yield overall better performance, because the
> overhead of the CPU performing the compression and de- compression is less
> than what is required to read more data from disk."
>
> I searched around for a bit and found this:
> http://gbif.blogspot.fr/2012/02/performance-evaluation-of-hbase.html. The
> author conducted a series of scan performance tests on tables of up to
> 200million rows and found that compression actually slowed down read
> performance slightly - albeit at lower CPU load.
>
> Thank you,
>
> /David
--
Oliver Meyn
Software Developer
Global Biodiversity Information Facility (GBIF)
+45 35 32 15 12
http://www.gbif.org
Re: Scan performance on compressed column families
Posted by Kevin O'dell <ke...@cloudera.com>.
Dave,
I would recommend trying it in your environment. Like most tests you can
find other blogs that argue the performance is better:
http://blog.erdemagaoglu.com/post/4605524309/lzo-vs-snappy-vs-lzf-vs-zlib-a-comparison-of
It will depend on your environment, filesize, how well it compresses, how
taxed your disks are, how wide your cells are, etc..
On Wed, Nov 7, 2012 at 8:09 AM, David Koch <og...@googlemail.com> wrote:
> *BUMP*
>
> Sorry,
>
> /David
>
> On Sat, Nov 3, 2012 at 3:57 PM, David Koch <og...@googlemail.com> wrote:
>
> > Hello,
> >
> > Are scans faster when compression is activated? The HBase book by Lars
> > George seems to suggest so (p424, Section on "Compression" in chapter
> > "Performance Tuning").
> >
> > "... compression usually will yield overall better performance, because
> > the overhead of the CPU performing the compression and de- compression is
> > less than what is required to read more data from disk."
> >
> > I searched around for a bit and found this:
> > http://gbif.blogspot.fr/2012/02/performance-evaluation-of-hbase.html.
> The
> > author conducted a series of scan performance tests on tables of up to
> > 200million rows and found that compression actually slowed down read
> > performance slightly - albeit at lower CPU load.
> >
> > Thank you,
> >
> > /David
> >
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: Scan performance on compressed column families
Posted by David Koch <og...@googlemail.com>.
*BUMP*
Sorry,
/David
On Sat, Nov 3, 2012 at 3:57 PM, David Koch <og...@googlemail.com> wrote:
> Hello,
>
> Are scans faster when compression is activated? The HBase book by Lars
> George seems to suggest so (p424, Section on "Compression" in chapter
> "Performance Tuning").
>
> "... compression usually will yield overall better performance, because
> the overhead of the CPU performing the compression and de- compression is
> less than what is required to read more data from disk."
>
> I searched around for a bit and found this:
> http://gbif.blogspot.fr/2012/02/performance-evaluation-of-hbase.html. The
> author conducted a series of scan performance tests on tables of up to
> 200million rows and found that compression actually slowed down read
> performance slightly - albeit at lower CPU load.
>
> Thank you,
>
> /David
>