You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by xkwang bruce <br...@gmail.com> on 2012/07/13 12:28:15 UTC
hbase improve random read/write performance
hi all,
I plan to use hbase to support a online application that need
high concurrency random read . There id hundreds of gb data every day and
that will be add hbase.
Any links, articals, suggestions, ideas that will improve hbase random
read/write performance will be appreciated.
bruce
best regards!!!
Re: hbase improve random read/write performance
Posted by xkwang bruce <br...@gmail.com>.
thanks
2012/7/13 Adrien Mogenet <ad...@gmail.com>
> You might have a look at the reference guide, or this article (
> http://blog.newitfarmer.com/nosql/hbase-nosql/4385/repost-hbasearchitecture
> )
> or the Lars one (
> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html)
> which explain HBase Storage architecture.
>
> Take care of block size, block cache, and row-key design.
>
> Hope that help, even I gave very general informations.
>
> On Fri, Jul 13, 2012 at 12:28 PM, xkwang bruce <bruce.xkwang1@gmail.com
> >wrote:
>
> > hi all,
> >
> >
> > I plan to use hbase to support a online application that need
> > high concurrency random read . There id hundreds of gb data every day and
> > that will be add hbase.
> > Any links, articals, suggestions, ideas that will improve hbase random
> > read/write performance will be appreciated.
> >
> >
> > bruce
> > best regards!!!
> >
>
>
>
> --
> Adrien Mogenet
> http://www.mogenet.me
>
Re: hbase improve random read/write performance
Posted by Adrien Mogenet <ad...@gmail.com>.
You might have a look at the reference guide, or this article (
http://blog.newitfarmer.com/nosql/hbase-nosql/4385/repost-hbasearchitecture)
or the Lars one (
http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html)
which explain HBase Storage architecture.
Take care of block size, block cache, and row-key design.
Hope that help, even I gave very general informations.
On Fri, Jul 13, 2012 at 12:28 PM, xkwang bruce <br...@gmail.com>wrote:
> hi all,
>
>
> I plan to use hbase to support a online application that need
> high concurrency random read . There id hundreds of gb data every day and
> that will be add hbase.
> Any links, articals, suggestions, ideas that will improve hbase random
> read/write performance will be appreciated.
>
>
> bruce
> best regards!!!
>
--
Adrien Mogenet
http://www.mogenet.me
Re: hbase improve random read/write performance
Posted by Davey Yan <da...@gmail.com>.
In the last few days, I have just tuned a test cluster with only 3
servers(16cores/24GB memory/SATA).
Random write rate improved from few MB/s to 90MB/s.
The key factors for me are:
1. snappy compression
2. random rowkey with a prefix hash value(left half of md5, 8Bytes)
3. pre-split with org.apache.hadoop.hbase.util.RegionSplitter.HexStringSplit
4. no auto-split with hbase.hregion.max.filesize => 100GB
5. tuning hbase.regionserver.handler.count/hbase..thrift.minWorkerThreads
according to your clients' request
6. thrift is also a CPU consuming guy, you should consider having
several instances of it if your client is non-java.
7. disable writeToWAL if you feel no pain
8. other tips: http://gbif.blogspot.dk/2012/07/optimizing-writes-in-hbase.html
FYI.
On Fri, Jul 13, 2012 at 6:28 PM, xkwang bruce <br...@gmail.com> wrote:
> hi all,
>
>
> I plan to use hbase to support a online application that need
> high concurrency random read . There id hundreds of gb data every day and
> that will be add hbase.
> Any links, articals, suggestions, ideas that will improve hbase random
> read/write performance will be appreciated.
>
>
> bruce
> best regards!!!
--
Davey Yan