Re: [SPAM] Re: c++ access hbase using thrift performance tuning

[LEI Xiaofeng <le...@ihep.ac.cn>]

Hi, Stack

I have not test the performance in big cluster, yet. I just test it in two node using my c++ project. One node is holding HMaster and Zookeeper server, the other is holding HRegionServer. My scan performance is about 5293 records per second. When I get those data from a local file, the performance is 208757 records per second. PS. the data are stored as a tree in the file. 

I have meant to get a better read performance through hbase, but actually I get an opposite result. Could you give me any advices to deal with it? Maybe some tuning tips or usage tips of HBase with  c++.


Thanks,
Xiaofeng  


> -----原始邮件-----
> 发件人: Stack <st...@duboce.net>
> 发送时间: 2014年1月11日 星期六
> 收件人: Hbase-User <us...@hbase.apache.org>
> 抄送: 
> 主题: [SPAM] Re: c++ access hbase using thrift performance tuning
> 
> On Thu, Jan 9, 2014 at 6:50 PM, LEI Xiaofeng <le...@ihep.ac.cn> wrote:
> 
> > Hi,
> > I have some strong reasons to make me use c++ access hbase. But the
> > reading performance is poor. I tried to tune hbase.client.scanner.caching,
> > hbase.regionserver.global.memstore.upperLimit and BLOCKSIZE and so on. But
> > these all made no obvious difference.
> >
> > Does anyone have any suggetion that could help me to promote the
> > performance?
> >
> 
> What are you seeing?  What is your setup like?  What do you need?  How does
> what you see compare to native java client?
> Thanks,
> St.Ack

Re: [SPAM] Re: c++ access hbase using thrift performance tuning

[Stack <st...@duboce.net>]

On Fri, Jan 10, 2014 at 7:55 PM, LEI Xiaofeng <le...@ihep.ac.cn> wrote:

> Hi, Stack
>
> I have not test the performance in big cluster, yet. I just test it in two
> node using my c++ project. One node is holding HMaster and Zookeeper
> server, the other is holding HRegionServer. My scan performance is about
> 5293 records per second. When I get those data from a local file, the
> performance is 208757 records per second. PS. the data are stored as a tree
> in the file.
>
>
Sounds like you are comparing different things.  Yes, going to the FS
direct versus going via the HBase API with its extra tiers handling its
'model' will be faster (see recent notes on this list on 'performance' for
some more explanation on why).

Would suggest you look at the reference guide to see how HBase does layout
in the FS and compare it to your file layout then look at the performance
section.  Get the java client running fast.  Then compare your c++ client
to the java result.  If not fast enough, come back to the list with more
detail on your setup and your data formats.

Thanks,

St.Ack