You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by LEI Xiaofeng <le...@ihep.ac.cn> on 2014/01/07 08:06:28 UTC
hbase read performance tuning failed
Hi,
I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The random read performance is ok, but the scan performance is poor.
I tried to increase "hbase.client.scanner.caching" to 100 to promote the scan performance but it made no difference. And when I tried to make smaller blocks by setting "BLOCKSIZE" when created tables to get better random read performance it made no difference too.
So, I am wondering if anyone could give some advice to solve this problem.
Thanks
Re: Re: hbase read performance tuning failed
Posted by Ted Yu <yu...@gmail.com>.
Xiaofeng:
Take a look at this:
http://hbase.apache.org/book.html#hbase.regionserver.checksum.verify
Cheers
On Tue, Jan 7, 2014 at 5:36 PM, LEI Xiaofeng <le...@ihep.ac.cn> wrote:
> I tried to set "BLOCKSIZE" to 0 when created table to tune the scan
> performance, but it did not work. Another thing is what do you mean by
> "turing HBase checksumming on"? I do not know how to do it. Could you give
> mw some detailed information?
>
> Thanks
>
>
> > -----原始邮件-----
> > 发件人: "Doug Meil" <do...@explorysmedical.com>
> > 发送时间: 2014年1月8日 星期三
> > 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>, "lars hofhansl" <
> larsh@apache.org>
> > 抄送:
> > 主题: Re: hbase read performance tuning failed
> >
> >
> > In addition to what Lars just said about the blocksize, this is a similar
> > question to another one that somebody asked, and it's always good to make
> > sure that you understand where your data is. As a sanity check, make sure
> > it's not all on one or two RSs (look at the hbase web pages or with tools
> > like Hannibal).
> >
> >
> > Also, you definitely want to to turn HBase checksumming on - and when you
> > do so you'll need to re-create the HFiles (e.g., you can't just change
> the
> > config and bounce the HBase cluster). That's a significant reduction in
> > I/O.
> >
> > Likewise, if you are doing a full-scan, make sure that you select only
> the
> > attributes you need...
> >
> > See this for more: http://hbase.apache.org/book.html#perf.reading
> >
> >
> >
> >
> >
> > On 1/7/14 1:24 PM, "lars hofhansl" <la...@apache.org> wrote:
> >
> > >If increasing hbase.client.scanner.caching makes no difference you have
> > >another issue.
> > >How many rows do you expect your to return?
> > >
> > >On contemporary hardware I manage to scan a few million KeyValues (i.e.
> > >columns) per second and per CPU core.
> > >Note that for scan performance you want to increase the BLOCKSIZE.
> > >
> > >
> > >-- Lars
> > >
> > >
> > >
> > >________________________________
> > > From: LEI Xiaofeng <le...@ihep.ac.cn>
> > >To: user@hbase.apache.org
> > >Sent: Monday, January 6, 2014 11:06 PM
> > >Subject: hbase read performance tuning failed
> > >
> > >
> > >Hi,
> > >I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
> > >random read performance is ok, but the scan performance is poor.
> > >I tried to increase "hbase.client.scanner.caching" to 100 to promote the
> > >scan performance but it made no difference. And when I tried to make
> > >smaller blocks by setting "BLOCKSIZE" when created tables to get better
> > >random read performance it made no difference too.
> > >So, I am wondering if anyone could give some advice to solve this
> problem.
> > >
> > >
> > >
> > >Thanks
> >
>
>
Re: Re: Re: hbase read performance tuning failed
Posted by Ted Yu <yu...@gmail.com>.
Please refer to https://issues.apache.org/jira/browse/HBASE-1015
Cheers
On Tue, Jan 7, 2014 at 5:54 PM, LEI Xiaofeng <le...@ihep.ac.cn> wrote:
> In addition, I am using c++ to access HBase. Will it make any difference
> compared to using Java?
>
>
> > -----原始邮件-----
> > 发件人: "LEI Xiaofeng" <le...@ihep.ac.cn>
> > 发送时间: 2014年1月8日 星期三
> > 收件人: user@hbase.apache.org, doug.meil@explorysmedical.com
> > 抄送:
> > 主题: Re: Re: hbase read performance tuning failed
> >
> > I tried to set "BLOCKSIZE" to 0 when created table to tune the scan
> performance, but it did not work. Another thing is what do you mean by
> "turing HBase checksumming on"? I do not know how to do it. Could you give
> mw some detailed information?
> >
> > Thanks
> >
> >
> > > -----原始邮件-----
> > > 发件人: "Doug Meil" <do...@explorysmedical.com>
> > > 发送时间: 2014年1月8日 星期三
> > > 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>, "lars hofhansl"
> <la...@apache.org>
> > > 抄送:
> > > 主题: Re: hbase read performance tuning failed
> > >
> > >
> > > In addition to what Lars just said about the blocksize, this is a
> similar
> > > question to another one that somebody asked, and it's always good to
> make
> > > sure that you understand where your data is. As a sanity check, make
> sure
> > > it's not all on one or two RSs (look at the hbase web pages or with
> tools
> > > like Hannibal).
> > >
> > >
> > > Also, you definitely want to to turn HBase checksumming on - and when
> you
> > > do so you'll need to re-create the HFiles (e.g., you can't just change
> the
> > > config and bounce the HBase cluster). That's a significant reduction
> in
> > > I/O.
> > >
> > > Likewise, if you are doing a full-scan, make sure that you select only
> the
> > > attributes you need...
> > >
> > > See this for more: http://hbase.apache.org/book.html#perf.reading
> > >
> > >
> > >
> > >
> > >
> > > On 1/7/14 1:24 PM, "lars hofhansl" <la...@apache.org> wrote:
> > >
> > > >If increasing hbase.client.scanner.caching makes no difference you
> have
> > > >another issue.
> > > >How many rows do you expect your to return?
> > > >
> > > >On contemporary hardware I manage to scan a few million KeyValues
> (i.e.
> > > >columns) per second and per CPU core.
> > > >Note that for scan performance you want to increase the BLOCKSIZE.
> > > >
> > > >
> > > >-- Lars
> > > >
> > > >
> > > >
> > > >________________________________
> > > > From: LEI Xiaofeng <le...@ihep.ac.cn>
> > > >To: user@hbase.apache.org
> > > >Sent: Monday, January 6, 2014 11:06 PM
> > > >Subject: hbase read performance tuning failed
> > > >
> > > >
> > > >Hi,
> > > >I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes.
> The
> > > >random read performance is ok, but the scan performance is poor.
> > > >I tried to increase "hbase.client.scanner.caching" to 100 to promote
> the
> > > >scan performance but it made no difference. And when I tried to make
> > > >smaller blocks by setting "BLOCKSIZE" when created tables to get
> better
> > > >random read performance it made no difference too.
> > > >So, I am wondering if anyone could give some advice to solve this
> problem.
> > > >
> > > >
> > > >
> > > >Thanks
> > >
> >
>
>
Re: Re: Re: hbase read performance tuning failed
Posted by LEI Xiaofeng <le...@ihep.ac.cn>.
In addition, I am using c++ to access HBase. Will it make any difference compared to using Java?
> -----原始邮件-----
> 发件人: "LEI Xiaofeng" <le...@ihep.ac.cn>
> 发送时间: 2014年1月8日 星期三
> 收件人: user@hbase.apache.org, doug.meil@explorysmedical.com
> 抄送:
> 主题: Re: Re: hbase read performance tuning failed
>
> I tried to set "BLOCKSIZE" to 0 when created table to tune the scan performance, but it did not work. Another thing is what do you mean by "turing HBase checksumming on"? I do not know how to do it. Could you give mw some detailed information?
>
> Thanks
>
>
> > -----原始邮件-----
> > 发件人: "Doug Meil" <do...@explorysmedical.com>
> > 发送时间: 2014年1月8日 星期三
> > 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>, "lars hofhansl" <la...@apache.org>
> > 抄送:
> > 主题: Re: hbase read performance tuning failed
> >
> >
> > In addition to what Lars just said about the blocksize, this is a similar
> > question to another one that somebody asked, and it's always good to make
> > sure that you understand where your data is. As a sanity check, make sure
> > it's not all on one or two RSs (look at the hbase web pages or with tools
> > like Hannibal).
> >
> >
> > Also, you definitely want to to turn HBase checksumming on - and when you
> > do so you'll need to re-create the HFiles (e.g., you can't just change the
> > config and bounce the HBase cluster). That's a significant reduction in
> > I/O.
> >
> > Likewise, if you are doing a full-scan, make sure that you select only the
> > attributes you need...
> >
> > See this for more: http://hbase.apache.org/book.html#perf.reading
> >
> >
> >
> >
> >
> > On 1/7/14 1:24 PM, "lars hofhansl" <la...@apache.org> wrote:
> >
> > >If increasing hbase.client.scanner.caching makes no difference you have
> > >another issue.
> > >How many rows do you expect your to return?
> > >
> > >On contemporary hardware I manage to scan a few million KeyValues (i.e.
> > >columns) per second and per CPU core.
> > >Note that for scan performance you want to increase the BLOCKSIZE.
> > >
> > >
> > >-- Lars
> > >
> > >
> > >
> > >________________________________
> > > From: LEI Xiaofeng <le...@ihep.ac.cn>
> > >To: user@hbase.apache.org
> > >Sent: Monday, January 6, 2014 11:06 PM
> > >Subject: hbase read performance tuning failed
> > >
> > >
> > >Hi,
> > >I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
> > >random read performance is ok, but the scan performance is poor.
> > >I tried to increase "hbase.client.scanner.caching" to 100 to promote the
> > >scan performance but it made no difference. And when I tried to make
> > >smaller blocks by setting "BLOCKSIZE" when created tables to get better
> > >random read performance it made no difference too.
> > >So, I am wondering if anyone could give some advice to solve this problem.
> > >
> > >
> > >
> > >Thanks
> >
>
Re: Re: hbase read performance tuning failed
Posted by LEI Xiaofeng <le...@ihep.ac.cn>.
I tried to set "BLOCKSIZE" to 0 when created table to tune the scan performance, but it did not work. Another thing is what do you mean by "turing HBase checksumming on"? I do not know how to do it. Could you give mw some detailed information?
Thanks
> -----原始邮件-----
> 发件人: "Doug Meil" <do...@explorysmedical.com>
> 发送时间: 2014年1月8日 星期三
> 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>, "lars hofhansl" <la...@apache.org>
> 抄送:
> 主题: Re: hbase read performance tuning failed
>
>
> In addition to what Lars just said about the blocksize, this is a similar
> question to another one that somebody asked, and it's always good to make
> sure that you understand where your data is. As a sanity check, make sure
> it's not all on one or two RSs (look at the hbase web pages or with tools
> like Hannibal).
>
>
> Also, you definitely want to to turn HBase checksumming on - and when you
> do so you'll need to re-create the HFiles (e.g., you can't just change the
> config and bounce the HBase cluster). That's a significant reduction in
> I/O.
>
> Likewise, if you are doing a full-scan, make sure that you select only the
> attributes you need...
>
> See this for more: http://hbase.apache.org/book.html#perf.reading
>
>
>
>
>
> On 1/7/14 1:24 PM, "lars hofhansl" <la...@apache.org> wrote:
>
> >If increasing hbase.client.scanner.caching makes no difference you have
> >another issue.
> >How many rows do you expect your to return?
> >
> >On contemporary hardware I manage to scan a few million KeyValues (i.e.
> >columns) per second and per CPU core.
> >Note that for scan performance you want to increase the BLOCKSIZE.
> >
> >
> >-- Lars
> >
> >
> >
> >________________________________
> > From: LEI Xiaofeng <le...@ihep.ac.cn>
> >To: user@hbase.apache.org
> >Sent: Monday, January 6, 2014 11:06 PM
> >Subject: hbase read performance tuning failed
> >
> >
> >Hi,
> >I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
> >random read performance is ok, but the scan performance is poor.
> >I tried to increase "hbase.client.scanner.caching" to 100 to promote the
> >scan performance but it made no difference. And when I tried to make
> >smaller blocks by setting "BLOCKSIZE" when created tables to get better
> >random read performance it made no difference too.
> >So, I am wondering if anyone could give some advice to solve this problem.
> >
> >
> >
> >Thanks
>
Re: hbase read performance tuning failed
Posted by Doug Meil <do...@explorysmedical.com>.
In addition to what Lars just said about the blocksize, this is a similar
question to another one that somebody asked, and it's always good to make
sure that you understand where your data is. As a sanity check, make sure
it's not all on one or two RSs (look at the hbase web pages or with tools
like Hannibal).
Also, you definitely want to to turn HBase checksumming on - and when you
do so you'll need to re-create the HFiles (e.g., you can't just change the
config and bounce the HBase cluster). That's a significant reduction in
I/O.
Likewise, if you are doing a full-scan, make sure that you select only the
attributes you need...
See this for more: http://hbase.apache.org/book.html#perf.reading
On 1/7/14 1:24 PM, "lars hofhansl" <la...@apache.org> wrote:
>If increasing hbase.client.scanner.caching makes no difference you have
>another issue.
>How many rows do you expect your to return?
>
>On contemporary hardware I manage to scan a few million KeyValues (i.e.
>columns) per second and per CPU core.
>Note that for scan performance you want to increase the BLOCKSIZE.
>
>
>-- Lars
>
>
>
>________________________________
> From: LEI Xiaofeng <le...@ihep.ac.cn>
>To: user@hbase.apache.org
>Sent: Monday, January 6, 2014 11:06 PM
>Subject: hbase read performance tuning failed
>
>
>Hi,
>I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
>random read performance is ok, but the scan performance is poor.
>I tried to increase "hbase.client.scanner.caching" to 100 to promote the
>scan performance but it made no difference. And when I tried to make
>smaller blocks by setting "BLOCKSIZE" when created tables to get better
>random read performance it made no difference too.
>So, I am wondering if anyone could give some advice to solve this problem.
>
>
>
>Thanks
Re: Re: hbase read performance tuning failed
Posted by Haosong Huang <ha...@gmail.com>.
>Ahh... You're using thrift.
Thrift server is the bottleneck. Use Java please.
On Wed, Jan 8, 2014 at 3:19 PM, lars hofhansl <la...@apache.org> wrote:
> Ahh... You're using thrift. I honestly do not know what performance
> characteristics you should expect from that.
> Maybe some folks here who use thrift can answer that...?
>
> -- Lars
>
>
>
> ________________________________
> From: LEI Xiaofeng <le...@ihep.ac.cn>
> To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Tuesday, January 7, 2014 5:29 PM
> Subject: Re: Re: hbase read performance tuning failed
>
>
> Lars,
>
> I use "scannerOpenWithPrefix" func to get my scanner and use
> "scannerGetList(rowResults, scanner,100)" func to get rows. I can only get
> about 5K records per second. But my exception is at least 220K records per
> second.
>
> I want to know what have you done makes you manage to get that permance as
> you said. Could you give me some more detailed suggestion?
>
> Thanks
>
>
>
> > -----原始邮件-----
> > 发件人: "lars hofhansl" <la...@apache.org>
> > 发送时间: 2014年1月8日 星期三
> > 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>
> > 抄送:
> > 主题: Re: hbase read performance tuning failed
> >
> > If increasing hbase.client.scanner.caching makes no difference you have
> another issue.
> > How many rows do you expect your to return?
> >
> > On contemporary hardware I manage to scan a few million KeyValues (i.e.
> columns) per second and per CPU core.
> > Note that for scan performance you want to increase the BLOCKSIZE.
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: LEI Xiaofeng <le...@ihep.ac.cn>
> > To: user@hbase.apache.org
> > Sent: Monday, January 6, 2014 11:06 PM
> > Subject: hbase read performance tuning failed
> >
> >
> > Hi,
> > I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
> random read performance is ok, but the scan performance is poor.
> > I tried to increase "hbase.client.scanner.caching" to 100 to promote the
> scan performance but it made no difference. And when I tried to make
> smaller blocks by setting "BLOCKSIZE" when created tables to get better
> random read performance it made no difference too.
> > So, I am wondering if anyone could give some advice to solve this
> problem.
> >
> >
> >
> > Thanks
>
--
Best Regards,
Haosdent Huang
Re: Re: hbase read performance tuning failed
Posted by lars hofhansl <la...@apache.org>.
Ahh... You're using thrift. I honestly do not know what performance characteristics you should expect from that.
Maybe some folks here who use thrift can answer that...?
-- Lars
________________________________
From: LEI Xiaofeng <le...@ihep.ac.cn>
To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Tuesday, January 7, 2014 5:29 PM
Subject: Re: Re: hbase read performance tuning failed
Lars,
I use "scannerOpenWithPrefix" func to get my scanner and use "scannerGetList(rowResults, scanner,100)" func to get rows. I can only get about 5K records per second. But my exception is at least 220K records per second.
I want to know what have you done makes you manage to get that permance as you said. Could you give me some more detailed suggestion?
Thanks
> -----原始邮件-----
> 发件人: "lars hofhansl" <la...@apache.org>
> 发送时间: 2014年1月8日 星期三
> 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>
> 抄送:
> 主题: Re: hbase read performance tuning failed
>
> If increasing hbase.client.scanner.caching makes no difference you have another issue.
> How many rows do you expect your to return?
>
> On contemporary hardware I manage to scan a few million KeyValues (i.e. columns) per second and per CPU core.
> Note that for scan performance you want to increase the BLOCKSIZE.
>
>
> -- Lars
>
>
>
> ________________________________
> From: LEI Xiaofeng <le...@ihep.ac.cn>
> To: user@hbase.apache.org
> Sent: Monday, January 6, 2014 11:06 PM
> Subject: hbase read performance tuning failed
>
>
> Hi,
> I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The random read performance is ok, but the scan performance is poor.
> I tried to increase "hbase.client.scanner.caching" to 100 to promote the scan performance but it made no difference. And when I tried to make smaller blocks by setting "BLOCKSIZE" when created tables to get better random read performance it made no difference too.
> So, I am wondering if anyone could give some advice to solve this problem.
>
>
>
> Thanks
Re: Re: hbase read performance tuning failed
Posted by LEI Xiaofeng <le...@ihep.ac.cn>.
Lars,
I use "scannerOpenWithPrefix" func to get my scanner and use "scannerGetList(rowResults, scanner,100)" func to get rows. I can only get about 5K records per second. But my exception is at least 220K records per second.
I want to know what have you done makes you manage to get that permance as you said. Could you give me some more detailed suggestion?
Thanks
> -----原始邮件-----
> 发件人: "lars hofhansl" <la...@apache.org>
> 发送时间: 2014年1月8日 星期三
> 收件人: "user@hbase.apache.org" <us...@hbase.apache.org>
> 抄送:
> 主题: Re: hbase read performance tuning failed
>
> If increasing hbase.client.scanner.caching makes no difference you have another issue.
> How many rows do you expect your to return?
>
> On contemporary hardware I manage to scan a few million KeyValues (i.e. columns) per second and per CPU core.
> Note that for scan performance you want to increase the BLOCKSIZE.
>
>
> -- Lars
>
>
>
> ________________________________
> From: LEI Xiaofeng <le...@ihep.ac.cn>
> To: user@hbase.apache.org
> Sent: Monday, January 6, 2014 11:06 PM
> Subject: hbase read performance tuning failed
>
>
> Hi,
> I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The random read performance is ok, but the scan performance is poor.
> I tried to increase "hbase.client.scanner.caching" to 100 to promote the scan performance but it made no difference. And when I tried to make smaller blocks by setting "BLOCKSIZE" when created tables to get better random read performance it made no difference too.
> So, I am wondering if anyone could give some advice to solve this problem.
>
>
>
> Thanks
Re: hbase read performance tuning failed
Posted by lars hofhansl <la...@apache.org>.
If increasing hbase.client.scanner.caching makes no difference you have another issue.
How many rows do you expect your to return?
On contemporary hardware I manage to scan a few million KeyValues (i.e. columns) per second and per CPU core.
Note that for scan performance you want to increase the BLOCKSIZE.
-- Lars
________________________________
From: LEI Xiaofeng <le...@ihep.ac.cn>
To: user@hbase.apache.org
Sent: Monday, January 6, 2014 11:06 PM
Subject: hbase read performance tuning failed
Hi,
I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The random read performance is ok, but the scan performance is poor.
I tried to increase "hbase.client.scanner.caching" to 100 to promote the scan performance but it made no difference. And when I tried to make smaller blocks by setting "BLOCKSIZE" when created tables to get better random read performance it made no difference too.
So, I am wondering if anyone could give some advice to solve this problem.
Thanks
Re: Re: hbase read performance tuning failed
Posted by LEI Xiaofeng <le...@ihep.ac.cn>.
I use "scannerOpenWithPrefix" func to get my scanner and use "scannerGetList(rowResults, scanner,100)" func to get rows. I can only get about 5K records per second. But my exception is at least 220K records per second.
> -----原始邮件-----
> 发件人: "Haosong Huang" <ha...@gmail.com>
> 发送时间: 2014年1月7日 星期二
> 收件人: user@hbase.apache.org
> 抄送:
> 主题: Re: hbase read performance tuning failed
>
> >but the scan performance is poor.
>
> Did you set start row and end row in the Scan? Besides, if the random read
> performance of your cluster is ok, maybe smallscan could help you.
> HBASE-9488 <https://issues.apache.org/jira/browse/HBASE-9488>
>
>
> On Tue, Jan 7, 2014 at 3:06 PM, LEI Xiaofeng <le...@ihep.ac.cn> wrote:
>
> > Hi,
> > I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
> > random read performance is ok, but the scan performance is poor.
> > I tried to increase "hbase.client.scanner.caching" to 100 to promote the
> > scan performance but it made no difference. And when I tried to make
> > smaller blocks by setting "BLOCKSIZE" when created tables to get better
> > random read performance it made no difference too.
> > So, I am wondering if anyone could give some advice to solve this problem.
> >
> >
> >
> > Thanks
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
Re: hbase read performance tuning failed
Posted by Haosong Huang <ha...@gmail.com>.
>but the scan performance is poor.
Did you set start row and end row in the Scan? Besides, if the random read
performance of your cluster is ok, maybe smallscan could help you.
HBASE-9488 <https://issues.apache.org/jira/browse/HBASE-9488>
On Tue, Jan 7, 2014 at 3:06 PM, LEI Xiaofeng <le...@ihep.ac.cn> wrote:
> Hi,
> I am running hbase-0.94.6-cdh4.5.0 and set up a cluster of 5 nodes. The
> random read performance is ok, but the scan performance is poor.
> I tried to increase "hbase.client.scanner.caching" to 100 to promote the
> scan performance but it made no difference. And when I tried to make
> smaller blocks by setting "BLOCKSIZE" when created tables to get better
> random read performance it made no difference too.
> So, I am wondering if anyone could give some advice to solve this problem.
>
>
>
> Thanks
>
--
Best Regards,
Haosdent Huang