You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by ShaoFeng Shi <sh...@gmail.com> on 2015/05/26 09:35:31 UTC
How to limit the HBase server bandwidth for scan requests from MapReduce?
Hello,
Currently we're running a MapReduce over live htables to do table merge
(introduced at https://hbase.apache.org/0.94/book/mapreduce.example.html);
At the samtime these tables are still serving user scan requests; As this
is a full table scan which may take much server resources, we want to
control the impact to users during the MapReduce, avoding remarkable
performance downgrade during the MR; I see there are two parameters might
be related: caching and cacheBlocks, like :
scan.setCaching(500);
scan.setCacheBlocks(false); // don't set to true for MR jobs
But still want to double check with the experts here, is there other ways
to control this? Thanks!
Shaofeng Shi
Apache Kylin (incubation)
Re: How to limit the HBase server bandwidth for scan requests from MapReduce?
Posted by ShaoFeng Shi <sh...@gmail.com>.
I see; we're still on 0.98, will verify this once upgrade the hbase; Thanks
for all the info!
2015-05-29 1:02 GMT+08:00 Nick Dimiduk <nd...@gmail.com>:
> On Thu, May 28, 2015 at 12:10 AM, ShaoFeng Shi <sh...@gmail.com>
> wrote:
>
> > Hi Ted, thanks for giving the link, our scenario is just such a case;
> We're
> > looking forward to see this feature in HBase 1.1; Thanks!
> >
>
> No need to "look forward to" -- 1.1 is released, go give it a spin!
>
> 2015-05-27 22:11 GMT+08:00 Ted Yu <yu...@gmail.com>:
> >
> > > Please see
> > >
> >
> https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
> > >
> > > Cheers
> > >
> > > On Tue, May 26, 2015 at 12:35 AM, ShaoFeng Shi <sh...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Currently we're running a MapReduce over live htables to do table
> merge
> > > > (introduced at
> > https://hbase.apache.org/0.94/book/mapreduce.example.html
> > > );
> > > > At the samtime these tables are still serving user scan requests; As
> > this
> > > > is a full table scan which may take much server resources, we want to
> > > > control the impact to users during the MapReduce, avoding remarkable
> > > > performance downgrade during the MR; I see there are two parameters
> > might
> > > > be related: caching and cacheBlocks, like :
> > > >
> > > >
> > > > scan.setCaching(500);
> > > >
> > > > scan.setCacheBlocks(false); // don't set to true for MR jobs
> > > >
> > > >
> > > > But still want to double check with the experts here, is there other
> > ways
> > > > to control this? Thanks!
> > > >
> > > > Shaofeng Shi
> > > > Apache Kylin (incubation)
> > > >
> > >
> >
>
Re: How to limit the HBase server bandwidth for scan requests from MapReduce?
Posted by Nick Dimiduk <nd...@gmail.com>.
On Thu, May 28, 2015 at 12:10 AM, ShaoFeng Shi <sh...@gmail.com>
wrote:
> Hi Ted, thanks for giving the link, our scenario is just such a case; We're
> looking forward to see this feature in HBase 1.1; Thanks!
>
No need to "look forward to" -- 1.1 is released, go give it a spin!
2015-05-27 22:11 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > Please see
> >
> https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
> >
> > Cheers
> >
> > On Tue, May 26, 2015 at 12:35 AM, ShaoFeng Shi <sh...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Currently we're running a MapReduce over live htables to do table merge
> > > (introduced at
> https://hbase.apache.org/0.94/book/mapreduce.example.html
> > );
> > > At the samtime these tables are still serving user scan requests; As
> this
> > > is a full table scan which may take much server resources, we want to
> > > control the impact to users during the MapReduce, avoding remarkable
> > > performance downgrade during the MR; I see there are two parameters
> might
> > > be related: caching and cacheBlocks, like :
> > >
> > >
> > > scan.setCaching(500);
> > >
> > > scan.setCacheBlocks(false); // don't set to true for MR jobs
> > >
> > >
> > > But still want to double check with the experts here, is there other
> ways
> > > to control this? Thanks!
> > >
> > > Shaofeng Shi
> > > Apache Kylin (incubation)
> > >
> >
>
RE: How to limit the HBase server bandwidth for scan requests from
MapReduce?
Posted by ashish singhi <as...@huawei.com>.
This feature is available in HBase 1.1 as part of HBASE-13205
Regards,
Ashish
-----Original Message-----
From: ShaoFeng Shi [mailto:shaofengshi@gmail.com]
Sent: 28 May 2015 12:40
To: user@hbase.apache.org
Subject: Re: How to limit the HBase server bandwidth for scan requests from MapReduce?
Hi Ted, thanks for giving the link, our scenario is just such a case; We're looking forward to see this feature in HBase 1.1; Thanks!
2015-05-27 22:11 GMT+08:00 Ted Yu <yu...@gmail.com>:
> Please see
> https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feat
> ure
>
> Cheers
>
> On Tue, May 26, 2015 at 12:35 AM, ShaoFeng Shi <sh...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Currently we're running a MapReduce over live htables to do table
> > merge (introduced at
> > https://hbase.apache.org/0.94/book/mapreduce.example.html
> );
> > At the samtime these tables are still serving user scan requests; As
> > this is a full table scan which may take much server resources, we
> > want to control the impact to users during the MapReduce, avoding
> > remarkable performance downgrade during the MR; I see there are two
> > parameters might be related: caching and cacheBlocks, like :
> >
> >
> > scan.setCaching(500);
> >
> > scan.setCacheBlocks(false); // don't set to true for MR jobs
> >
> >
> > But still want to double check with the experts here, is there other
> > ways to control this? Thanks!
> >
> > Shaofeng Shi
> > Apache Kylin (incubation)
> >
>
Re: How to limit the HBase server bandwidth for scan requests from MapReduce?
Posted by ShaoFeng Shi <sh...@gmail.com>.
Hi Ted, thanks for giving the link, our scenario is just such a case; We're
looking forward to see this feature in HBase 1.1; Thanks!
2015-05-27 22:11 GMT+08:00 Ted Yu <yu...@gmail.com>:
> Please see
> https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
>
> Cheers
>
> On Tue, May 26, 2015 at 12:35 AM, ShaoFeng Shi <sh...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Currently we're running a MapReduce over live htables to do table merge
> > (introduced at https://hbase.apache.org/0.94/book/mapreduce.example.html
> );
> > At the samtime these tables are still serving user scan requests; As this
> > is a full table scan which may take much server resources, we want to
> > control the impact to users during the MapReduce, avoding remarkable
> > performance downgrade during the MR; I see there are two parameters might
> > be related: caching and cacheBlocks, like :
> >
> >
> > scan.setCaching(500);
> >
> > scan.setCacheBlocks(false); // don't set to true for MR jobs
> >
> >
> > But still want to double check with the experts here, is there other ways
> > to control this? Thanks!
> >
> > Shaofeng Shi
> > Apache Kylin (incubation)
> >
>
Re: How to limit the HBase server bandwidth for scan requests from MapReduce?
Posted by Ted Yu <yu...@gmail.com>.
Please see
https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
Cheers
On Tue, May 26, 2015 at 12:35 AM, ShaoFeng Shi <sh...@gmail.com>
wrote:
> Hello,
>
> Currently we're running a MapReduce over live htables to do table merge
> (introduced at https://hbase.apache.org/0.94/book/mapreduce.example.html);
> At the samtime these tables are still serving user scan requests; As this
> is a full table scan which may take much server resources, we want to
> control the impact to users during the MapReduce, avoding remarkable
> performance downgrade during the MR; I see there are two parameters might
> be related: caching and cacheBlocks, like :
>
>
> scan.setCaching(500);
>
> scan.setCacheBlocks(false); // don't set to true for MR jobs
>
>
> But still want to double check with the experts here, is there other ways
> to control this? Thanks!
>
> Shaofeng Shi
> Apache Kylin (incubation)
>