You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Juraj jiv <fa...@gmail.com> on 2014/08/15 09:44:41 UTC
Hbase region count and RS count for 2TB+
Hi all,
i read in documentation that max region count per region server should be
around 200. I unfortunately have currently 800 regions for RS.
Those tables are not presplitted (all handled by hbase), does manual
spliting helps in this case? hdfs fodler /hbase/data currently hold around
2.2TB of data and i have 4 region servers, is it enough or i need more RS?
Also will Hbase compact these tables automatically to have less regions?
Hbase version 0.96.1.1-cdh5.0.1
Thanks for any guidance.
JV
Re: Hbase region count and RS count for 2TB+
Posted by Ted Yu <yu...@gmail.com>.
For your last question, compaction wouldn't reduce the number of regions.
Please take a look at HBASE-7403 for online merge.
Cheers
On Aug 15, 2014, at 12:44 AM, Juraj jiv <fa...@gmail.com> wrote:
> Hi all,
> i read in documentation that max region count per region server should be
> around 200. I unfortunately have currently 800 regions for RS.
> Those tables are not presplitted (all handled by hbase), does manual
> spliting helps in this case? hdfs fodler /hbase/data currently hold around
> 2.2TB of data and i have 4 region servers, is it enough or i need more RS?
>
> Also will Hbase compact these tables automatically to have less regions?
>
> Hbase version 0.96.1.1-cdh5.0.1
> Thanks for any guidance.
>
> JV
Re: Hbase region count and RS count for 2TB+
Posted by Ted Yu <yu...@gmail.com>.
What is the value for hbase.hregion.max.filesize ?
Cheers
On Aug 15, 2014, at 2:08 AM, Juraj jiv <fa...@gmail.com> wrote:
> Hi,
> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>
> So i think yes, 650MB per RS. If its small then why hbase created so many
> regions?
>
> Thanks
> JV
>
>
> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
> wrote:
>
>> So you have 3200 regions in total, and that means each region is about
>> 650mb in size only (if you have 2.2 Tb in total), unless I'm missing
>> something? If that's the case your regions are probably way too small.
>>
>> -Mikhail
>>
>>
>> 2014-08-15 0:44 GMT-07:00 Juraj jiv <fa...@gmail.com>:
>>
>>> Hi all,
>>> i read in documentation that max region count per region server should be
>>> around 200. I unfortunately have currently 800 regions for RS.
>>> Those tables are not presplitted (all handled by hbase), does manual
>>> spliting helps in this case? hdfs fodler /hbase/data currently hold
>> around
>>> 2.2TB of data and i have 4 region servers, is it enough or i need more
>> RS?
>>>
>>> Also will Hbase compact these tables automatically to have less regions?
>>>
>>> Hbase version 0.96.1.1-cdh5.0.1
>>> Thanks for any guidance.
>>>
>>> JV
>>
>>
>>
>> --
>> Thanks,
>> Michael Antonov
>>
Re: Hbase region count and RS count for 2TB+
Posted by Juraj jiv <fa...@gmail.com>.
Hello all,
thanks for replies.
hbase.hregion.max.filesize is set to default size 1GB. I set it to 10GB as
recommended.
JV
On Fri, Aug 15, 2014 at 9:15 PM, Ted Yu <yu...@gmail.com> wrote:
> In that case, HBASE-7403 would come to the rescue.
>
> Cheers
>
>
> On Fri, Aug 15, 2014 at 12:09 PM, Andrey Stepachev <oc...@gmail.com>
> wrote:
>
>> Here can be another problem - timestamp based keys or too many deletion.
>> Compaction removes most and regions become nearly empty.
>>
>> lars hofhansl <la...@apache.org>
>> 15 Aug 2014 22:12
>> HBase initially tries to spread the load out to more region servers by
>> splitting regions early when there aren't many (per table).
>> As more regions are seen split size approaches the configured max file
>> size.
>> This early splitting turned out to be a bit too aggressive (see
>> HBASE-10501, which modifies it a bit)... Even then 800 regions is a lot.
>>
>> As others have asked, what's
>>
>> hbase.hregion.max.filesize set to? Make sure it's at least 10gb.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Juraj jiv <fa...@gmail.com> <fa...@gmail.com>
>> To: user@hbase.apache.org
>> Sent: Friday, August 15, 2014 2:08 AM
>> Subject: Re: Hbase region count and RS count for 2TB+
>>
>>
>> Hi,
>> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>>
>> So i think yes, 650MB per RS. If its small then why hbase created so many
>> regions?
>>
>> Thanks
>> JV
>>
>>
>>
>> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
>> <ol...@gmail.com>
>> Juraj jiv <fa...@gmail.com>
>> 15 Aug 2014 13:08
>> Hi,
>> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>>
>> So i think yes, 650MB per RS. If its small then why hbase created so many
>> regions?
>>
>> Thanks
>> JV
>>
>>
>> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
>> <ol...@gmail.com>
>>
>> Mikhail Antonov <ol...@gmail.com>
>> 15 Aug 2014 11:51
>> So you have 3200 regions in total, and that means each region is about
>> 650mb in size only (if you have 2.2 Tb in total), unless I'm missing
>> something? If that's the case your regions are probably way too small.
>>
>> -Mikhail
>>
>>
>>
>>
>>
>> Juraj jiv <fa...@gmail.com>
>> 15 Aug 2014 11:44
>> Hi all,
>> i read in documentation that max region count per region server should be
>> around 200. I unfortunately have currently 800 regions for RS.
>> Those tables are not presplitted (all handled by hbase), does manual
>> spliting helps in this case? hdfs fodler /hbase/data currently hold around
>> 2.2TB of data and i have 4 region servers, is it enough or i need more RS?
>>
>> Also will Hbase compact these tables automatically to have less regions?
>>
>> Hbase version 0.96.1.1-cdh5.0.1
>> Thanks for any guidance.
>>
>> JV
>>
>>
>
Re: Hbase region count and RS count for 2TB+
Posted by Ted Yu <yu...@gmail.com>.
In that case, HBASE-7403 would come to the rescue.
Cheers
On Fri, Aug 15, 2014 at 12:09 PM, Andrey Stepachev <oc...@gmail.com> wrote:
> Here can be another problem - timestamp based keys or too many deletion.
> Compaction removes most and regions become nearly empty.
>
> lars hofhansl <la...@apache.org>
> 15 Aug 2014 22:12
> HBase initially tries to spread the load out to more region servers by
> splitting regions early when there aren't many (per table).
> As more regions are seen split size approaches the configured max file
> size.
> This early splitting turned out to be a bit too aggressive (see
> HBASE-10501, which modifies it a bit)... Even then 800 regions is a lot.
>
> As others have asked, what's
>
> hbase.hregion.max.filesize set to? Make sure it's at least 10gb.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Juraj jiv <fa...@gmail.com> <fa...@gmail.com>
> To: user@hbase.apache.org
> Sent: Friday, August 15, 2014 2:08 AM
> Subject: Re: Hbase region count and RS count for 2TB+
>
>
> Hi,
> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>
> So i think yes, 650MB per RS. If its small then why hbase created so many
> regions?
>
> Thanks
> JV
>
>
>
> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
> <ol...@gmail.com>
> Juraj jiv <fa...@gmail.com>
> 15 Aug 2014 13:08
> Hi,
> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>
> So i think yes, 650MB per RS. If its small then why hbase created so many
> regions?
>
> Thanks
> JV
>
>
> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
> <ol...@gmail.com>
>
> Mikhail Antonov <ol...@gmail.com>
> 15 Aug 2014 11:51
> So you have 3200 regions in total, and that means each region is about
> 650mb in size only (if you have 2.2 Tb in total), unless I'm missing
> something? If that's the case your regions are probably way too small.
>
> -Mikhail
>
>
>
>
>
> Juraj jiv <fa...@gmail.com>
> 15 Aug 2014 11:44
> Hi all,
> i read in documentation that max region count per region server should be
> around 200. I unfortunately have currently 800 regions for RS.
> Those tables are not presplitted (all handled by hbase), does manual
> spliting helps in this case? hdfs fodler /hbase/data currently hold around
> 2.2TB of data and i have 4 region servers, is it enough or i need more RS?
>
> Also will Hbase compact these tables automatically to have less regions?
>
> Hbase version 0.96.1.1-cdh5.0.1
> Thanks for any guidance.
>
> JV
>
>
Re: Hbase region count and RS count for 2TB+
Posted by Andrey Stepachev <oc...@gmail.com>.
Here can be another problem - timestamp based keys or too many deletion.
Compaction removes most and regions become nearly empty.
> lars hofhansl <ma...@apache.org>
> 15 Aug 2014 22:12
> HBase initially tries to spread the load out to more region servers by
> splitting regions early when there aren't many (per table).
> As more regions are seen split size approaches the configured max file
> size.
> This early splitting turned out to be a bit too aggressive (see
> HBASE-10501, which modifies it a bit)... Even then 800 regions is a lot.
>
> As others have asked, what's
>
> hbase.hregion.max.filesize set to? Make sure it's at least 10gb.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Juraj jiv <fa...@gmail.com>
> To: user@hbase.apache.org
> Sent: Friday, August 15, 2014 2:08 AM
> Subject: Re: Hbase region count and RS count for 2TB+
>
>
> Hi,
> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>
> So i think yes, 650MB per RS. If its small then why hbase created so many
> regions?
>
> Thanks
> JV
>
>
>
> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
> Juraj jiv <ma...@gmail.com>
> 15 Aug 2014 13:08
> Hi,
> size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
>
> So i think yes, 650MB per RS. If its small then why hbase created so many
> regions?
>
> Thanks
> JV
>
>
> On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
>
> Mikhail Antonov <ma...@gmail.com>
> 15 Aug 2014 11:51
> So you have 3200 regions in total, and that means each region is about
> 650mb in size only (if you have 2.2 Tb in total), unless I'm missing
> something? If that's the case your regions are probably way too small.
>
> -Mikhail
>
>
>
>
>
> Juraj jiv <ma...@gmail.com>
> 15 Aug 2014 11:44
> Hi all,
> i read in documentation that max region count per region server should be
> around 200. I unfortunately have currently 800 regions for RS.
> Those tables are not presplitted (all handled by hbase), does manual
> spliting helps in this case? hdfs fodler /hbase/data currently hold around
> 2.2TB of data and i have 4 region servers, is it enough or i need more RS?
>
> Also will Hbase compact these tables automatically to have less regions?
>
> Hbase version 0.96.1.1-cdh5.0.1
> Thanks for any guidance.
>
> JV
>
Re: Hbase region count and RS count for 2TB+
Posted by lars hofhansl <la...@apache.org>.
HBase initially tries to spread the load out to more region servers by splitting regions early when there aren't many (per table).
As more regions are seen split size approaches the configured max file size.
This early splitting turned out to be a bit too aggressive (see HBASE-10501, which modifies it a bit)... Even then 800 regions is a lot.
As others have asked, what's
hbase.hregion.max.filesize set to? Make sure it's at least 10gb.
-- Lars
________________________________
From: Juraj jiv <fa...@gmail.com>
To: user@hbase.apache.org
Sent: Friday, August 15, 2014 2:08 AM
Subject: Re: Hbase region count and RS count for 2TB+
Hi,
size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
So i think yes, 650MB per RS. If its small then why hbase created so many
regions?
Thanks
JV
On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
wrote:
> So you have 3200 regions in total, and that means each region is about
> 650mb in size only (if you have 2.2 Tb in total), unless I'm missing
> something? If that's the case your regions are probably way too small.
>
> -Mikhail
>
>
> 2014-08-15 0:44 GMT-07:00 Juraj jiv <fa...@gmail.com>:
>
> > Hi all,
> > i read in documentation that max region count per region server should be
> > around 200. I unfortunately have currently 800 regions for RS.
> > Those tables are not presplitted (all handled by hbase), does manual
> > spliting helps in this case? hdfs fodler /hbase/data currently hold
> around
> > 2.2TB of data and i have 4 region servers, is it enough or i need more
> RS?
> >
> > Also will Hbase compact these tables automatically to have less regions?
> >
> > Hbase version 0.96.1.1-cdh5.0.1
> > Thanks for any guidance.
> >
> > JV
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Hbase region count and RS count for 2TB+
Posted by Juraj jiv <fa...@gmail.com>.
Hi,
size 2.2TB i got from cmd "hdfs dfs -du -h /hbase/data"
So i think yes, 650MB per RS. If its small then why hbase created so many
regions?
Thanks
JV
On Fri, Aug 15, 2014 at 9:51 AM, Mikhail Antonov <ol...@gmail.com>
wrote:
> So you have 3200 regions in total, and that means each region is about
> 650mb in size only (if you have 2.2 Tb in total), unless I'm missing
> something? If that's the case your regions are probably way too small.
>
> -Mikhail
>
>
> 2014-08-15 0:44 GMT-07:00 Juraj jiv <fa...@gmail.com>:
>
> > Hi all,
> > i read in documentation that max region count per region server should be
> > around 200. I unfortunately have currently 800 regions for RS.
> > Those tables are not presplitted (all handled by hbase), does manual
> > spliting helps in this case? hdfs fodler /hbase/data currently hold
> around
> > 2.2TB of data and i have 4 region servers, is it enough or i need more
> RS?
> >
> > Also will Hbase compact these tables automatically to have less regions?
> >
> > Hbase version 0.96.1.1-cdh5.0.1
> > Thanks for any guidance.
> >
> > JV
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Hbase region count and RS count for 2TB+
Posted by Mikhail Antonov <ol...@gmail.com>.
So you have 3200 regions in total, and that means each region is about
650mb in size only (if you have 2.2 Tb in total), unless I'm missing
something? If that's the case your regions are probably way too small.
-Mikhail
2014-08-15 0:44 GMT-07:00 Juraj jiv <fa...@gmail.com>:
> Hi all,
> i read in documentation that max region count per region server should be
> around 200. I unfortunately have currently 800 regions for RS.
> Those tables are not presplitted (all handled by hbase), does manual
> spliting helps in this case? hdfs fodler /hbase/data currently hold around
> 2.2TB of data and i have 4 region servers, is it enough or i need more RS?
>
> Also will Hbase compact these tables automatically to have less regions?
>
> Hbase version 0.96.1.1-cdh5.0.1
> Thanks for any guidance.
>
> JV
>
--
Thanks,
Michael Antonov