Too many non-used regions

[Serega Sheypak <se...@gmail.com>]

Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB each.
Suddenly we met write perfomance issues.

Our workload is:
1. We have 100+ region table for write load. It has constant set of keys
(~6* 10^7) and we write "updates". No versions or stuff like that
2. We have several read-only tables, theirs size could be from 1 egion up
to 256 regions.
3. We have several non-used tables  theirs size could be from 1 region up
to 256 regions. These tables are previous versions of read-only tables.

Right now each RegionServer serves >100 regions, but only half of them are
active. Really nothing changed for the last few weeks, we got more non-used
tables, but quantity on active read-only tables is the same, single
wirte-only table dodn't get more bytes to write.

Can big count of non-used(!!!) regions affect write/read perfomance for
tables which are in use.

Re: Too many non-used regions

[Jean-Marc Spaggiari <je...@spaggiari.org>]

To reply to the question "Can big count of non-used(!!!) regions affect
write/read perfomance for tables which are in use.", respons is: No.

If regions are not receiving writes, memstore will not take memory from the
reserved write memory area.
If regions are not receiving reads, there will be no blocks cached for
those tables.

There will still be some memory used for the meta informations for those
regions on the region servers, but this memory is not part of the memstore
nor he blockcache areas and will not impact the performances of the other
operations as long as you still have enough memory to store metadata of all
those regions.

On average, on a server, how many regions total you have, and part from
that, how many get writes, how many get reads? Also, what is the configured
heap size and have you change the configuration for the block cache and the
memstore areas?

JM

2014-10-14 6:57 GMT-04:00 Ted Yu <yu...@gmail.com>:

> If cluster restarts, master still needs to assign such regions.
> This would affect MTTR.
>
> Otherwise in a stable cluster, they shouldn't consume much resource.
>
> Cheers
>
> On Oct 14, 2014, at 3:45 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Ok, I got.
> > And in general, region which has no R/W requests doesn't consume ny
> > resources except record in META?
> >
> > 2014-10-14 13:42 GMT+04:00 Ted Yu <yu...@gmail.com>:
> >
> >> In 0.94, load balancer doesn't take load each region receives into
> >> account. It only considers the number of regions as the sole criteria
> for
> >> balancing.
> >>
> >> Thus the high number of unused regions may cause imbalance.
> >>
> >> Please consider reducing unused regions.
> >>
> >> Cheers
> >>
> >> On Oct 14, 2014, at 2:33 AM, Serega Sheypak <se...@gmail.com>
> >> wrote:
> >>
> >>> We are using CDH 4.6 (hbase-0.94.15+86)
> >>> Regions are balanced.
> >>> - Read/write request are pretty even,
> >>> - count or regions per RegionServer is pretty even.
> >>> - there are no "hot spots"
> >>>
> >>>
> >>> 2014-10-14 12:50 GMT+04:00 Qiang Tian <ti...@gmail.com>:
> >>>
> >>>> region unbalanced?
> >>>> which hbase release?
> >>>>
> >>>>
> >>>> On Tue, Oct 14, 2014 at 3:39 PM, Serega Sheypak <
> >> serega.sheypak@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB
> each.
> >>>>> Suddenly we met write perfomance issues.
> >>>>>
> >>>>> Our workload is:
> >>>>> 1. We have 100+ region table for write load. It has constant set of
> >> keys
> >>>>> (~6* 10^7) and we write "updates". No versions or stuff like that
> >>>>> 2. We have several read-only tables, theirs size could be from 1
> egion
> >> up
> >>>>> to 256 regions.
> >>>>> 3. We have several non-used tables  theirs size could be from 1
> region
> >> up
> >>>>> to 256 regions. These tables are previous versions of read-only
> tables.
> >>>>>
> >>>>> Right now each RegionServer serves >100 regions, but only half of
> them
> >>>> are
> >>>>> active. Really nothing changed for the last few weeks, we got more
> >>>> non-used
> >>>>> tables, but quantity on active read-only tables is the same, single
> >>>>> wirte-only table dodn't get more bytes to write.
> >>>>>
> >>>>> Can big count of non-used(!!!) regions affect write/read perfomance
> for
> >>>>> tables which are in use.
> >>
>

Re: Too many non-used regions

[Ted Yu <yu...@gmail.com>]

If cluster restarts, master still needs to assign such regions. 
This would affect MTTR. 

Otherwise in a stable cluster, they shouldn't consume much resource. 

Cheers

On Oct 14, 2014, at 3:45 AM, Serega Sheypak <se...@gmail.com> wrote:

> Ok, I got.
> And in general, region which has no R/W requests doesn't consume ny
> resources except record in META?
> 
> 2014-10-14 13:42 GMT+04:00 Ted Yu <yu...@gmail.com>:
> 
>> In 0.94, load balancer doesn't take load each region receives into
>> account. It only considers the number of regions as the sole criteria for
>> balancing.
>> 
>> Thus the high number of unused regions may cause imbalance.
>> 
>> Please consider reducing unused regions.
>> 
>> Cheers
>> 
>> On Oct 14, 2014, at 2:33 AM, Serega Sheypak <se...@gmail.com>
>> wrote:
>> 
>>> We are using CDH 4.6 (hbase-0.94.15+86)
>>> Regions are balanced.
>>> - Read/write request are pretty even,
>>> - count or regions per RegionServer is pretty even.
>>> - there are no "hot spots"
>>> 
>>> 
>>> 2014-10-14 12:50 GMT+04:00 Qiang Tian <ti...@gmail.com>:
>>> 
>>>> region unbalanced?
>>>> which hbase release?
>>>> 
>>>> 
>>>> On Tue, Oct 14, 2014 at 3:39 PM, Serega Sheypak <
>> serega.sheypak@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB each.
>>>>> Suddenly we met write perfomance issues.
>>>>> 
>>>>> Our workload is:
>>>>> 1. We have 100+ region table for write load. It has constant set of
>> keys
>>>>> (~6* 10^7) and we write "updates". No versions or stuff like that
>>>>> 2. We have several read-only tables, theirs size could be from 1 egion
>> up
>>>>> to 256 regions.
>>>>> 3. We have several non-used tables  theirs size could be from 1 region
>> up
>>>>> to 256 regions. These tables are previous versions of read-only tables.
>>>>> 
>>>>> Right now each RegionServer serves >100 regions, but only half of them
>>>> are
>>>>> active. Really nothing changed for the last few weeks, we got more
>>>> non-used
>>>>> tables, but quantity on active read-only tables is the same, single
>>>>> wirte-only table dodn't get more bytes to write.
>>>>> 
>>>>> Can big count of non-used(!!!) regions affect write/read perfomance for
>>>>> tables which are in use.
>> 

Re: Too many non-used regions

[Serega Sheypak <se...@gmail.com>]

Ok, I got.
And in general, region which has no R/W requests doesn't consume ny
resources except record in META?

2014-10-14 13:42 GMT+04:00 Ted Yu <yu...@gmail.com>:

> In 0.94, load balancer doesn't take load each region receives into
> account. It only considers the number of regions as the sole criteria for
> balancing.
>
> Thus the high number of unused regions may cause imbalance.
>
> Please consider reducing unused regions.
>
> Cheers
>
> On Oct 14, 2014, at 2:33 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > We are using CDH 4.6 (hbase-0.94.15+86)
> > Regions are balanced.
> > - Read/write request are pretty even,
> > - count or regions per RegionServer is pretty even.
> > - there are no "hot spots"
> >
> >
> > 2014-10-14 12:50 GMT+04:00 Qiang Tian <ti...@gmail.com>:
> >
> >> region unbalanced?
> >> which hbase release?
> >>
> >>
> >> On Tue, Oct 14, 2014 at 3:39 PM, Serega Sheypak <
> serega.sheypak@gmail.com>
> >> wrote:
> >>
> >>> Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB each.
> >>> Suddenly we met write perfomance issues.
> >>>
> >>> Our workload is:
> >>> 1. We have 100+ region table for write load. It has constant set of
> keys
> >>> (~6* 10^7) and we write "updates". No versions or stuff like that
> >>> 2. We have several read-only tables, theirs size could be from 1 egion
> up
> >>> to 256 regions.
> >>> 3. We have several non-used tables  theirs size could be from 1 region
> up
> >>> to 256 regions. These tables are previous versions of read-only tables.
> >>>
> >>> Right now each RegionServer serves >100 regions, but only half of them
> >> are
> >>> active. Really nothing changed for the last few weeks, we got more
> >> non-used
> >>> tables, but quantity on active read-only tables is the same, single
> >>> wirte-only table dodn't get more bytes to write.
> >>>
> >>> Can big count of non-used(!!!) regions affect write/read perfomance for
> >>> tables which are in use.
> >>
>

Re: Too many non-used regions

[Ted Yu <yu...@gmail.com>]

In 0.94, load balancer doesn't take load each region receives into account. It only considers the number of regions as the sole criteria for balancing. 

Thus the high number of unused regions may cause imbalance. 

Please consider reducing unused regions. 

Cheers

On Oct 14, 2014, at 2:33 AM, Serega Sheypak <se...@gmail.com> wrote:

> We are using CDH 4.6 (hbase-0.94.15+86)
> Regions are balanced.
> - Read/write request are pretty even,
> - count or regions per RegionServer is pretty even.
> - there are no "hot spots"
> 
> 
> 2014-10-14 12:50 GMT+04:00 Qiang Tian <ti...@gmail.com>:
> 
>> region unbalanced?
>> which hbase release?
>> 
>> 
>> On Tue, Oct 14, 2014 at 3:39 PM, Serega Sheypak <se...@gmail.com>
>> wrote:
>> 
>>> Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB each.
>>> Suddenly we met write perfomance issues.
>>> 
>>> Our workload is:
>>> 1. We have 100+ region table for write load. It has constant set of keys
>>> (~6* 10^7) and we write "updates". No versions or stuff like that
>>> 2. We have several read-only tables, theirs size could be from 1 egion up
>>> to 256 regions.
>>> 3. We have several non-used tables  theirs size could be from 1 region up
>>> to 256 regions. These tables are previous versions of read-only tables.
>>> 
>>> Right now each RegionServer serves >100 regions, but only half of them
>> are
>>> active. Really nothing changed for the last few weeks, we got more
>> non-used
>>> tables, but quantity on active read-only tables is the same, single
>>> wirte-only table dodn't get more bytes to write.
>>> 
>>> Can big count of non-used(!!!) regions affect write/read perfomance for
>>> tables which are in use.
>> 

Re: Too many non-used regions

[Serega Sheypak <se...@gmail.com>]

We are using CDH 4.6 (hbase-0.94.15+86)
Regions are balanced.
- Read/write request are pretty even,
- count or regions per RegionServer is pretty even.
- there are no "hot spots"


2014-10-14 12:50 GMT+04:00 Qiang Tian <ti...@gmail.com>:

> region unbalanced?
> which hbase release?
>
>
> On Tue, Oct 14, 2014 at 3:39 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB each.
> > Suddenly we met write perfomance issues.
> >
> > Our workload is:
> > 1. We have 100+ region table for write load. It has constant set of keys
> > (~6* 10^7) and we write "updates". No versions or stuff like that
> > 2. We have several read-only tables, theirs size could be from 1 egion up
> > to 256 regions.
> > 3. We have several non-used tables  theirs size could be from 1 region up
> > to 256 regions. These tables are previous versions of read-only tables.
> >
> > Right now each RegionServer serves >100 regions, but only half of them
> are
> > active. Really nothing changed for the last few weeks, we got more
> non-used
> > tables, but quantity on active read-only tables is the same, single
> > wirte-only table dodn't get more bytes to write.
> >
> > Can big count of non-used(!!!) regions affect write/read perfomance for
> > tables which are in use.
> >
>

Re: Too many non-used regions

[Qiang Tian <ti...@gmail.com>]

region unbalanced?
which hbase release?


On Tue, Oct 14, 2014 at 3:39 PM, Serega Sheypak <se...@gmail.com>
wrote:

> Hi, we have 10 nodes cluster with 10HDD, 256 GB RAM and 10HDD 3TB each.
> Suddenly we met write perfomance issues.
>
> Our workload is:
> 1. We have 100+ region table for write load. It has constant set of keys
> (~6* 10^7) and we write "updates". No versions or stuff like that
> 2. We have several read-only tables, theirs size could be from 1 egion up
> to 256 regions.
> 3. We have several non-used tables  theirs size could be from 1 region up
> to 256 regions. These tables are previous versions of read-only tables.
>
> Right now each RegionServer serves >100 regions, but only half of them are
> active. Really nothing changed for the last few weeks, we got more non-used
> tables, but quantity on active read-only tables is the same, single
> wirte-only table dodn't get more bytes to write.
>
> Can big count of non-used(!!!) regions affect write/read perfomance for
> tables which are in use.
>