You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Brad McCarty <mc...@gmail.com> on 2010/02/17 04:28:49 UTC
Question regarding scalability of regionservers
Hello all, we're looking at using HBase for the backend datastore for a large-scale site where many Tomcat app servers would access HBase in realtime. Our data access pattern is not completely random, we would have to access some common rows from many app servers.
I read in another post that if one has a "hot" row in a table, meaning very heavy read access to the same row, that the regionserver managing the region with that row can become a single bottleneck.
Is my understanding accurate? If so, then assuming I can cache the data in the memstore, will CPU utilization become the likely limiting resource on that regionserver? Also, if I'm hitting the region server from many client servers (Tomcat app servers), will the socket connection management overhead on the regionserver overwhelm that server?
If that's true, are there any other steps that can be taken to mitigate that risk, other than buying bigger hardware?
Thanks very much,
Brad McCarty
Re: Question regarding scalability of regionservers
Posted by Stack <st...@duboce.net>.
On Thu, Feb 18, 2010 at 12:58 AM, Erik Rozendaal <dl...@gmail.com> wrote:
> We're getting about 2500 gets/sec max per regionserver when it serves data from the block cache using HBase 0.20.3. All data is in the same region and the region has just one store file for the family we're querying. HBASE-2180 didn't seem to help here.
HBASE-2180 is about concurrency getting at files over in hdfs. It
will have no effect on data being read form memstores.
At this point the region server is using about 1.5 cores. Since we
have 8 cores available the bottleneck does not seem to be CPU.
>
> The number of clients does not seem to matter (we tested with 1-4 clients).
>
> The only other bottleneck that I can think of is network, but we test on a local 1 Gbit LAN. Our web server can handle 15,000 requests/sec, so that doesn't seem to be the case.
>
> Maybe there is some synchronization going limiting the CPU scalability of a single region server?
It would seem so.
Let me make an issue to profile it (Any chance of you wanting to dig
in?). It might be the row lock but if you are getting same numbers
whether 1 or 4 clients, it would seem to say no?
Thanks Erik.
St.Ack
>
> Regards,
> Erik
>
>
> On 17 feb 2010, at 20:44, Brad McCarty wrote:
>
>> I don't really have any idea yet what that request rate would be that would overwhelm a single RS; I assume the CPU overhead to server from cache is pretty small so it'd have to be a very high rate.
>>
>> I was originally more concerned about the aggregate CPU utilization on the RS caused by both connection management and cache serving, but if we have connection pools per Tomcat server that's less of a concern now.
>>
>> We're looking a serving a eCommerce catalog from HBase and our data access patterns won't be as random as some of the examples I've seen of other production sites using HBase, hence the concern.
>>
>> We'll know more in the next couple of weeks.
>>
>> Brad
>>
>>
>> On Feb 17, 2010, at 11:36 AM, Stack wrote:
>>
>>> On Wed, Feb 17, 2010 at 11:27 AM, Brad McCarty <mc...@gmail.com> wrote:
>>>>> This is hbase. You don't buy bigger hardware, you just add nodes (smile).
>>>>
>>>>
>>>> Not sure if that was tongue-in-cheek, because adding nodes wouldn't address the hot row issue would it??
>>>>
>>>
>>> Yes. I was trying to do tongue-in-cheek but, yes, you are right, it
>>> would not address the hot row issue (I misread).
>>>
>>> What kind of request rate are you thinking would overwhelm serving a
>>> row from cache?
>>>
>>> St.Ack
>>
>
>
Re: Question regarding scalability of regionservers
Posted by Erik Rozendaal <dl...@gmail.com>.
We're actually getting ~44,000 random rows. Each row is about 250 bytes.
On 18 feb 2010, at 10:18, Ryan Rawson wrote:
> Gets to a single row are serialized... could be it.
>
Re: Question regarding scalability of regionservers
Posted by Ryan Rawson <ry...@gmail.com>.
Gets to a single row are serialized... could be it.
On Feb 18, 2010 12:58 AM, "Erik Rozendaal" <dl...@gmail.com> wrote:
We're getting about 2500 gets/sec max per regionserver when it serves data
from the block cache using HBase 0.20.3. All data is in the same region and
the region has just one store file for the family we're querying. HBASE-2180
didn't seem to help here. At this point the region server is using about 1.5
cores. Since we have 8 cores available the bottleneck does not seem to be
CPU.
The number of clients does not seem to matter (we tested with 1-4 clients).
The only other bottleneck that I can think of is network, but we test on a
local 1 Gbit LAN. Our web server can handle 15,000 requests/sec, so that
doesn't seem to be the case.
Maybe there is some synchronization going limiting the CPU scalability of a
single region server?
Regards,
Erik
On 17 feb 2010, at 20:44, Brad McCarty wrote:
> I don't really have any idea yet what that reques...
Re: Question regarding scalability of regionservers
Posted by Erik Rozendaal <dl...@gmail.com>.
We're getting about 2500 gets/sec max per regionserver when it serves data from the block cache using HBase 0.20.3. All data is in the same region and the region has just one store file for the family we're querying. HBASE-2180 didn't seem to help here. At this point the region server is using about 1.5 cores. Since we have 8 cores available the bottleneck does not seem to be CPU.
The number of clients does not seem to matter (we tested with 1-4 clients).
The only other bottleneck that I can think of is network, but we test on a local 1 Gbit LAN. Our web server can handle 15,000 requests/sec, so that doesn't seem to be the case.
Maybe there is some synchronization going limiting the CPU scalability of a single region server?
Regards,
Erik
On 17 feb 2010, at 20:44, Brad McCarty wrote:
> I don't really have any idea yet what that request rate would be that would overwhelm a single RS; I assume the CPU overhead to server from cache is pretty small so it'd have to be a very high rate.
>
> I was originally more concerned about the aggregate CPU utilization on the RS caused by both connection management and cache serving, but if we have connection pools per Tomcat server that's less of a concern now.
>
> We're looking a serving a eCommerce catalog from HBase and our data access patterns won't be as random as some of the examples I've seen of other production sites using HBase, hence the concern.
>
> We'll know more in the next couple of weeks.
>
> Brad
>
>
> On Feb 17, 2010, at 11:36 AM, Stack wrote:
>
>> On Wed, Feb 17, 2010 at 11:27 AM, Brad McCarty <mc...@gmail.com> wrote:
>>>> This is hbase. You don't buy bigger hardware, you just add nodes (smile).
>>>
>>>
>>> Not sure if that was tongue-in-cheek, because adding nodes wouldn't address the hot row issue would it??
>>>
>>
>> Yes. I was trying to do tongue-in-cheek but, yes, you are right, it
>> would not address the hot row issue (I misread).
>>
>> What kind of request rate are you thinking would overwhelm serving a
>> row from cache?
>>
>> St.Ack
>
Re: Question regarding scalability of regionservers
Posted by Brad McCarty <mc...@gmail.com>.
I don't really have any idea yet what that request rate would be that would overwhelm a single RS; I assume the CPU overhead to server from cache is pretty small so it'd have to be a very high rate.
I was originally more concerned about the aggregate CPU utilization on the RS caused by both connection management and cache serving, but if we have connection pools per Tomcat server that's less of a concern now.
We're looking a serving a eCommerce catalog from HBase and our data access patterns won't be as random as some of the examples I've seen of other production sites using HBase, hence the concern.
We'll know more in the next couple of weeks.
Brad
On Feb 17, 2010, at 11:36 AM, Stack wrote:
> On Wed, Feb 17, 2010 at 11:27 AM, Brad McCarty <mc...@gmail.com> wrote:
>>> This is hbase. You don't buy bigger hardware, you just add nodes (smile).
>>
>>
>> Not sure if that was tongue-in-cheek, because adding nodes wouldn't address the hot row issue would it??
>>
>
> Yes. I was trying to do tongue-in-cheek but, yes, you are right, it
> would not address the hot row issue (I misread).
>
> What kind of request rate are you thinking would overwhelm serving a
> row from cache?
>
> St.Ack
Re: Question regarding scalability of regionservers
Posted by Stack <st...@duboce.net>.
On Wed, Feb 17, 2010 at 11:27 AM, Brad McCarty <mc...@gmail.com> wrote:
>> This is hbase. You don't buy bigger hardware, you just add nodes (smile).
>
>
> Not sure if that was tongue-in-cheek, because adding nodes wouldn't address the hot row issue would it??
>
Yes. I was trying to do tongue-in-cheek but, yes, you are right, it
would not address the hot row issue (I misread).
What kind of request rate are you thinking would overwhelm serving a
row from cache?
St.Ack
Re: Question regarding scalability of regionservers
Posted by Brad McCarty <mc...@gmail.com>.
Thanks for the answers. We're just about ready with our test cluster and we will try this test specifically.
The number of Tomcat servers hitting a common row is currently 40 with potentially up to 100 threads each (at peak intervals).
You also said:
> This is hbase. You don't buy bigger hardware, you just add nodes (smile).
Not sure if that was tongue-in-cheek, because adding nodes wouldn't address the hot row issue would it??
Thanks again
Brad
On Feb 16, 2010, at 9:23 PM, Stack wrote:
> On Tue, Feb 16, 2010 at 7:28 PM, Brad McCarty <mc...@gmail.com> wrote:
>
>> I read in another post that if one has a "hot" row in a table, meaning very heavy read access to the same row, that the regionserver managing the region with that row can become a single bottleneck.
>>
>
> If hot, it'll probably get stapled into cache.
>
>
>> Is my understanding accurate? If so, then assuming I can cache the data in the memstore, will CPU utilization become the likely limiting resource on that regionserver?
>
> Yes. That should be the case.
>
>
> Also, if I'm hitting the region server from many client servers
> (Tomcat app servers), will the socket connection management overhead
> on the regionserver overwhelm that server?
>>
>
> How many clients? 4 or 500 tomcat threads?
>
> The way the ipc between hbase client and server works is that it keeps
> up a single socket connection and multiplexes request/response over
> this one connection. This is how hadoop rpc works.
>
>
>> If that's true, are there any other steps that can be taken to mitigate that risk, other than buying bigger hardware?
>>
>
> This is hbase. You don't buy bigger hardware, you just add nodes (smile).
>
> The proper answer to your questions above is for you to give it a test
> run. Try setting up a cluster of about 5 hbase nodes and try a tomcat
> server requesting playing a query log that resembles what you might
> have in production.
>
> Yours,
> St.Ack
Re: Question regarding scalability of regionservers
Posted by Stack <st...@duboce.net>.
On Tue, Feb 16, 2010 at 7:28 PM, Brad McCarty <mc...@gmail.com> wrote:
> I read in another post that if one has a "hot" row in a table, meaning very heavy read access to the same row, that the regionserver managing the region with that row can become a single bottleneck.
>
If hot, it'll probably get stapled into cache.
> Is my understanding accurate? If so, then assuming I can cache the data in the memstore, will CPU utilization become the likely limiting resource on that regionserver?
Yes. That should be the case.
Also, if I'm hitting the region server from many client servers
(Tomcat app servers), will the socket connection management overhead
on the regionserver overwhelm that server?
>
How many clients? 4 or 500 tomcat threads?
The way the ipc between hbase client and server works is that it keeps
up a single socket connection and multiplexes request/response over
this one connection. This is how hadoop rpc works.
> If that's true, are there any other steps that can be taken to mitigate that risk, other than buying bigger hardware?
>
This is hbase. You don't buy bigger hardware, you just add nodes (smile).
The proper answer to your questions above is for you to give it a test
run. Try setting up a cluster of about 5 hbase nodes and try a tomcat
server requesting playing a query log that resembles what you might
have in production.
Yours,
St.Ack