Re: Meta region hotspotting

[Ben West <bw...@yahoo.com>]

Thanks Jack.

We couldn't find any "bad" queries (though perhaps we weren't looking hard enough) but an upgrade to 0.92rc3 seems to have resolved the problem. Not sure why, but there were a couple modifications to HTablePool that might've done it.

Thanks for your help everyone,
-Ben


----- Original Message -----
From: Jack Levin <ma...@gmail.com>
To: user@hbase.apache.org
Cc: Ben West <bw...@yahoo.com>
Sent: Monday, December 26, 2011 7:32 PM
Subject: Re: Meta region hotspotting

Some time ago, we had a situation where our REST server was slammed
with queries that did not find any matches for rows in Hbase.  When
that happened we sustained 50k rpc/sec to META region server as
reported by the master web page.  After digging deeper we found that
reach request with 'wrong' url, caused full scan of META (more than
30k records for us),  This was in version 0.89.   I do not know if
this was fixed in 0.90.X but, its possible that there is a bug with
REST that causes full scan of META for any misses.   Just something to
check.  (as soon as we removed the erroneous queries to REST this
issue was resolved).

-Jack

On Fri, Dec 23, 2011 at 5:42 PM, Stack <st...@duboce.net> wrote:
> On Fri, Dec 23, 2011 at 1:42 PM, Ben West <bw...@yahoo.com> wrote:
>> For those following at home, I tried this change and it does not appear to have solved the problem. Back to the drawing board.
>>
>>
>
> Thanks for digging in Ben.
> St.Ack

Re: Meta region hotspotting

[Andrew Purtell <ap...@apache.org>]

> I do not know if this was fixed in 0.90.X but, its possible that there is a bug with REST that causes full scan of META for any misses. 


It's possible but REST relies on HTable / HTablePool and tries to do the right thing with the client API, to minimize REST specific problems interacting with HBase, HBASE-5228 not withstanding, a mistake for which I apologize.


Best regards


       - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)


----- Original Message -----
> From: Ben West <bw...@yahoo.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc: 
> Sent: Thursday, January 19, 2012 9:46 AM
> Subject: Re: Meta region hotspotting
> 
>T hanks Jack.
> 
> We couldn't find any "bad" queries (though perhaps we weren't 
> looking hard enough) but an upgrade to 0.92rc3 seems to have resolved the 
> problem. Not sure why, but there were a couple modifications to HTablePool that 
> might've done it.
> 
> Thanks for your help everyone,
> -Ben
> 
> 
> ----- Original Message -----
> From: Jack Levin <ma...@gmail.com>
> To: user@hbase.apache.org
> Cc: Ben West <bw...@yahoo.com>
> Sent: Monday, December 26, 2011 7:32 PM
> Subject: Re: Meta region hotspotting
> 
> Some time ago, we had a situation where our REST server was slammed
> with queries that did not find any matches for rows in Hbase.  When
> that happened we sustained 50k rpc/sec to META region server as
> reported by the master web page.  After digging deeper we found that
> reach request with 'wrong' url, caused full scan of META (more than
> 30k records for us),  This was in version 0.89.   I do not know if
> this was fixed in 0.90.X but, its possible that there is a bug with
> REST that causes full scan of META for any misses.   Just something to
> check.  (as soon as we removed the erroneous queries to REST this
> issue was resolved).
> 
> -Jack
> 
> On Fri, Dec 23, 2011 at 5:42 PM, Stack <st...@duboce.net> wrote:
>>  On Fri, Dec 23, 2011 at 1:42 PM, Ben West <bw...@yahoo.com> 
> wrote:
>>>  For those following at home, I tried this change and it does not appear 
> to have solved the problem. Back to the drawing board.
>>> 
>>> 
>> 
>>  Thanks for digging in Ben.
>>  St.Ack
>