You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ben West <bw...@yahoo.com> on 2012/01/17 20:25:30 UTC
HBase 0.92rc3 rest performance
Hi all
We're trying out .92rc3 instead of .90.4, and for the most part everything seems fine. But we have a simple test of REST performance which is basically a large number of cURL jobs getting random rows, and this test is running *a lot* slower under .92.
When we run just a single client doing REST GETs, the performance is fine. But once I have dozens or hundreds of clients, performance is ~20x worse than under .90.4 (response time is 7-800ms instead of 40-50ms).
YCSB has pretty much the same performance under both versions, as do other internal tools measuring Thrift and native performance, so I don't feel like this is a problem with HBase coresetup (although it could be). I don't see anything suspicious in any logs, IO and CPU utilization are both low. Has anyone run into this or have thoughts on how to troubleshoot?
Thanks!
Ben
Re: HBase 0.92rc3 rest performance
Posted by Andrew Purtell <ap...@apache.org>.
Thanks Ben.
See https://issues.apache.org/jira/browse/HBASE-5228
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: Ben West <bw...@yahoo.com>
>To: "user@hbase.apache.org" <us...@hbase.apache.org>; Andrew Purtell <ap...@apache.org>
>Sent: Wednesday, January 18, 2012 7:18 AM
>Subject: Re: HBase 0.92rc3 rest performance
>
>Thanks for the quick responses!
>
>No higher CPU or memory usage as far as I can tell. No WARNs. We're using the default Jersey (1.4) and Jetty (6.1.26).
>
>Yes, you only need to start up some number of concurrent GETs. Here is my test script, if that helps. It is quite simple (in ksh):
>
>maxId=274894038
>for i in {1..1000}
>do
> # get a random number
> hex=`dd if=/dev/urandom bs=1 count=8 2>/dev/null |
> od -tx1 | head -1 | cut -d' ' -f2- |
> tr -d ' ' | tr '[a-f]' '[A-F]'`
> # convert from hexadecimal to decimal:
> dec=`echo "ibase=16; $hex" | bc`
> # echo >&2 "DEBUG: hex=<$hex>; dec=<$dec>"
> dec=$(( dec % maxId))
> #echo "$dec"
> start=$(date +%s%N)
> curl -silent http://server.com:8080/table/$dec > /dev/null
> elapsed=$(($(date +%s%N) - $start))
> echo $elapsed
>done
>
>I have another script like
>
>for i in {1..100}
>do
> ksh myScript >> logfile &
>done
>
>
>I will try jstack to see if I can find anything, thanks for the suggestion.
>
>
>----- Original Message -----
>From: Andrew Purtell <ap...@apache.org>
>To: "user@hbase.apache.org" <us...@hbase.apache.org>
>Cc:
>Sent: Tuesday, January 17, 2012 5:48 PM
>Subject: Re: HBase 0.92rc3 rest performance
>
>jstacks could help point out if the REST server has some internal lock or monitor contention.
>
>Internal to the REST server is just the HBase Java client. REST uses a HTablePool to manage a small pool of HTable instances that interact with the cluster according to the request at hand. Client changes could show up in REST but it would be odd (and possibly a REST bug) to see them only there.
>
>Other things to check:
>
> - What version of Jetty?
>
> - What version of Jersey?
>
> - Any WARNs in logs from the REST server?
>
>Reproducing this would be as simple as starting up 50 or so concurrent fetches?
>
>Best regards,
>
>
> - Andy
>
>Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>----- Original Message -----
>> From: Jean-Daniel Cryans <jd...@apache.org>
>> To: user@hbase.apache.org
>> Cc:
>> Sent: Tuesday, January 17, 2012 1:45 PM
>> Subject: Re: HBase 0.92rc3 rest performance
>>
>>T his seems to single out the REST server since thrift and native
>> clients stayed the same.
>>
>> Can you provide us your test so we can do testing on our side too?
>>
>> Maybe doing a few jstacks on the REST server could point out the
>> obvious bottlenecks.
>>
>> J-D
>>
>> On Tue, Jan 17, 2012 at 11:25 AM, Ben West <bw...@yahoo.com>
>> wrote:
>>> Hi all
>>>
>>> We're trying out .92rc3 instead of .90.4, and for the most part
>> everything seems fine. But we have a simple test of REST performance which is
>> basically a large number of cURL jobs getting random rows, and this test is
>> running *a lot* slower under .92.
>>>
>>> When we run just a single client doing REST GETs, the performance is fine.
>> But once I have dozens or hundreds of clients, performance is ~20x worse than
>> under .90.4 (response time is 7-800ms instead of 40-50ms).
>>>
>>> YCSB has pretty much the same performance under both versions, as do other
>> internal tools measuring Thrift and native performance, so I don't feel like
>> this is a problem with HBase coresetup (although it could be). I don't see
>> anything suspicious in any logs, IO and CPU utilization are both low. Has anyone
>> run into this or have thoughts on how to troubleshoot?
>>>
>>> Thanks!
>>> Ben
>>
>
>
>
>
Re: HBase 0.92rc3 rest performance
Posted by Ben West <bw...@yahoo.com>.
Thanks for the quick responses!
No higher CPU or memory usage as far as I can tell. No WARNs. We're using the default Jersey (1.4) and Jetty (6.1.26).
Yes, you only need to start up some number of concurrent GETs. Here is my test script, if that helps. It is quite simple (in ksh):
maxId=274894038
for i in {1..1000}
do
# get a random number
hex=`dd if=/dev/urandom bs=1 count=8 2>/dev/null |
od -tx1 | head -1 | cut -d' ' -f2- |
tr -d ' ' | tr '[a-f]' '[A-F]'`
# convert from hexadecimal to decimal:
dec=`echo "ibase=16; $hex" | bc`
# echo >&2 "DEBUG: hex=<$hex>; dec=<$dec>"
dec=$(( dec % maxId))
#echo "$dec"
start=$(date +%s%N)
curl -silent http://server.com:8080/table/$dec > /dev/null
elapsed=$(($(date +%s%N) - $start))
echo $elapsed
done
I have another script like
for i in {1..100}
do
ksh myScript >> logfile &
done
I will try jstack to see if I can find anything, thanks for the suggestion.
----- Original Message -----
From: Andrew Purtell <ap...@apache.org>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc:
Sent: Tuesday, January 17, 2012 5:48 PM
Subject: Re: HBase 0.92rc3 rest performance
jstacks could help point out if the REST server has some internal lock or monitor contention.
Internal to the REST server is just the HBase Java client. REST uses a HTablePool to manage a small pool of HTable instances that interact with the cluster according to the request at hand. Client changes could show up in REST but it would be odd (and possibly a REST bug) to see them only there.
Other things to check:
- What version of Jetty?
- What version of Jersey?
- Any WARNs in logs from the REST server?
Reproducing this would be as simple as starting up 50 or so concurrent fetches?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Cc:
> Sent: Tuesday, January 17, 2012 1:45 PM
> Subject: Re: HBase 0.92rc3 rest performance
>
>T his seems to single out the REST server since thrift and native
> clients stayed the same.
>
> Can you provide us your test so we can do testing on our side too?
>
> Maybe doing a few jstacks on the REST server could point out the
> obvious bottlenecks.
>
> J-D
>
> On Tue, Jan 17, 2012 at 11:25 AM, Ben West <bw...@yahoo.com>
> wrote:
>> Hi all
>>
>> We're trying out .92rc3 instead of .90.4, and for the most part
> everything seems fine. But we have a simple test of REST performance which is
> basically a large number of cURL jobs getting random rows, and this test is
> running *a lot* slower under .92.
>>
>> When we run just a single client doing REST GETs, the performance is fine.
> But once I have dozens or hundreds of clients, performance is ~20x worse than
> under .90.4 (response time is 7-800ms instead of 40-50ms).
>>
>> YCSB has pretty much the same performance under both versions, as do other
> internal tools measuring Thrift and native performance, so I don't feel like
> this is a problem with HBase coresetup (although it could be). I don't see
> anything suspicious in any logs, IO and CPU utilization are both low. Has anyone
> run into this or have thoughts on how to troubleshoot?
>>
>> Thanks!
>> Ben
>
Re: HBase 0.92rc3 rest performance
Posted by Andrew Purtell <ap...@apache.org>.
jstacks could help point out if the REST server has some internal lock or monitor contention.
Internal to the REST server is just the HBase Java client. REST uses a HTablePool to manage a small pool of HTable instances that interact with the cluster according to the request at hand. Client changes could show up in REST but it would be odd (and possibly a REST bug) to see them only there.
Other things to check:
- What version of Jetty?
- What version of Jersey?
- Any WARNs in logs from the REST server?
Reproducing this would be as simple as starting up 50 or so concurrent fetches?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Cc:
> Sent: Tuesday, January 17, 2012 1:45 PM
> Subject: Re: HBase 0.92rc3 rest performance
>
>T his seems to single out the REST server since thrift and native
> clients stayed the same.
>
> Can you provide us your test so we can do testing on our side too?
>
> Maybe doing a few jstacks on the REST server could point out the
> obvious bottlenecks.
>
> J-D
>
> On Tue, Jan 17, 2012 at 11:25 AM, Ben West <bw...@yahoo.com>
> wrote:
>> Hi all
>>
>> We're trying out .92rc3 instead of .90.4, and for the most part
> everything seems fine. But we have a simple test of REST performance which is
> basically a large number of cURL jobs getting random rows, and this test is
> running *a lot* slower under .92.
>>
>> When we run just a single client doing REST GETs, the performance is fine.
> But once I have dozens or hundreds of clients, performance is ~20x worse than
> under .90.4 (response time is 7-800ms instead of 40-50ms).
>>
>> YCSB has pretty much the same performance under both versions, as do other
> internal tools measuring Thrift and native performance, so I don't feel like
> this is a problem with HBase coresetup (although it could be). I don't see
> anything suspicious in any logs, IO and CPU utilization are both low. Has anyone
> run into this or have thoughts on how to troubleshoot?
>>
>> Thanks!
>> Ben
>
Re: HBase 0.92rc3 rest performance
Posted by Jean-Daniel Cryans <jd...@apache.org>.
This seems to single out the REST server since thrift and native
clients stayed the same.
Can you provide us your test so we can do testing on our side too?
Maybe doing a few jstacks on the REST server could point out the
obvious bottlenecks.
J-D
On Tue, Jan 17, 2012 at 11:25 AM, Ben West <bw...@yahoo.com> wrote:
> Hi all
>
> We're trying out .92rc3 instead of .90.4, and for the most part everything seems fine. But we have a simple test of REST performance which is basically a large number of cURL jobs getting random rows, and this test is running *a lot* slower under .92.
>
> When we run just a single client doing REST GETs, the performance is fine. But once I have dozens or hundreds of clients, performance is ~20x worse than under .90.4 (response time is 7-800ms instead of 40-50ms).
>
> YCSB has pretty much the same performance under both versions, as do other internal tools measuring Thrift and native performance, so I don't feel like this is a problem with HBase coresetup (although it could be). I don't see anything suspicious in any logs, IO and CPU utilization are both low. Has anyone run into this or have thoughts on how to troubleshoot?
>
> Thanks!
> Ben
Re: HBase 0.92rc3 rest performance
Posted by lars hofhansl <lh...@yahoo.com>.
Do you see different CPU/Memory utilization?
Where do things differ? Client or server?
Thanks.
-- Lars
----- Original Message -----
From: Ben West <bw...@yahoo.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc:
Sent: Tuesday, January 17, 2012 11:25 AM
Subject: HBase 0.92rc3 rest performance
Hi all
We're trying out .92rc3 instead of .90.4, and for the most part everything seems fine. But we have a simple test of REST performance which is basically a large number of cURL jobs getting random rows, and this test is running *a lot* slower under .92.
When we run just a single client doing REST GETs, the performance is fine. But once I have dozens or hundreds of clients, performance is ~20x worse than under .90.4 (response time is 7-800ms instead of 40-50ms).
YCSB has pretty much the same performance under both versions, as do other internal tools measuring Thrift and native performance, so I don't feel like this is a problem with HBase coresetup (although it could be). I don't see anything suspicious in any logs, IO and CPU utilization are both low. Has anyone run into this or have thoughts on how to troubleshoot?
Thanks!
Ben