You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kristoffer Sjögren <st...@gmail.com> on 2013/12/21 20:17:09 UTC
Performance tuning
Hi
I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
couple of days and need some help.
Background.
- 23 machine cluster, 32 cores, 4GB heap per RS.
- Table t_24 have 24 online regions (24 salt buckets).
- Table t_96 have 96 online regions (96 salt buckets).
- 10.5 million rows per table.
- Count query - select (*) from ...
- Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
and T <= 2147482800) group by A, B, C;
What I found ultimately is that region servers 19, 20, 21, 22 and 23
are consistently
2-3x slower than the others. This hurts overall latency pretty bad since
queries are executed in parallel on the RS and then aggregated at the
client (through Phoenix). In Hannibal regions spread out evenly over region
servers, according to salt buckets (phoenix feature, pre-create regions and
a rowkey prefix).
As far as I can tell, there is no network or hardware configuration
divergence between the machines. No CPU, network or other notable divergence
in Ganglia. No RS metric differences in HBase master console.
The only thing that may be of interest is that pings (within the cluster) to
bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
this is significant,
but I get a bad feeling about it since it match exactly with the RS that
stood out in my performance tests.
Any ideas of how I might find the source of this problem?
Cheers,
-Kristoffer
RE: Performance tuning
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
You will get better overall performance by increasing # of regions (salt buckets)
With 23 x 32 cpu I would try 300 regions at least.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: James Taylor [jtaylor@salesforce.com]
Sent: Saturday, December 21, 2013 3:30 PM
To: user@hbase.apache.org
Cc: lars hofhansl
Subject: Re: Performance tuning
FYI, scanner caching defaults to 1000 in Phoenix, but as folks have pointed
out, that's not relevant in this case b/c only a single row is returned
from the server for a COUNT(*) query.
On Sat, Dec 21, 2013 at 2:51 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Yeah, im doing a count(*) query on the 96 region table. Do you mean to
> check network traffic between RS?
>
> From debugging phoenix code I can see that there are 96 scans sent and each
> response returned back to the client contain only the sum of rows, which
> are then aggregated and returned. So the traffic between client and each RS
> is very small.
>
>
>
>
> On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Thanks Kristoffer,
> >
> > yeah, that's the right metric. I would put my bet on the slower network.
> > But you're also doing a select count(*) query in Phoenix, right? So
> > nothing should really be sent across the network.
> >
> > When you do the queries, can you check whether there is any network
> > traffic?
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Kristoffer Sjögren <st...@gmail.com>
> > To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> > Sent: Saturday, December 21, 2013 1:28 PM
> > Subject: Re: Performance tuning
> >
> >
> > @pradeep scanner caching should not be an issue since data transferred to
> > the client is tiny.
> >
> > @lars Yes, the data might be small for this particular case :-)
> >
> > I have checked everything I can think of on RS (CPU, network, Hbase
> > console, uptime etc) and nothing stands out, except for the pings
> (network
> > pings).
> > There are 5 regions on 7, 18, 19, and 23 the others have 4.
> > hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
> >
> > -Kristoffer
> >
> >
> >
> >
> > On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
> >
> > > Hi Kristoffer,
> > > For this particular problem. Are many regions on the same
> RegionServers?
> > > Did you profile those RegionServers? Anything weird on that box?
> > > Pings slower might well be an issue. How's the data locality? (You can
> > > check on a RegionServer's overview page).
> > > If needed, you can issue a major compaction to reestablish local data
> on
> > > all RegionServers.
> > >
> > >
> > > 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
> > > dataset it doesn't matter anyway.
> > >
> > > 10m rows across 96 regions is just about 100k rows per region. You
> won't
> > > see many of the nice properties for HBase.
> > > Try with 100m (or better 1bn rows). Then we're talking. For anything
> > below
> > > this you wouldn't want to use HBase anyway.
> > > (100k rows I could scan on my phone with a Perl script in less than 1s)
> > >
> > >
> > > With "ping" you mean an actual network ping, or some operation on top
> of
> > > HBase?
> > >
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Kristoffer Sjögren <st...@gmail.com>
> > > To: user@hbase.apache.org
> > > Sent: Saturday, December 21, 2013 11:17 AM
> > > Subject: Performance tuning
> > >
> > >
> > > Hi
> > >
> > > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
> > last
> > > couple of days and need some help.
> > >
> > > Background.
> > >
> > > - 23 machine cluster, 32 cores, 4GB heap per RS.
> > > - Table t_24 have 24 online regions (24 salt buckets).
> > > - Table t_96 have 96 online regions (96 salt buckets).
> > > - 10.5 million rows per table.
> > > - Count query - select (*) from ...
> > > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T
> >= 0
> > > and T <= 2147482800) group by A, B, C;
> > >
> > > What I found ultimately is that region servers 19, 20, 21, 22 and 23
> > > are consistently
> > > 2-3x slower than the others. This hurts overall latency pretty bad
> since
> > > queries are executed in parallel on the RS and then aggregated at the
> > > client (through Phoenix). In Hannibal regions spread out evenly over
> > region
> > > servers, according to salt buckets (phoenix feature, pre-create regions
> > and
> > > a rowkey prefix).
> > >
> > > As far as I can tell, there is no network or hardware configuration
> > > divergence between the machines. No CPU, network or other notable
> > > divergence
> > > in Ganglia. No RS metric differences in HBase master console.
> > >
> > > The only thing that may be of interest is that pings (within the
> cluster)
> > > to
> > > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> > > this is significant,
> > > but I get a bad feeling about it since it match exactly with the RS
> that
> > > stood out in my performance tests.
> > >
> > > Any ideas of how I might find the source of this problem?
> > >
> > > Cheers,
> > > -Kristoffer
> > >
> >
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Performance tuning
Posted by James Taylor <jt...@salesforce.com>.
FYI, scanner caching defaults to 1000 in Phoenix, but as folks have pointed
out, that's not relevant in this case b/c only a single row is returned
from the server for a COUNT(*) query.
On Sat, Dec 21, 2013 at 2:51 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Yeah, im doing a count(*) query on the 96 region table. Do you mean to
> check network traffic between RS?
>
> From debugging phoenix code I can see that there are 96 scans sent and each
> response returned back to the client contain only the sum of rows, which
> are then aggregated and returned. So the traffic between client and each RS
> is very small.
>
>
>
>
> On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Thanks Kristoffer,
> >
> > yeah, that's the right metric. I would put my bet on the slower network.
> > But you're also doing a select count(*) query in Phoenix, right? So
> > nothing should really be sent across the network.
> >
> > When you do the queries, can you check whether there is any network
> > traffic?
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Kristoffer Sjögren <st...@gmail.com>
> > To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> > Sent: Saturday, December 21, 2013 1:28 PM
> > Subject: Re: Performance tuning
> >
> >
> > @pradeep scanner caching should not be an issue since data transferred to
> > the client is tiny.
> >
> > @lars Yes, the data might be small for this particular case :-)
> >
> > I have checked everything I can think of on RS (CPU, network, Hbase
> > console, uptime etc) and nothing stands out, except for the pings
> (network
> > pings).
> > There are 5 regions on 7, 18, 19, and 23 the others have 4.
> > hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
> >
> > -Kristoffer
> >
> >
> >
> >
> > On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
> >
> > > Hi Kristoffer,
> > > For this particular problem. Are many regions on the same
> RegionServers?
> > > Did you profile those RegionServers? Anything weird on that box?
> > > Pings slower might well be an issue. How's the data locality? (You can
> > > check on a RegionServer's overview page).
> > > If needed, you can issue a major compaction to reestablish local data
> on
> > > all RegionServers.
> > >
> > >
> > > 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
> > > dataset it doesn't matter anyway.
> > >
> > > 10m rows across 96 regions is just about 100k rows per region. You
> won't
> > > see many of the nice properties for HBase.
> > > Try with 100m (or better 1bn rows). Then we're talking. For anything
> > below
> > > this you wouldn't want to use HBase anyway.
> > > (100k rows I could scan on my phone with a Perl script in less than 1s)
> > >
> > >
> > > With "ping" you mean an actual network ping, or some operation on top
> of
> > > HBase?
> > >
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Kristoffer Sjögren <st...@gmail.com>
> > > To: user@hbase.apache.org
> > > Sent: Saturday, December 21, 2013 11:17 AM
> > > Subject: Performance tuning
> > >
> > >
> > > Hi
> > >
> > > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
> > last
> > > couple of days and need some help.
> > >
> > > Background.
> > >
> > > - 23 machine cluster, 32 cores, 4GB heap per RS.
> > > - Table t_24 have 24 online regions (24 salt buckets).
> > > - Table t_96 have 96 online regions (96 salt buckets).
> > > - 10.5 million rows per table.
> > > - Count query - select (*) from ...
> > > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T
> >= 0
> > > and T <= 2147482800) group by A, B, C;
> > >
> > > What I found ultimately is that region servers 19, 20, 21, 22 and 23
> > > are consistently
> > > 2-3x slower than the others. This hurts overall latency pretty bad
> since
> > > queries are executed in parallel on the RS and then aggregated at the
> > > client (through Phoenix). In Hannibal regions spread out evenly over
> > region
> > > servers, according to salt buckets (phoenix feature, pre-create regions
> > and
> > > a rowkey prefix).
> > >
> > > As far as I can tell, there is no network or hardware configuration
> > > divergence between the machines. No CPU, network or other notable
> > > divergence
> > > in Ganglia. No RS metric differences in HBase master console.
> > >
> > > The only thing that may be of interest is that pings (within the
> cluster)
> > > to
> > > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> > > this is significant,
> > > but I get a bad feeling about it since it match exactly with the RS
> that
> > > stood out in my performance tests.
> > >
> > > Any ideas of how I might find the source of this problem?
> > >
> > > Cheers,
> > > -Kristoffer
> > >
> >
>
Re: Performance tuning
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Most probably a question for the Phoenix mailing list...
Le 2013-12-28 15:40, "Asaf Mesika" <as...@gmail.com> a écrit :
> Does Phoenix exposes metrics about its code execution? Network time,
> coprocessor time, client time, etc?
>
> On Sunday, December 22, 2013, lars hofhansl wrote:
>
> > You would have to measure the incoming/outgoing traffic on the affected
> > machine.
> >
> > The easiest is to periodically check the output of ifconfig. If all data
> > is local and the query just returns a count I would not expect much
> (any?)
> > network traffic even after you ran the query multiple times.
> >
> > Beyond that I can't think of anything further. You said you checked the
> > machines, they are configured the same, etc.
> > If you run any local benchmarks on the boxes (any benchmark will do), do
> > they really perform all the same?
> >
> > Lastly, I assume all the regions are of the same size, again, check on
> the
> > regionserver UI pages (or maybe Hannibal tells you?)
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Kristoffer Sjögren <stoffe@gmail.com <javascript:;>>
> > To: user@hbase.apache.org <javascript:;>; lars hofhansl <
> larsh@apache.org<javascript:;>
> > >
> > Sent: Saturday, December 21, 2013 3:17 PM
> > Subject: Re: Performance tuning
> >
> >
> >
> > There are quite a lot of established and time wait connections between
> the
> > RS on port 50010, but i dont know a good way of monitoring how much data
> is
> > going through each connection (if that's what you meant)?
> >
> >
> >
> > On Sun, Dec 22, 2013 at 12:00 AM, Kristoffer Sjögren <st...@gmail.com>
> > wrote:
> >
> > Scans on RS 19 and 23, which have 5 regions instead of 4, stands out more
> > than scans on RS 20, 21, 22. But scans on RS 7 and 18, that also have 5
> > regions are doing fine, not best, but still in the mid-range.
> > >
> > >
> > >
> > >On Sat, Dec 21, 2013 at 11:51 PM, Kristoffer Sjögren <st...@gmail.com>
> > wrote:
> > >
> > >Yeah, im doing a count(*) query on the 96 region table. Do you mean to
> > check network traffic between RS?
> > >>
> > >>
> > >>From debugging phoenix code I can see that there are 96 scans sent and
> > each response returned back to the client contain only the sum of rows,
> > which are then aggregated and returned. So the traffic between client and
> > each RS is very small.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >>
> > >>Thanks Kristoffer,
> > >>>
> > >>>yeah, that's the right metric. I would put my bet on the slower
> network.
> > >>>But you're also doing a select count(*) query in Phoenix, right? So
> > nothing should really be sent across the network.
> > >>>
> > >>>When you do the queries, can you check whether there is any network
> > traffic?
> > >>>
> > >>>
> > >>>-- Lars
> > >>>
> > >>>
> > >>>
> > >>>________________________________
> > >>> From: Kristoffer Sjögren <st...@gmail.com>
> > >>>To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> > >>>Sent: Saturday, December 21, 2013 1:28 PM
> > >>>Subject: Re: Performance tuning
> > >>>
> > >>>
> > >>>
> > >>>@pradeep scanner caching should not be an issue since data transferred
> > to
> > >>>the client is tiny.
> > >>>
> > >>>@lars Yes, the data might be small for this particular case :-)
> > >>>
> > >>>I have checked everything I can think of on RS (CPU, network, Hbase
> > >>>console, uptime etc) and nothing stands out, except for the pings
> > (network
> > >>>pings).
> > >>>There are 5 regions on 7, 18, 19, and 23 the others have 4.
> > >>>hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
> > >>>
> > >>>-Kristoffer
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >>>
> > >>>> Hi Kristoffer,
> > >>>> For this particular problem. Are many regions on the same
> > RegionServers?
> > >>>> Did you profile those RegionServers? Anything weird on that box?
> > >>>> Pings slower might well be an issue. How's the data locality? (You
> can
> > >>>> check on a RegionServer's overview page).
> > >>>> If needed, you can issue a major compaction to reestablish local
> data
> > on
> > >>>> all RegionServers.
> > >>>>
> > >>>>
> > >>>> 32 cores matched with only 4G of RAM is a bit weird, but with your
> > tiny
> > >>>> dataset it doesn't matter anyway.
> > >>>>
> > >>>> 10m rows across 96 regions is just about 100k rows per region. You
> > won't
> > >>>> see many of the nice properties for HBase.
> > >>>> Try with 100m (or better 1bn rows). Then we're talking. For anything
> > below
> > >>>> this you wouldn't want to use HBase anyway.
> > >>>> (100k rows I could scan on my phone with a Perl script in less than
> > 1s)
> > >>>>
> > >>>>
> > >>>> With "ping" you mean an actual network ping, or some operation on
> top
> > of
> > >>>> HBase?
> > >>>>
> > >>>>
> > >>>> -- Lars
> > >>>>
> > >>>>
> >
>
Re: Performance tuning
Posted by Asaf Mesika <as...@gmail.com>.
Does Phoenix exposes metrics about its code execution? Network time,
coprocessor time, client time, etc?
On Sunday, December 22, 2013, lars hofhansl wrote:
> You would have to measure the incoming/outgoing traffic on the affected
> machine.
>
> The easiest is to periodically check the output of ifconfig. If all data
> is local and the query just returns a count I would not expect much (any?)
> network traffic even after you ran the query multiple times.
>
> Beyond that I can't think of anything further. You said you checked the
> machines, they are configured the same, etc.
> If you run any local benchmarks on the boxes (any benchmark will do), do
> they really perform all the same?
>
> Lastly, I assume all the regions are of the same size, again, check on the
> regionserver UI pages (or maybe Hannibal tells you?)
>
> -- Lars
>
>
>
> ________________________________
> From: Kristoffer Sjögren <stoffe@gmail.com <javascript:;>>
> To: user@hbase.apache.org <javascript:;>; lars hofhansl <larsh@apache.org<javascript:;>
> >
> Sent: Saturday, December 21, 2013 3:17 PM
> Subject: Re: Performance tuning
>
>
>
> There are quite a lot of established and time wait connections between the
> RS on port 50010, but i dont know a good way of monitoring how much data is
> going through each connection (if that's what you meant)?
>
>
>
> On Sun, Dec 22, 2013 at 12:00 AM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
>
> Scans on RS 19 and 23, which have 5 regions instead of 4, stands out more
> than scans on RS 20, 21, 22. But scans on RS 7 and 18, that also have 5
> regions are doing fine, not best, but still in the mid-range.
> >
> >
> >
> >On Sat, Dec 21, 2013 at 11:51 PM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
> >
> >Yeah, im doing a count(*) query on the 96 region table. Do you mean to
> check network traffic between RS?
> >>
> >>
> >>From debugging phoenix code I can see that there are 96 scans sent and
> each response returned back to the client contain only the sum of rows,
> which are then aggregated and returned. So the traffic between client and
> each RS is very small.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org>
> wrote:
> >>
> >>Thanks Kristoffer,
> >>>
> >>>yeah, that's the right metric. I would put my bet on the slower network.
> >>>But you're also doing a select count(*) query in Phoenix, right? So
> nothing should really be sent across the network.
> >>>
> >>>When you do the queries, can you check whether there is any network
> traffic?
> >>>
> >>>
> >>>-- Lars
> >>>
> >>>
> >>>
> >>>________________________________
> >>> From: Kristoffer Sjögren <st...@gmail.com>
> >>>To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> >>>Sent: Saturday, December 21, 2013 1:28 PM
> >>>Subject: Re: Performance tuning
> >>>
> >>>
> >>>
> >>>@pradeep scanner caching should not be an issue since data transferred
> to
> >>>the client is tiny.
> >>>
> >>>@lars Yes, the data might be small for this particular case :-)
> >>>
> >>>I have checked everything I can think of on RS (CPU, network, Hbase
> >>>console, uptime etc) and nothing stands out, except for the pings
> (network
> >>>pings).
> >>>There are 5 regions on 7, 18, 19, and 23 the others have 4.
> >>>hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
> >>>
> >>>-Kristoffer
> >>>
> >>>
> >>>
> >>>
> >>>On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org>
> wrote:
> >>>
> >>>> Hi Kristoffer,
> >>>> For this particular problem. Are many regions on the same
> RegionServers?
> >>>> Did you profile those RegionServers? Anything weird on that box?
> >>>> Pings slower might well be an issue. How's the data locality? (You can
> >>>> check on a RegionServer's overview page).
> >>>> If needed, you can issue a major compaction to reestablish local data
> on
> >>>> all RegionServers.
> >>>>
> >>>>
> >>>> 32 cores matched with only 4G of RAM is a bit weird, but with your
> tiny
> >>>> dataset it doesn't matter anyway.
> >>>>
> >>>> 10m rows across 96 regions is just about 100k rows per region. You
> won't
> >>>> see many of the nice properties for HBase.
> >>>> Try with 100m (or better 1bn rows). Then we're talking. For anything
> below
> >>>> this you wouldn't want to use HBase anyway.
> >>>> (100k rows I could scan on my phone with a Perl script in less than
> 1s)
> >>>>
> >>>>
> >>>> With "ping" you mean an actual network ping, or some operation on top
> of
> >>>> HBase?
> >>>>
> >>>>
> >>>> -- Lars
> >>>>
> >>>>
>
Re: Performance tuning
Posted by lars hofhansl <la...@apache.org>.
You would have to measure the incoming/outgoing traffic on the affected machine.
The easiest is to periodically check the output of ifconfig. If all data is local and the query just returns a count I would not expect much (any?) network traffic even after you ran the query multiple times.
Beyond that I can't think of anything further. You said you checked the machines, they are configured the same, etc.
If you run any local benchmarks on the boxes (any benchmark will do), do they really perform all the same?
Lastly, I assume all the regions are of the same size, again, check on the regionserver UI pages (or maybe Hannibal tells you?)
-- Lars
________________________________
From: Kristoffer Sjögren <st...@gmail.com>
To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Saturday, December 21, 2013 3:17 PM
Subject: Re: Performance tuning
There are quite a lot of established and time wait connections between the RS on port 50010, but i dont know a good way of monitoring how much data is going through each connection (if that's what you meant)?
On Sun, Dec 22, 2013 at 12:00 AM, Kristoffer Sjögren <st...@gmail.com> wrote:
Scans on RS 19 and 23, which have 5 regions instead of 4, stands out more than scans on RS 20, 21, 22. But scans on RS 7 and 18, that also have 5 regions are doing fine, not best, but still in the mid-range.
>
>
>
>On Sat, Dec 21, 2013 at 11:51 PM, Kristoffer Sjögren <st...@gmail.com> wrote:
>
>Yeah, im doing a count(*) query on the 96 region table. Do you mean to check network traffic between RS?
>>
>>
>>From debugging phoenix code I can see that there are 96 scans sent and each response returned back to the client contain only the sum of rows, which are then aggregated and returned. So the traffic between client and each RS is very small.
>>
>>
>>
>>
>>
>>
>>
>>On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org> wrote:
>>
>>Thanks Kristoffer,
>>>
>>>yeah, that's the right metric. I would put my bet on the slower network.
>>>But you're also doing a select count(*) query in Phoenix, right? So nothing should really be sent across the network.
>>>
>>>When you do the queries, can you check whether there is any network traffic?
>>>
>>>
>>>-- Lars
>>>
>>>
>>>
>>>________________________________
>>> From: Kristoffer Sjögren <st...@gmail.com>
>>>To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
>>>Sent: Saturday, December 21, 2013 1:28 PM
>>>Subject: Re: Performance tuning
>>>
>>>
>>>
>>>@pradeep scanner caching should not be an issue since data transferred to
>>>the client is tiny.
>>>
>>>@lars Yes, the data might be small for this particular case :-)
>>>
>>>I have checked everything I can think of on RS (CPU, network, Hbase
>>>console, uptime etc) and nothing stands out, except for the pings (network
>>>pings).
>>>There are 5 regions on 7, 18, 19, and 23 the others have 4.
>>>hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
>>>
>>>-Kristoffer
>>>
>>>
>>>
>>>
>>>On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
>>>
>>>> Hi Kristoffer,
>>>> For this particular problem. Are many regions on the same RegionServers?
>>>> Did you profile those RegionServers? Anything weird on that box?
>>>> Pings slower might well be an issue. How's the data locality? (You can
>>>> check on a RegionServer's overview page).
>>>> If needed, you can issue a major compaction to reestablish local data on
>>>> all RegionServers.
>>>>
>>>>
>>>> 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
>>>> dataset it doesn't matter anyway.
>>>>
>>>> 10m rows across 96 regions is just about 100k rows per region. You won't
>>>> see many of the nice properties for HBase.
>>>> Try with 100m (or better 1bn rows). Then we're talking. For anything below
>>>> this you wouldn't want to use HBase anyway.
>>>> (100k rows I could scan on my phone with a Perl script in less than 1s)
>>>>
>>>>
>>>> With "ping" you mean an actual network ping, or some operation on top of
>>>> HBase?
>>>>
>>>>
>>>> -- Lars
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Kristoffer Sjögren <st...@gmail.com>
>>>> To: user@hbase.apache.org
>>>> Sent: Saturday, December 21, 2013 11:17 AM
>>>> Subject: Performance tuning
>>>>
>>>>
>>>> Hi
>>>>
>>>> I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
>>>> couple of days and need some help.
>>>>
>>>> Background.
>>>>
>>>> - 23 machine cluster, 32 cores, 4GB heap per RS.
>>>> - Table t_24 have 24 online regions (24 salt buckets).
>>>> - Table t_96 have 96 online regions (96 salt buckets).
>>>> - 10.5 million rows per table.
>>>> - Count query - select (*) from ...
>>>> - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
>>>> and T <= 2147482800) group by A, B, C;
>>>>
>>>> What I found ultimately is that region servers 19, 20, 21, 22 and 23
>>>> are consistently
>>>> 2-3x slower than the others. This hurts overall latency pretty bad since
>>>> queries are executed in parallel on the RS and then aggregated at the
>>>> client (through Phoenix). In Hannibal regions spread out evenly over region
>>>> servers, according to salt buckets (phoenix feature, pre-create regions and
>>>> a rowkey prefix).
>>>>
>>>> As far as I can tell, there is no network or hardware configuration
>>>> divergence between the machines. No CPU, network or other notable
>>>> divergence
>>>> in Ganglia. No RS metric differences in HBase master console.
>>>>
>>>> The only thing that may be of interest is that pings (within the cluster)
>>>> to
>>>> bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
>>>> this is significant,
>>>> but I get a bad feeling about it since it match exactly with the RS that
>>>> stood out in my performance tests.
>>>>
>>>> Any ideas of how I might find the source of this problem?
>>>>
>>>> Cheers,
>>>> -Kristoffer
>>>>
>>
>
Re: Performance tuning
Posted by Kristoffer Sjögren <st...@gmail.com>.
There are quite a lot of established and time wait connections between the
RS on port 50010, but i dont know a good way of monitoring how much data is
going through each connection (if that's what you meant)?
On Sun, Dec 22, 2013 at 12:00 AM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Scans on RS 19 and 23, which have 5 regions instead of 4, stands out more
> than scans on RS 20, 21, 22. But scans on RS 7 and 18, that also have 5
> regions are doing fine, not best, but still in the mid-range.
>
>
> On Sat, Dec 21, 2013 at 11:51 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
>
>> Yeah, im doing a count(*) query on the 96 region table. Do you mean to
>> check network traffic between RS?
>>
>> From debugging phoenix code I can see that there are 96 scans sent and
>> each response returned back to the client contain only the sum of rows,
>> which are then aggregated and returned. So the traffic between client and
>> each RS is very small.
>>
>>
>>
>>
>> On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org> wrote:
>>
>>> Thanks Kristoffer,
>>>
>>> yeah, that's the right metric. I would put my bet on the slower network.
>>> But you're also doing a select count(*) query in Phoenix, right? So
>>> nothing should really be sent across the network.
>>>
>>> When you do the queries, can you check whether there is any network
>>> traffic?
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ________________________________
>>> From: Kristoffer Sjögren <st...@gmail.com>
>>> To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
>>> Sent: Saturday, December 21, 2013 1:28 PM
>>> Subject: Re: Performance tuning
>>>
>>>
>>> @pradeep scanner caching should not be an issue since data transferred to
>>> the client is tiny.
>>>
>>> @lars Yes, the data might be small for this particular case :-)
>>>
>>> I have checked everything I can think of on RS (CPU, network, Hbase
>>> console, uptime etc) and nothing stands out, except for the pings
>>> (network
>>> pings).
>>> There are 5 regions on 7, 18, 19, and 23 the others have 4.
>>> hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
>>>
>>> -Kristoffer
>>>
>>>
>>>
>>>
>>> On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
>>>
>>> > Hi Kristoffer,
>>> > For this particular problem. Are many regions on the same
>>> RegionServers?
>>> > Did you profile those RegionServers? Anything weird on that box?
>>> > Pings slower might well be an issue. How's the data locality? (You can
>>> > check on a RegionServer's overview page).
>>> > If needed, you can issue a major compaction to reestablish local data
>>> on
>>> > all RegionServers.
>>> >
>>> >
>>> > 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
>>> > dataset it doesn't matter anyway.
>>> >
>>> > 10m rows across 96 regions is just about 100k rows per region. You
>>> won't
>>> > see many of the nice properties for HBase.
>>> > Try with 100m (or better 1bn rows). Then we're talking. For anything
>>> below
>>> > this you wouldn't want to use HBase anyway.
>>> > (100k rows I could scan on my phone with a Perl script in less than 1s)
>>> >
>>> >
>>> > With "ping" you mean an actual network ping, or some operation on top
>>> of
>>> > HBase?
>>> >
>>> >
>>> > -- Lars
>>> >
>>> >
>>> >
>>> > ________________________________
>>> > From: Kristoffer Sjögren <st...@gmail.com>
>>> > To: user@hbase.apache.org
>>> > Sent: Saturday, December 21, 2013 11:17 AM
>>> > Subject: Performance tuning
>>> >
>>> >
>>> > Hi
>>> >
>>> > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
>>> last
>>> > couple of days and need some help.
>>> >
>>> > Background.
>>> >
>>> > - 23 machine cluster, 32 cores, 4GB heap per RS.
>>> > - Table t_24 have 24 online regions (24 salt buckets).
>>> > - Table t_96 have 96 online regions (96 salt buckets).
>>> > - 10.5 million rows per table.
>>> > - Count query - select (*) from ...
>>> > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T
>>> >= 0
>>> > and T <= 2147482800) group by A, B, C;
>>> >
>>> > What I found ultimately is that region servers 19, 20, 21, 22 and 23
>>> > are consistently
>>> > 2-3x slower than the others. This hurts overall latency pretty bad
>>> since
>>> > queries are executed in parallel on the RS and then aggregated at the
>>> > client (through Phoenix). In Hannibal regions spread out evenly over
>>> region
>>> > servers, according to salt buckets (phoenix feature, pre-create
>>> regions and
>>> > a rowkey prefix).
>>> >
>>> > As far as I can tell, there is no network or hardware configuration
>>> > divergence between the machines. No CPU, network or other notable
>>> > divergence
>>> > in Ganglia. No RS metric differences in HBase master console.
>>> >
>>> > The only thing that may be of interest is that pings (within the
>>> cluster)
>>> > to
>>> > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
>>> > this is significant,
>>> > but I get a bad feeling about it since it match exactly with the RS
>>> that
>>> > stood out in my performance tests.
>>> >
>>> > Any ideas of how I might find the source of this problem?
>>> >
>>> > Cheers,
>>> > -Kristoffer
>>> >
>>>
>>
>>
>
Re: Performance tuning
Posted by Kristoffer Sjögren <st...@gmail.com>.
Scans on RS 19 and 23, which have 5 regions instead of 4, stands out more
than scans on RS 20, 21, 22. But scans on RS 7 and 18, that also have 5
regions are doing fine, not best, but still in the mid-range.
On Sat, Dec 21, 2013 at 11:51 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Yeah, im doing a count(*) query on the 96 region table. Do you mean to
> check network traffic between RS?
>
> From debugging phoenix code I can see that there are 96 scans sent and
> each response returned back to the client contain only the sum of rows,
> which are then aggregated and returned. So the traffic between client and
> each RS is very small.
>
>
>
>
> On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org> wrote:
>
>> Thanks Kristoffer,
>>
>> yeah, that's the right metric. I would put my bet on the slower network.
>> But you're also doing a select count(*) query in Phoenix, right? So
>> nothing should really be sent across the network.
>>
>> When you do the queries, can you check whether there is any network
>> traffic?
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Kristoffer Sjögren <st...@gmail.com>
>> To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
>> Sent: Saturday, December 21, 2013 1:28 PM
>> Subject: Re: Performance tuning
>>
>>
>> @pradeep scanner caching should not be an issue since data transferred to
>> the client is tiny.
>>
>> @lars Yes, the data might be small for this particular case :-)
>>
>> I have checked everything I can think of on RS (CPU, network, Hbase
>> console, uptime etc) and nothing stands out, except for the pings (network
>> pings).
>> There are 5 regions on 7, 18, 19, and 23 the others have 4.
>> hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
>>
>> -Kristoffer
>>
>>
>>
>>
>> On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
>>
>> > Hi Kristoffer,
>> > For this particular problem. Are many regions on the same RegionServers?
>> > Did you profile those RegionServers? Anything weird on that box?
>> > Pings slower might well be an issue. How's the data locality? (You can
>> > check on a RegionServer's overview page).
>> > If needed, you can issue a major compaction to reestablish local data on
>> > all RegionServers.
>> >
>> >
>> > 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
>> > dataset it doesn't matter anyway.
>> >
>> > 10m rows across 96 regions is just about 100k rows per region. You won't
>> > see many of the nice properties for HBase.
>> > Try with 100m (or better 1bn rows). Then we're talking. For anything
>> below
>> > this you wouldn't want to use HBase anyway.
>> > (100k rows I could scan on my phone with a Perl script in less than 1s)
>> >
>> >
>> > With "ping" you mean an actual network ping, or some operation on top of
>> > HBase?
>> >
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ________________________________
>> > From: Kristoffer Sjögren <st...@gmail.com>
>> > To: user@hbase.apache.org
>> > Sent: Saturday, December 21, 2013 11:17 AM
>> > Subject: Performance tuning
>> >
>> >
>> > Hi
>> >
>> > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
>> last
>> > couple of days and need some help.
>> >
>> > Background.
>> >
>> > - 23 machine cluster, 32 cores, 4GB heap per RS.
>> > - Table t_24 have 24 online regions (24 salt buckets).
>> > - Table t_96 have 96 online regions (96 salt buckets).
>> > - 10.5 million rows per table.
>> > - Count query - select (*) from ...
>> > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >=
>> 0
>> > and T <= 2147482800) group by A, B, C;
>> >
>> > What I found ultimately is that region servers 19, 20, 21, 22 and 23
>> > are consistently
>> > 2-3x slower than the others. This hurts overall latency pretty bad since
>> > queries are executed in parallel on the RS and then aggregated at the
>> > client (through Phoenix). In Hannibal regions spread out evenly over
>> region
>> > servers, according to salt buckets (phoenix feature, pre-create regions
>> and
>> > a rowkey prefix).
>> >
>> > As far as I can tell, there is no network or hardware configuration
>> > divergence between the machines. No CPU, network or other notable
>> > divergence
>> > in Ganglia. No RS metric differences in HBase master console.
>> >
>> > The only thing that may be of interest is that pings (within the
>> cluster)
>> > to
>> > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
>> > this is significant,
>> > but I get a bad feeling about it since it match exactly with the RS that
>> > stood out in my performance tests.
>> >
>> > Any ideas of how I might find the source of this problem?
>> >
>> > Cheers,
>> > -Kristoffer
>> >
>>
>
>
Re: Performance tuning
Posted by Kristoffer Sjögren <st...@gmail.com>.
Yeah, im doing a count(*) query on the 96 region table. Do you mean to
check network traffic between RS?
>From debugging phoenix code I can see that there are 96 scans sent and each
response returned back to the client contain only the sum of rows, which
are then aggregated and returned. So the traffic between client and each RS
is very small.
On Sat, Dec 21, 2013 at 11:35 PM, lars hofhansl <la...@apache.org> wrote:
> Thanks Kristoffer,
>
> yeah, that's the right metric. I would put my bet on the slower network.
> But you're also doing a select count(*) query in Phoenix, right? So
> nothing should really be sent across the network.
>
> When you do the queries, can you check whether there is any network
> traffic?
>
> -- Lars
>
>
>
> ________________________________
> From: Kristoffer Sjögren <st...@gmail.com>
> To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Saturday, December 21, 2013 1:28 PM
> Subject: Re: Performance tuning
>
>
> @pradeep scanner caching should not be an issue since data transferred to
> the client is tiny.
>
> @lars Yes, the data might be small for this particular case :-)
>
> I have checked everything I can think of on RS (CPU, network, Hbase
> console, uptime etc) and nothing stands out, except for the pings (network
> pings).
> There are 5 regions on 7, 18, 19, and 23 the others have 4.
> hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
>
> -Kristoffer
>
>
>
>
> On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Hi Kristoffer,
> > For this particular problem. Are many regions on the same RegionServers?
> > Did you profile those RegionServers? Anything weird on that box?
> > Pings slower might well be an issue. How's the data locality? (You can
> > check on a RegionServer's overview page).
> > If needed, you can issue a major compaction to reestablish local data on
> > all RegionServers.
> >
> >
> > 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
> > dataset it doesn't matter anyway.
> >
> > 10m rows across 96 regions is just about 100k rows per region. You won't
> > see many of the nice properties for HBase.
> > Try with 100m (or better 1bn rows). Then we're talking. For anything
> below
> > this you wouldn't want to use HBase anyway.
> > (100k rows I could scan on my phone with a Perl script in less than 1s)
> >
> >
> > With "ping" you mean an actual network ping, or some operation on top of
> > HBase?
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Kristoffer Sjögren <st...@gmail.com>
> > To: user@hbase.apache.org
> > Sent: Saturday, December 21, 2013 11:17 AM
> > Subject: Performance tuning
> >
> >
> > Hi
> >
> > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
> last
> > couple of days and need some help.
> >
> > Background.
> >
> > - 23 machine cluster, 32 cores, 4GB heap per RS.
> > - Table t_24 have 24 online regions (24 salt buckets).
> > - Table t_96 have 96 online regions (96 salt buckets).
> > - 10.5 million rows per table.
> > - Count query - select (*) from ...
> > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
> > and T <= 2147482800) group by A, B, C;
> >
> > What I found ultimately is that region servers 19, 20, 21, 22 and 23
> > are consistently
> > 2-3x slower than the others. This hurts overall latency pretty bad since
> > queries are executed in parallel on the RS and then aggregated at the
> > client (through Phoenix). In Hannibal regions spread out evenly over
> region
> > servers, according to salt buckets (phoenix feature, pre-create regions
> and
> > a rowkey prefix).
> >
> > As far as I can tell, there is no network or hardware configuration
> > divergence between the machines. No CPU, network or other notable
> > divergence
> > in Ganglia. No RS metric differences in HBase master console.
> >
> > The only thing that may be of interest is that pings (within the cluster)
> > to
> > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> > this is significant,
> > but I get a bad feeling about it since it match exactly with the RS that
> > stood out in my performance tests.
> >
> > Any ideas of how I might find the source of this problem?
> >
> > Cheers,
> > -Kristoffer
> >
>
Re: Performance tuning
Posted by lars hofhansl <la...@apache.org>.
Thanks Kristoffer,
yeah, that's the right metric. I would put my bet on the slower network.
But you're also doing a select count(*) query in Phoenix, right? So nothing should really be sent across the network.
When you do the queries, can you check whether there is any network traffic?
-- Lars
________________________________
From: Kristoffer Sjögren <st...@gmail.com>
To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Saturday, December 21, 2013 1:28 PM
Subject: Re: Performance tuning
@pradeep scanner caching should not be an issue since data transferred to
the client is tiny.
@lars Yes, the data might be small for this particular case :-)
I have checked everything I can think of on RS (CPU, network, Hbase
console, uptime etc) and nothing stands out, except for the pings (network
pings).
There are 5 regions on 7, 18, 19, and 23 the others have 4.
hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
-Kristoffer
On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
> Hi Kristoffer,
> For this particular problem. Are many regions on the same RegionServers?
> Did you profile those RegionServers? Anything weird on that box?
> Pings slower might well be an issue. How's the data locality? (You can
> check on a RegionServer's overview page).
> If needed, you can issue a major compaction to reestablish local data on
> all RegionServers.
>
>
> 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
> dataset it doesn't matter anyway.
>
> 10m rows across 96 regions is just about 100k rows per region. You won't
> see many of the nice properties for HBase.
> Try with 100m (or better 1bn rows). Then we're talking. For anything below
> this you wouldn't want to use HBase anyway.
> (100k rows I could scan on my phone with a Perl script in less than 1s)
>
>
> With "ping" you mean an actual network ping, or some operation on top of
> HBase?
>
>
> -- Lars
>
>
>
> ________________________________
> From: Kristoffer Sjögren <st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Saturday, December 21, 2013 11:17 AM
> Subject: Performance tuning
>
>
> Hi
>
> I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
> couple of days and need some help.
>
> Background.
>
> - 23 machine cluster, 32 cores, 4GB heap per RS.
> - Table t_24 have 24 online regions (24 salt buckets).
> - Table t_96 have 96 online regions (96 salt buckets).
> - 10.5 million rows per table.
> - Count query - select (*) from ...
> - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
> and T <= 2147482800) group by A, B, C;
>
> What I found ultimately is that region servers 19, 20, 21, 22 and 23
> are consistently
> 2-3x slower than the others. This hurts overall latency pretty bad since
> queries are executed in parallel on the RS and then aggregated at the
> client (through Phoenix). In Hannibal regions spread out evenly over region
> servers, according to salt buckets (phoenix feature, pre-create regions and
> a rowkey prefix).
>
> As far as I can tell, there is no network or hardware configuration
> divergence between the machines. No CPU, network or other notable
> divergence
> in Ganglia. No RS metric differences in HBase master console.
>
> The only thing that may be of interest is that pings (within the cluster)
> to
> bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> this is significant,
> but I get a bad feeling about it since it match exactly with the RS that
> stood out in my performance tests.
>
> Any ideas of how I might find the source of this problem?
>
> Cheers,
> -Kristoffer
>
Re: Performance tuning
Posted by Kristoffer Sjögren <st...@gmail.com>.
Btw, I have tried different number of rows with similar symptom on the bad
RS.
On Sat, Dec 21, 2013 at 10:28 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
> @pradeep scanner caching should not be an issue since data transferred to
> the client is tiny.
>
> @lars Yes, the data might be small for this particular case :-)
>
> I have checked everything I can think of on RS (CPU, network, Hbase
> console, uptime etc) and nothing stands out, except for the pings (network
> pings).
> There are 5 regions on 7, 18, 19, and 23 the others have 4.
> hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
>
> -Kristoffer
>
>
>
> On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
>
>> Hi Kristoffer,
>> For this particular problem. Are many regions on the same RegionServers?
>> Did you profile those RegionServers? Anything weird on that box?
>> Pings slower might well be an issue. How's the data locality? (You can
>> check on a RegionServer's overview page).
>> If needed, you can issue a major compaction to reestablish local data on
>> all RegionServers.
>>
>>
>> 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
>> dataset it doesn't matter anyway.
>>
>> 10m rows across 96 regions is just about 100k rows per region. You won't
>> see many of the nice properties for HBase.
>> Try with 100m (or better 1bn rows). Then we're talking. For anything
>> below this you wouldn't want to use HBase anyway.
>> (100k rows I could scan on my phone with a Perl script in less than 1s)
>>
>>
>> With "ping" you mean an actual network ping, or some operation on top of
>> HBase?
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Kristoffer Sjögren <st...@gmail.com>
>> To: user@hbase.apache.org
>> Sent: Saturday, December 21, 2013 11:17 AM
>> Subject: Performance tuning
>>
>>
>> Hi
>>
>> I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
>> couple of days and need some help.
>>
>> Background.
>>
>> - 23 machine cluster, 32 cores, 4GB heap per RS.
>> - Table t_24 have 24 online regions (24 salt buckets).
>> - Table t_96 have 96 online regions (96 salt buckets).
>> - 10.5 million rows per table.
>> - Count query - select (*) from ...
>> - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
>> and T <= 2147482800) group by A, B, C;
>>
>> What I found ultimately is that region servers 19, 20, 21, 22 and 23
>> are consistently
>> 2-3x slower than the others. This hurts overall latency pretty bad since
>> queries are executed in parallel on the RS and then aggregated at the
>> client (through Phoenix). In Hannibal regions spread out evenly over
>> region
>> servers, according to salt buckets (phoenix feature, pre-create regions
>> and
>> a rowkey prefix).
>>
>> As far as I can tell, there is no network or hardware configuration
>> divergence between the machines. No CPU, network or other notable
>> divergence
>> in Ganglia. No RS metric differences in HBase master console.
>>
>> The only thing that may be of interest is that pings (within the cluster)
>> to
>> bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
>> this is significant,
>> but I get a bad feeling about it since it match exactly with the RS that
>> stood out in my performance tests.
>>
>> Any ideas of how I might find the source of this problem?
>>
>> Cheers,
>> -Kristoffer
>>
>
>
Re: Performance tuning
Posted by Kristoffer Sjögren <st...@gmail.com>.
@pradeep scanner caching should not be an issue since data transferred to
the client is tiny.
@lars Yes, the data might be small for this particular case :-)
I have checked everything I can think of on RS (CPU, network, Hbase
console, uptime etc) and nothing stands out, except for the pings (network
pings).
There are 5 regions on 7, 18, 19, and 23 the others have 4.
hdfsBlocksLocalityIndex=100 on all RS (was that the correct metric?)
-Kristoffer
On Sat, Dec 21, 2013 at 9:44 PM, lars hofhansl <la...@apache.org> wrote:
> Hi Kristoffer,
> For this particular problem. Are many regions on the same RegionServers?
> Did you profile those RegionServers? Anything weird on that box?
> Pings slower might well be an issue. How's the data locality? (You can
> check on a RegionServer's overview page).
> If needed, you can issue a major compaction to reestablish local data on
> all RegionServers.
>
>
> 32 cores matched with only 4G of RAM is a bit weird, but with your tiny
> dataset it doesn't matter anyway.
>
> 10m rows across 96 regions is just about 100k rows per region. You won't
> see many of the nice properties for HBase.
> Try with 100m (or better 1bn rows). Then we're talking. For anything below
> this you wouldn't want to use HBase anyway.
> (100k rows I could scan on my phone with a Perl script in less than 1s)
>
>
> With "ping" you mean an actual network ping, or some operation on top of
> HBase?
>
>
> -- Lars
>
>
>
> ________________________________
> From: Kristoffer Sjögren <st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Saturday, December 21, 2013 11:17 AM
> Subject: Performance tuning
>
>
> Hi
>
> I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
> couple of days and need some help.
>
> Background.
>
> - 23 machine cluster, 32 cores, 4GB heap per RS.
> - Table t_24 have 24 online regions (24 salt buckets).
> - Table t_96 have 96 online regions (96 salt buckets).
> - 10.5 million rows per table.
> - Count query - select (*) from ...
> - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
> and T <= 2147482800) group by A, B, C;
>
> What I found ultimately is that region servers 19, 20, 21, 22 and 23
> are consistently
> 2-3x slower than the others. This hurts overall latency pretty bad since
> queries are executed in parallel on the RS and then aggregated at the
> client (through Phoenix). In Hannibal regions spread out evenly over region
> servers, according to salt buckets (phoenix feature, pre-create regions and
> a rowkey prefix).
>
> As far as I can tell, there is no network or hardware configuration
> divergence between the machines. No CPU, network or other notable
> divergence
> in Ganglia. No RS metric differences in HBase master console.
>
> The only thing that may be of interest is that pings (within the cluster)
> to
> bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> this is significant,
> but I get a bad feeling about it since it match exactly with the RS that
> stood out in my performance tests.
>
> Any ideas of how I might find the source of this problem?
>
> Cheers,
> -Kristoffer
>
Re: Performance tuning
Posted by lars hofhansl <la...@apache.org>.
Hi Kristoffer,
For this particular problem. Are many regions on the same RegionServers? Did you profile those RegionServers? Anything weird on that box?
Pings slower might well be an issue. How's the data locality? (You can check on a RegionServer's overview page).
If needed, you can issue a major compaction to reestablish local data on all RegionServers.
32 cores matched with only 4G of RAM is a bit weird, but with your tiny dataset it doesn't matter anyway.
10m rows across 96 regions is just about 100k rows per region. You won't see many of the nice properties for HBase.
Try with 100m (or better 1bn rows). Then we're talking. For anything below this you wouldn't want to use HBase anyway.
(100k rows I could scan on my phone with a Perl script in less than 1s)
With "ping" you mean an actual network ping, or some operation on top of HBase?
-- Lars
________________________________
From: Kristoffer Sjögren <st...@gmail.com>
To: user@hbase.apache.org
Sent: Saturday, December 21, 2013 11:17 AM
Subject: Performance tuning
Hi
I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
couple of days and need some help.
Background.
- 23 machine cluster, 32 cores, 4GB heap per RS.
- Table t_24 have 24 online regions (24 salt buckets).
- Table t_96 have 96 online regions (96 salt buckets).
- 10.5 million rows per table.
- Count query - select (*) from ...
- Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
and T <= 2147482800) group by A, B, C;
What I found ultimately is that region servers 19, 20, 21, 22 and 23
are consistently
2-3x slower than the others. This hurts overall latency pretty bad since
queries are executed in parallel on the RS and then aggregated at the
client (through Phoenix). In Hannibal regions spread out evenly over region
servers, according to salt buckets (phoenix feature, pre-create regions and
a rowkey prefix).
As far as I can tell, there is no network or hardware configuration
divergence between the machines. No CPU, network or other notable divergence
in Ganglia. No RS metric differences in HBase master console.
The only thing that may be of interest is that pings (within the cluster) to
bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
this is significant,
but I get a bad feeling about it since it match exactly with the RS that
stood out in my performance tests.
Any ideas of how I might find the source of this problem?
Cheers,
-Kristoffer
Re: Performance tuning
Posted by Pradeep Gollakota <pr...@gmail.com>.
What is your scanner caching set to? I haven't worked with Phoenix so I'm
not sure what defaults if any it uses. In 0.94 HBase, I believe the default
caching is set to 1. This could be exacerbating your problem.
On Sat, Dec 21, 2013 at 7:52 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Yes, im waiting on a response from them. It's just.. the ping difference is
> tiny while the scan difference is huge, 2sec vs 4sec.
>
> Note the ping I mentioned is within the cluster. Ping from outside into the
> cluster have hardly any (if at all) noticeable difference.
>
>
> On Sat, Dec 21, 2013 at 8:37 PM, Pradeep Gollakota <pradeepg26@gmail.com
> >wrote:
>
> > Do you know if machines 19-23 are on a different rack? It seems to me
> that
> > your problem might be a networking problem. Whether it is hardware,
> > configuration or something else entirely, I'm not sure. It might be
> > worthwhile to talk to your systems administrator to see why pings to
> these
> > machines are slow. What are the pings like from a bad RS to another bad
> RS?
> >
> >
> > On Sat, Dec 21, 2013 at 7:17 PM, Kristoffer Sjögren <stoffe@gmail.com
> > >wrote:
> >
> > > Hi
> > >
> > > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
> > last
> > > couple of days and need some help.
> > >
> > > Background.
> > >
> > > - 23 machine cluster, 32 cores, 4GB heap per RS.
> > > - Table t_24 have 24 online regions (24 salt buckets).
> > > - Table t_96 have 96 online regions (96 salt buckets).
> > > - 10.5 million rows per table.
> > > - Count query - select (*) from ...
> > > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T
> >= 0
> > > and T <= 2147482800) group by A, B, C;
> > >
> > > What I found ultimately is that region servers 19, 20, 21, 22 and 23
> > > are consistently
> > > 2-3x slower than the others. This hurts overall latency pretty bad
> since
> > > queries are executed in parallel on the RS and then aggregated at the
> > > client (through Phoenix). In Hannibal regions spread out evenly over
> > region
> > > servers, according to salt buckets (phoenix feature, pre-create regions
> > and
> > > a rowkey prefix).
> > >
> > > As far as I can tell, there is no network or hardware configuration
> > > divergence between the machines. No CPU, network or other notable
> > > divergence
> > > in Ganglia. No RS metric differences in HBase master console.
> > >
> > > The only thing that may be of interest is that pings (within the
> cluster)
> > > to
> > > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> > > this is significant,
> > > but I get a bad feeling about it since it match exactly with the RS
> that
> > > stood out in my performance tests.
> > >
> > > Any ideas of how I might find the source of this problem?
> > >
> > > Cheers,
> > > -Kristoffer
> > >
> >
>
Re: Performance tuning
Posted by Kristoffer Sjögren <st...@gmail.com>.
Yes, im waiting on a response from them. It's just.. the ping difference is
tiny while the scan difference is huge, 2sec vs 4sec.
Note the ping I mentioned is within the cluster. Ping from outside into the
cluster have hardly any (if at all) noticeable difference.
On Sat, Dec 21, 2013 at 8:37 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Do you know if machines 19-23 are on a different rack? It seems to me that
> your problem might be a networking problem. Whether it is hardware,
> configuration or something else entirely, I'm not sure. It might be
> worthwhile to talk to your systems administrator to see why pings to these
> machines are slow. What are the pings like from a bad RS to another bad RS?
>
>
> On Sat, Dec 21, 2013 at 7:17 PM, Kristoffer Sjögren <stoffe@gmail.com
> >wrote:
>
> > Hi
> >
> > I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the
> last
> > couple of days and need some help.
> >
> > Background.
> >
> > - 23 machine cluster, 32 cores, 4GB heap per RS.
> > - Table t_24 have 24 online regions (24 salt buckets).
> > - Table t_96 have 96 online regions (96 salt buckets).
> > - 10.5 million rows per table.
> > - Count query - select (*) from ...
> > - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
> > and T <= 2147482800) group by A, B, C;
> >
> > What I found ultimately is that region servers 19, 20, 21, 22 and 23
> > are consistently
> > 2-3x slower than the others. This hurts overall latency pretty bad since
> > queries are executed in parallel on the RS and then aggregated at the
> > client (through Phoenix). In Hannibal regions spread out evenly over
> region
> > servers, according to salt buckets (phoenix feature, pre-create regions
> and
> > a rowkey prefix).
> >
> > As far as I can tell, there is no network or hardware configuration
> > divergence between the machines. No CPU, network or other notable
> > divergence
> > in Ganglia. No RS metric differences in HBase master console.
> >
> > The only thing that may be of interest is that pings (within the cluster)
> > to
> > bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> > this is significant,
> > but I get a bad feeling about it since it match exactly with the RS that
> > stood out in my performance tests.
> >
> > Any ideas of how I might find the source of this problem?
> >
> > Cheers,
> > -Kristoffer
> >
>
Re: Performance tuning
Posted by Pradeep Gollakota <pr...@gmail.com>.
Do you know if machines 19-23 are on a different rack? It seems to me that
your problem might be a networking problem. Whether it is hardware,
configuration or something else entirely, I'm not sure. It might be
worthwhile to talk to your systems administrator to see why pings to these
machines are slow. What are the pings like from a bad RS to another bad RS?
On Sat, Dec 21, 2013 at 7:17 PM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Hi
>
> I have been performance tuning HBase 0.94.6 running Phoenix 2.2.0 the last
> couple of days and need some help.
>
> Background.
>
> - 23 machine cluster, 32 cores, 4GB heap per RS.
> - Table t_24 have 24 online regions (24 salt buckets).
> - Table t_96 have 96 online regions (96 salt buckets).
> - 10.5 million rows per table.
> - Count query - select (*) from ...
> - Group by query - select A, B, C sum(D) from ... where (A = 1 and T >= 0
> and T <= 2147482800) group by A, B, C;
>
> What I found ultimately is that region servers 19, 20, 21, 22 and 23
> are consistently
> 2-3x slower than the others. This hurts overall latency pretty bad since
> queries are executed in parallel on the RS and then aggregated at the
> client (through Phoenix). In Hannibal regions spread out evenly over region
> servers, according to salt buckets (phoenix feature, pre-create regions and
> a rowkey prefix).
>
> As far as I can tell, there is no network or hardware configuration
> divergence between the machines. No CPU, network or other notable
> divergence
> in Ganglia. No RS metric differences in HBase master console.
>
> The only thing that may be of interest is that pings (within the cluster)
> to
> bad RS is about 2-3x slower, around 0.050ms vs 0.130ms. Not sure if
> this is significant,
> but I get a bad feeling about it since it match exactly with the RS that
> stood out in my performance tests.
>
> Any ideas of how I might find the source of this problem?
>
> Cheers,
> -Kristoffer
>