You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dejan Menges <de...@gmail.com> on 2015/05/04 10:31:40 UTC
Re: Right value for hbase.rpc.timeout
Hi Ted,
Max filesize for region is set to 75G in our case. Regarding split policy
we use most likely ConstantSizeRegionSplitPolicy
<http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html>
(it's
0.98.0 with bunch of patches and that should be default one).
Also, regarding link you sent me in 98.3 - I can not find anywhere what's
default value for hbase.regionserver.lease.period? Is this parameter still
called like this?
On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
> Please take a look at 98.3 under
> http://hbase.apache.org/book.html#trouble.client
>
> BTW what's the value for hbase.hregion.max.filesize ?
> Which split policy do you use ?
>
> Cheers
>
> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <de...@gmail.com>
> wrote:
>
> > Basically how I came to this question - this happened super rarely, and
> we
> > narrowed it down to hotspotting. Map was timing out on three regions
> which
> > were 4-5 times bigger then other regions for the same table, and region
> > split fixed this.
> >
> > However, was just thinking about if there are maybe some recommendations
> or
> > something about this, as it's also super hard to reproduce again same
> > situation to retest it.
> >
> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <michael_segel@hotmail.com
> >
> > wrote:
> >
> > > There is no single ‘right’ value.
> > >
> > > As you pointed out… some of your Mapper.map() iterations are taking
> > longer
> > > than 60 seconds.
> > >
> > > The first thing is to determine why that happens. (It could be normal,
> > or
> > > it could be bad code on your developers part. We don’t know.)
> > >
> > > The other thing is that if you determine that your code is perfect and
> it
> > > does what you want it to do… and its a major part of your use case… you
> > > then increase your timeouts to 120 seconds.
> > >
> > > The reason why its a tough issue is that we don’t know what hardware
> you
> > > are using. How many nodes… code quality.. etc … too many factors.
> > >
> > >
> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <de...@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > What's the best practice to calculate this value for your cluster, if
> > > there
> > > > is some?
> > > >
> > > > In some situations we saw that some maps are taking more than default
> > 60
> > > > seconds which was failing specific map job (as if it failed once, it
> > > failed
> > > > also every other time by number of configured retries).
> > > >
> > > > I would like to tune RPC parameters a bit, but googling and looking
> > into
> > > > HBase Book doesn't tell me how to calculate right values, and what
> else
> > > to
> > > > take a look beside hbase.rpc.timeout.
> > > >
> > > > Thanks a lot,
> > > > Dejan
> > >
> > >
> >
>
Re: Right value for hbase.rpc.timeout
Posted by Michael Segel <mi...@hotmail.com>.
Silly question…
How much memory do you have on your machine and how much do you allocate to HBase?
More to the point, how much memory is allocated to your memstore?
Suppose you have maxfilesize set to 100GB and your memstore is only 1GB in size. (round numbers to make the math easier…)
If we don’t have any flushes and only flush when the memstore is full, that would mean you would have ~100 or so HFiles for the region.
No?
How does that impact performance?
Or am I missing something?
> On May 5, 2015, at 3:01 AM, Dejan Menges <de...@gmail.com> wrote:
>
> Hi Lars,
>
> Regarding region sizes - it was kinda conclusion we got after reading bunch
> of articles trying to figure out what optimal region and memstore size for
> us would be in the process of migrating from 'old' cluster which was under
> very high load to this new and more performant one. Trying to find those
> articles, but not something I can quickly find again in five minutes, but
> clearly remember mentioning 100G as top limit, and then manually splitting
> if you see hotspotting or issues like that. So in the process, we set
> memstore to 256M and max region size 75G.
>
> Speaking about that, in table that was 'problematic' we actually didn't
> have any region bigger than 50G. Checking region sizes, I saw that, out of
> 250 regions there are ~20 between 40 and 50G, there were also ~30 regions
> between 20 and 40G, and all other were not bigger than 15G. I correlated it
> in one moment with number of mappers that fail, and when I started
> splitting manually biggest regions I saw that failing mappers are
> decreasing. Currently, same table don't have regions bigger than 30G, and
> all is good. This table is 900G in size.
>
> On another side, we have another table - 7.1T - where I see currently
> average region size of 40G, but usage pattern for this table is different,
> and that's why we never hit issue like this.
>
> And yeah, this cluster is configured for 600T of data, currently around 60%
> is used.
>
> Some cluster specific stuff I wouldn't put to the list, but I can send it
> directly to you if you are interested in it. Also every region server have
> 32G heap size and is collocated together with DataNode and NodeManager.
> Average off peak load is 20-25k requests per second, when it's really
> utilised it goes to 700k.
>
> So what would be your preferred value for region size? To leave it as
> default 10G, or eventually double it to 20G (what would in our case trigger
> region splitting on other tables and bigger number of regions)?
>
> On Mon, May 4, 2015 at 9:03 PM lars hofhansl <la...@apache.org> wrote:
>
>> Why do you have regions that large? The 0.92 default was 1G (admittedly,
>> that was much too small), the 0.98 default is 10G, which should be good in
>> most cases.Mappers divide their work based on regions, so very large region
>> lead to more uneven execution time, unless you truly have a a very large
>> amount of data.Compactions are in units of regions, etc.
>>
>> Can I ask how much data you have overall (i.e. how many of these 75G
>> regions you have)?
>>
>> Thanks.
>>
>> -- Lars
>> From: Dejan Menges <de...@gmail.com>
>> To: "user@hbase.apache.org" <us...@hbase.apache.org>
>> Sent: Monday, May 4, 2015 1:31 AM
>> Subject: Re: Right value for hbase.rpc.timeout
>>
>> Hi Ted,
>>
>> Max filesize for region is set to 75G in our case. Regarding split policy
>> we use most likely ConstantSizeRegionSplitPolicy
>> <
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html
>>>
>> (it's
>> 0.98.0 with bunch of patches and that should be default one).
>>
>> Also, regarding link you sent me in 98.3 - I can not find anywhere what's
>> default value for hbase.regionserver.lease.period? Is this parameter still
>> called like this?
>>
>>
>>
>> On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
>>
>>> Please take a look at 98.3 under
>>> http://hbase.apache.org/book.html#trouble.client
>>>
>>> BTW what's the value for hbase.hregion.max.filesize ?
>>> Which split policy do you use ?
>>>
>>> Cheers
>>>
>>> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <de...@gmail.com>
>>> wrote:
>>>
>>>> Basically how I came to this question - this happened super rarely, and
>>> we
>>>> narrowed it down to hotspotting. Map was timing out on three regions
>>> which
>>>> were 4-5 times bigger then other regions for the same table, and region
>>>> split fixed this.
>>>>
>>>> However, was just thinking about if there are maybe some
>> recommendations
>>> or
>>>> something about this, as it's also super hard to reproduce again same
>>>> situation to retest it.
>>>>
>>>> On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
>> michael_segel@hotmail.com
>>>>
>>>> wrote:
>>>>
>>>>> There is no single ‘right’ value.
>>>>>
>>>>> As you pointed out… some of your Mapper.map() iterations are taking
>>>> longer
>>>>> than 60 seconds.
>>>>>
>>>>> The first thing is to determine why that happens. (It could be
>> normal,
>>>> or
>>>>> it could be bad code on your developers part. We don’t know.)
>>>>>
>>>>> The other thing is that if you determine that your code is perfect
>> and
>>> it
>>>>> does what you want it to do… and its a major part of your use case…
>> you
>>>>> then increase your timeouts to 120 seconds.
>>>>>
>>>>> The reason why its a tough issue is that we don’t know what hardware
>>> you
>>>>> are using. How many nodes… code quality.. etc … too many factors.
>>>>>
>>>>>
>>>>>> On Apr 30, 2015, at 6:51 AM, Dejan Menges <de...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What's the best practice to calculate this value for your cluster,
>> if
>>>>> there
>>>>>> is some?
>>>>>>
>>>>>> In some situations we saw that some maps are taking more than
>> default
>>>> 60
>>>>>> seconds which was failing specific map job (as if it failed once,
>> it
>>>>> failed
>>>>>> also every other time by number of configured retries).
>>>>>>
>>>>>> I would like to tune RPC parameters a bit, but googling and looking
>>>> into
>>>>>> HBase Book doesn't tell me how to calculate right values, and what
>>> else
>>>>> to
>>>>>> take a look beside hbase.rpc.timeout.
>>>>>>
>>>>>> Thanks a lot,
>>>>>> Dejan
>>>>>
>>>>>
>>>>
>>>
>>
>>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Right value for hbase.rpc.timeout
Posted by Dejan Menges <de...@gmail.com>.
Hi Lars,
Regarding region sizes - it was kinda conclusion we got after reading bunch
of articles trying to figure out what optimal region and memstore size for
us would be in the process of migrating from 'old' cluster which was under
very high load to this new and more performant one. Trying to find those
articles, but not something I can quickly find again in five minutes, but
clearly remember mentioning 100G as top limit, and then manually splitting
if you see hotspotting or issues like that. So in the process, we set
memstore to 256M and max region size 75G.
Speaking about that, in table that was 'problematic' we actually didn't
have any region bigger than 50G. Checking region sizes, I saw that, out of
250 regions there are ~20 between 40 and 50G, there were also ~30 regions
between 20 and 40G, and all other were not bigger than 15G. I correlated it
in one moment with number of mappers that fail, and when I started
splitting manually biggest regions I saw that failing mappers are
decreasing. Currently, same table don't have regions bigger than 30G, and
all is good. This table is 900G in size.
On another side, we have another table - 7.1T - where I see currently
average region size of 40G, but usage pattern for this table is different,
and that's why we never hit issue like this.
And yeah, this cluster is configured for 600T of data, currently around 60%
is used.
Some cluster specific stuff I wouldn't put to the list, but I can send it
directly to you if you are interested in it. Also every region server have
32G heap size and is collocated together with DataNode and NodeManager.
Average off peak load is 20-25k requests per second, when it's really
utilised it goes to 700k.
So what would be your preferred value for region size? To leave it as
default 10G, or eventually double it to 20G (what would in our case trigger
region splitting on other tables and bigger number of regions)?
On Mon, May 4, 2015 at 9:03 PM lars hofhansl <la...@apache.org> wrote:
> Why do you have regions that large? The 0.92 default was 1G (admittedly,
> that was much too small), the 0.98 default is 10G, which should be good in
> most cases.Mappers divide their work based on regions, so very large region
> lead to more uneven execution time, unless you truly have a a very large
> amount of data.Compactions are in units of regions, etc.
>
> Can I ask how much data you have overall (i.e. how many of these 75G
> regions you have)?
>
> Thanks.
>
> -- Lars
> From: Dejan Menges <de...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Monday, May 4, 2015 1:31 AM
> Subject: Re: Right value for hbase.rpc.timeout
>
> Hi Ted,
>
> Max filesize for region is set to 75G in our case. Regarding split policy
> we use most likely ConstantSizeRegionSplitPolicy
> <
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html
> >
> (it's
> 0.98.0 with bunch of patches and that should be default one).
>
> Also, regarding link you sent me in 98.3 - I can not find anywhere what's
> default value for hbase.regionserver.lease.period? Is this parameter still
> called like this?
>
>
>
> On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
>
> > Please take a look at 98.3 under
> > http://hbase.apache.org/book.html#trouble.client
> >
> > BTW what's the value for hbase.hregion.max.filesize ?
> > Which split policy do you use ?
> >
> > Cheers
> >
> > On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <de...@gmail.com>
> > wrote:
> >
> > > Basically how I came to this question - this happened super rarely, and
> > we
> > > narrowed it down to hotspotting. Map was timing out on three regions
> > which
> > > were 4-5 times bigger then other regions for the same table, and region
> > > split fixed this.
> > >
> > > However, was just thinking about if there are maybe some
> recommendations
> > or
> > > something about this, as it's also super hard to reproduce again same
> > > situation to retest it.
> > >
> > > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
> michael_segel@hotmail.com
> > >
> > > wrote:
> > >
> > > > There is no single ‘right’ value.
> > > >
> > > > As you pointed out… some of your Mapper.map() iterations are taking
> > > longer
> > > > than 60 seconds.
> > > >
> > > > The first thing is to determine why that happens. (It could be
> normal,
> > > or
> > > > it could be bad code on your developers part. We don’t know.)
> > > >
> > > > The other thing is that if you determine that your code is perfect
> and
> > it
> > > > does what you want it to do… and its a major part of your use case…
> you
> > > > then increase your timeouts to 120 seconds.
> > > >
> > > > The reason why its a tough issue is that we don’t know what hardware
> > you
> > > > are using. How many nodes… code quality.. etc … too many factors.
> > > >
> > > >
> > > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <de...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > What's the best practice to calculate this value for your cluster,
> if
> > > > there
> > > > > is some?
> > > > >
> > > > > In some situations we saw that some maps are taking more than
> default
> > > 60
> > > > > seconds which was failing specific map job (as if it failed once,
> it
> > > > failed
> > > > > also every other time by number of configured retries).
> > > > >
> > > > > I would like to tune RPC parameters a bit, but googling and looking
> > > into
> > > > > HBase Book doesn't tell me how to calculate right values, and what
> > else
> > > > to
> > > > > take a look beside hbase.rpc.timeout.
> > > > >
> > > > > Thanks a lot,
> > > > > Dejan
> > > >
> > > >
> > >
> >
>
>
Re: Right value for hbase.rpc.timeout
Posted by lars hofhansl <la...@apache.org>.
Why do you have regions that large? The 0.92 default was 1G (admittedly, that was much too small), the 0.98 default is 10G, which should be good in most cases.Mappers divide their work based on regions, so very large region lead to more uneven execution time, unless you truly have a a very large amount of data.Compactions are in units of regions, etc.
Can I ask how much data you have overall (i.e. how many of these 75G regions you have)?
Thanks.
-- Lars
From: Dejan Menges <de...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, May 4, 2015 1:31 AM
Subject: Re: Right value for hbase.rpc.timeout
Hi Ted,
Max filesize for region is set to 75G in our case. Regarding split policy
we use most likely ConstantSizeRegionSplitPolicy
<http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html>
(it's
0.98.0 with bunch of patches and that should be default one).
Also, regarding link you sent me in 98.3 - I can not find anywhere what's
default value for hbase.regionserver.lease.period? Is this parameter still
called like this?
On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
> Please take a look at 98.3 under
> http://hbase.apache.org/book.html#trouble.client
>
> BTW what's the value for hbase.hregion.max.filesize ?
> Which split policy do you use ?
>
> Cheers
>
> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <de...@gmail.com>
> wrote:
>
> > Basically how I came to this question - this happened super rarely, and
> we
> > narrowed it down to hotspotting. Map was timing out on three regions
> which
> > were 4-5 times bigger then other regions for the same table, and region
> > split fixed this.
> >
> > However, was just thinking about if there are maybe some recommendations
> or
> > something about this, as it's also super hard to reproduce again same
> > situation to retest it.
> >
> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <michael_segel@hotmail.com
> >
> > wrote:
> >
> > > There is no single ‘right’ value.
> > >
> > > As you pointed out… some of your Mapper.map() iterations are taking
> > longer
> > > than 60 seconds.
> > >
> > > The first thing is to determine why that happens. (It could be normal,
> > or
> > > it could be bad code on your developers part. We don’t know.)
> > >
> > > The other thing is that if you determine that your code is perfect and
> it
> > > does what you want it to do… and its a major part of your use case… you
> > > then increase your timeouts to 120 seconds.
> > >
> > > The reason why its a tough issue is that we don’t know what hardware
> you
> > > are using. How many nodes… code quality.. etc … too many factors.
> > >
> > >
> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <de...@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > What's the best practice to calculate this value for your cluster, if
> > > there
> > > > is some?
> > > >
> > > > In some situations we saw that some maps are taking more than default
> > 60
> > > > seconds which was failing specific map job (as if it failed once, it
> > > failed
> > > > also every other time by number of configured retries).
> > > >
> > > > I would like to tune RPC parameters a bit, but googling and looking
> > into
> > > > HBase Book doesn't tell me how to calculate right values, and what
> else
> > > to
> > > > take a look beside hbase.rpc.timeout.
> > > >
> > > > Thanks a lot,
> > > > Dejan
> > >
> > >
> >
>
Re: Right value for hbase.rpc.timeout
Posted by Dejan Menges <de...@gmail.com>.
Hm, one thing we did - going back through thread - we decided to manually
set hbase.master.loadbalance.bytable to true, no matter that by
documentation it's true by default. And on our 0.98.0 (2.1.10 by
Hortonworks) it finally really balanced regions evenly by tables we had :/
On Mon, May 4, 2015 at 4:24 PM Dejan Menges <de...@gmail.com> wrote:
> Cool, thanks a lot! :)
>
> On Mon, May 4, 2015 at 4:14 PM Ted Yu <yu...@gmail.com> wrote:
>
>> Please see https://issues.apache.org/jira/browse/HBASE-10501 and
>> https://issues.apache.org/jira/browse/HBASE-10716
>>
>> Cheers
>>
>> On Mon, May 4, 2015 at 1:43 AM, Dejan Menges <de...@gmail.com>
>> wrote:
>>
>> > Sorry, maybe I'm wrong: I made my conclusion based on what I read
>> > http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
>> > that's default; now just found in HBase Book that:
>> >
>> > hbase.regionserver.region.split.policy
>> > Description
>> >
>> > A split policy determines when a region should be split. The various
>> other
>> > split policies that are available currently are
>> > ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy,
>> > DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.
>> > Default
>> >
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy
>> >
>> > Otherwise, I can not find any default set in any preinstalled config
>> files.
>> > So what's exactly the truth? :)
>> >
>> >
>> > On Mon, May 4, 2015 at 10:31 AM Dejan Menges <de...@gmail.com>
>> > wrote:
>> >
>> > > Hi Ted,
>> > >
>> > > Max filesize for region is set to 75G in our case. Regarding split
>> policy
>> > > we use most likely ConstantSizeRegionSplitPolicy
>> > > <
>> >
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html
>> >
>> > (it's
>> > > 0.98.0 with bunch of patches and that should be default one).
>> > >
>> > > Also, regarding link you sent me in 98.3 - I can not find anywhere
>> what's
>> > > default value for hbase.regionserver.lease.period? Is this parameter
>> > still
>> > > called like this?
>> > >
>> > > On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
>> > >
>> > >> Please take a look at 98.3 under
>> > >> http://hbase.apache.org/book.html#trouble.client
>> > >>
>> > >> BTW what's the value for hbase.hregion.max.filesize ?
>> > >> Which split policy do you use ?
>> > >>
>> > >> Cheers
>> > >>
>> > >> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <
>> dejan.menges@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Basically how I came to this question - this happened super rarely,
>> > and
>> > >> we
>> > >> > narrowed it down to hotspotting. Map was timing out on three
>> regions
>> > >> which
>> > >> > were 4-5 times bigger then other regions for the same table, and
>> > region
>> > >> > split fixed this.
>> > >> >
>> > >> > However, was just thinking about if there are maybe some
>> > >> recommendations or
>> > >> > something about this, as it's also super hard to reproduce again
>> same
>> > >> > situation to retest it.
>> > >> >
>> > >> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
>> > >> michael_segel@hotmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > There is no single ‘right’ value.
>> > >> > >
>> > >> > > As you pointed out… some of your Mapper.map() iterations are
>> taking
>> > >> > longer
>> > >> > > than 60 seconds.
>> > >> > >
>> > >> > > The first thing is to determine why that happens. (It could be
>> > >> normal,
>> > >> > or
>> > >> > > it could be bad code on your developers part. We don’t know.)
>> > >> > >
>> > >> > > The other thing is that if you determine that your code is
>> perfect
>> > >> and it
>> > >> > > does what you want it to do… and its a major part of your use
>> case…
>> > >> you
>> > >> > > then increase your timeouts to 120 seconds.
>> > >> > >
>> > >> > > The reason why its a tough issue is that we don’t know what
>> hardware
>> > >> you
>> > >> > > are using. How many nodes… code quality.. etc … too many factors.
>> > >> > >
>> > >> > >
>> > >> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <
>> dejan.menges@gmail.com
>> > >
>> > >> > > wrote:
>> > >> > > >
>> > >> > > > Hi,
>> > >> > > >
>> > >> > > > What's the best practice to calculate this value for your
>> cluster,
>> > >> if
>> > >> > > there
>> > >> > > > is some?
>> > >> > > >
>> > >> > > > In some situations we saw that some maps are taking more than
>> > >> default
>> > >> > 60
>> > >> > > > seconds which was failing specific map job (as if it failed
>> once,
>> > it
>> > >> > > failed
>> > >> > > > also every other time by number of configured retries).
>> > >> > > >
>> > >> > > > I would like to tune RPC parameters a bit, but googling and
>> > looking
>> > >> > into
>> > >> > > > HBase Book doesn't tell me how to calculate right values, and
>> what
>> > >> else
>> > >> > > to
>> > >> > > > take a look beside hbase.rpc.timeout.
>> > >> > > >
>> > >> > > > Thanks a lot,
>> > >> > > > Dejan
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: Right value for hbase.rpc.timeout
Posted by Dejan Menges <de...@gmail.com>.
Cool, thanks a lot! :)
On Mon, May 4, 2015 at 4:14 PM Ted Yu <yu...@gmail.com> wrote:
> Please see https://issues.apache.org/jira/browse/HBASE-10501 and
> https://issues.apache.org/jira/browse/HBASE-10716
>
> Cheers
>
> On Mon, May 4, 2015 at 1:43 AM, Dejan Menges <de...@gmail.com>
> wrote:
>
> > Sorry, maybe I'm wrong: I made my conclusion based on what I read
> > http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
> > that's default; now just found in HBase Book that:
> >
> > hbase.regionserver.region.split.policy
> > Description
> >
> > A split policy determines when a region should be split. The various
> other
> > split policies that are available currently are
> > ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy,
> > DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.
> > Default
> >
> >
> >
> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy
> >
> > Otherwise, I can not find any default set in any preinstalled config
> files.
> > So what's exactly the truth? :)
> >
> >
> > On Mon, May 4, 2015 at 10:31 AM Dejan Menges <de...@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > > Max filesize for region is set to 75G in our case. Regarding split
> policy
> > > we use most likely ConstantSizeRegionSplitPolicy
> > > <
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html
> >
> > (it's
> > > 0.98.0 with bunch of patches and that should be default one).
> > >
> > > Also, regarding link you sent me in 98.3 - I can not find anywhere
> what's
> > > default value for hbase.regionserver.lease.period? Is this parameter
> > still
> > > called like this?
> > >
> > > On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
> > >
> > >> Please take a look at 98.3 under
> > >> http://hbase.apache.org/book.html#trouble.client
> > >>
> > >> BTW what's the value for hbase.hregion.max.filesize ?
> > >> Which split policy do you use ?
> > >>
> > >> Cheers
> > >>
> > >> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <dejan.menges@gmail.com
> >
> > >> wrote:
> > >>
> > >> > Basically how I came to this question - this happened super rarely,
> > and
> > >> we
> > >> > narrowed it down to hotspotting. Map was timing out on three regions
> > >> which
> > >> > were 4-5 times bigger then other regions for the same table, and
> > region
> > >> > split fixed this.
> > >> >
> > >> > However, was just thinking about if there are maybe some
> > >> recommendations or
> > >> > something about this, as it's also super hard to reproduce again
> same
> > >> > situation to retest it.
> > >> >
> > >> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
> > >> michael_segel@hotmail.com>
> > >> > wrote:
> > >> >
> > >> > > There is no single ‘right’ value.
> > >> > >
> > >> > > As you pointed out… some of your Mapper.map() iterations are
> taking
> > >> > longer
> > >> > > than 60 seconds.
> > >> > >
> > >> > > The first thing is to determine why that happens. (It could be
> > >> normal,
> > >> > or
> > >> > > it could be bad code on your developers part. We don’t know.)
> > >> > >
> > >> > > The other thing is that if you determine that your code is perfect
> > >> and it
> > >> > > does what you want it to do… and its a major part of your use
> case…
> > >> you
> > >> > > then increase your timeouts to 120 seconds.
> > >> > >
> > >> > > The reason why its a tough issue is that we don’t know what
> hardware
> > >> you
> > >> > > are using. How many nodes… code quality.. etc … too many factors.
> > >> > >
> > >> > >
> > >> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <
> dejan.menges@gmail.com
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > What's the best practice to calculate this value for your
> cluster,
> > >> if
> > >> > > there
> > >> > > > is some?
> > >> > > >
> > >> > > > In some situations we saw that some maps are taking more than
> > >> default
> > >> > 60
> > >> > > > seconds which was failing specific map job (as if it failed
> once,
> > it
> > >> > > failed
> > >> > > > also every other time by number of configured retries).
> > >> > > >
> > >> > > > I would like to tune RPC parameters a bit, but googling and
> > looking
> > >> > into
> > >> > > > HBase Book doesn't tell me how to calculate right values, and
> what
> > >> else
> > >> > > to
> > >> > > > take a look beside hbase.rpc.timeout.
> > >> > > >
> > >> > > > Thanks a lot,
> > >> > > > Dejan
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Right value for hbase.rpc.timeout
Posted by Ted Yu <yu...@gmail.com>.
Please see https://issues.apache.org/jira/browse/HBASE-10501 and
https://issues.apache.org/jira/browse/HBASE-10716
Cheers
On Mon, May 4, 2015 at 1:43 AM, Dejan Menges <de...@gmail.com> wrote:
> Sorry, maybe I'm wrong: I made my conclusion based on what I read
> http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
> that's default; now just found in HBase Book that:
>
> hbase.regionserver.region.split.policy
> Description
>
> A split policy determines when a region should be split. The various other
> split policies that are available currently are
> ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy,
> DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.
> Default
>
>
> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy
>
> Otherwise, I can not find any default set in any preinstalled config files.
> So what's exactly the truth? :)
>
>
> On Mon, May 4, 2015 at 10:31 AM Dejan Menges <de...@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > Max filesize for region is set to 75G in our case. Regarding split policy
> > we use most likely ConstantSizeRegionSplitPolicy
> > <
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html>
> (it's
> > 0.98.0 with bunch of patches and that should be default one).
> >
> > Also, regarding link you sent me in 98.3 - I can not find anywhere what's
> > default value for hbase.regionserver.lease.period? Is this parameter
> still
> > called like this?
> >
> > On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
> >
> >> Please take a look at 98.3 under
> >> http://hbase.apache.org/book.html#trouble.client
> >>
> >> BTW what's the value for hbase.hregion.max.filesize ?
> >> Which split policy do you use ?
> >>
> >> Cheers
> >>
> >> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <de...@gmail.com>
> >> wrote:
> >>
> >> > Basically how I came to this question - this happened super rarely,
> and
> >> we
> >> > narrowed it down to hotspotting. Map was timing out on three regions
> >> which
> >> > were 4-5 times bigger then other regions for the same table, and
> region
> >> > split fixed this.
> >> >
> >> > However, was just thinking about if there are maybe some
> >> recommendations or
> >> > something about this, as it's also super hard to reproduce again same
> >> > situation to retest it.
> >> >
> >> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
> >> michael_segel@hotmail.com>
> >> > wrote:
> >> >
> >> > > There is no single ‘right’ value.
> >> > >
> >> > > As you pointed out… some of your Mapper.map() iterations are taking
> >> > longer
> >> > > than 60 seconds.
> >> > >
> >> > > The first thing is to determine why that happens. (It could be
> >> normal,
> >> > or
> >> > > it could be bad code on your developers part. We don’t know.)
> >> > >
> >> > > The other thing is that if you determine that your code is perfect
> >> and it
> >> > > does what you want it to do… and its a major part of your use case…
> >> you
> >> > > then increase your timeouts to 120 seconds.
> >> > >
> >> > > The reason why its a tough issue is that we don’t know what hardware
> >> you
> >> > > are using. How many nodes… code quality.. etc … too many factors.
> >> > >
> >> > >
> >> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <dejan.menges@gmail.com
> >
> >> > > wrote:
> >> > > >
> >> > > > Hi,
> >> > > >
> >> > > > What's the best practice to calculate this value for your cluster,
> >> if
> >> > > there
> >> > > > is some?
> >> > > >
> >> > > > In some situations we saw that some maps are taking more than
> >> default
> >> > 60
> >> > > > seconds which was failing specific map job (as if it failed once,
> it
> >> > > failed
> >> > > > also every other time by number of configured retries).
> >> > > >
> >> > > > I would like to tune RPC parameters a bit, but googling and
> looking
> >> > into
> >> > > > HBase Book doesn't tell me how to calculate right values, and what
> >> else
> >> > > to
> >> > > > take a look beside hbase.rpc.timeout.
> >> > > >
> >> > > > Thanks a lot,
> >> > > > Dejan
> >> > >
> >> > >
> >> >
> >>
> >
>
Re: Right value for hbase.rpc.timeout
Posted by Dejan Menges <de...@gmail.com>.
Sorry, maybe I'm wrong: I made my conclusion based on what I read
http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
that's default; now just found in HBase Book that:
hbase.regionserver.region.split.policy
Description
A split policy determines when a region should be split. The various other
split policies that are available currently are
ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy,
DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.
Default
org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy
Otherwise, I can not find any default set in any preinstalled config files.
So what's exactly the truth? :)
On Mon, May 4, 2015 at 10:31 AM Dejan Menges <de...@gmail.com> wrote:
> Hi Ted,
>
> Max filesize for region is set to 75G in our case. Regarding split policy
> we use most likely ConstantSizeRegionSplitPolicy
> <http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html> (it's
> 0.98.0 with bunch of patches and that should be default one).
>
> Also, regarding link you sent me in 98.3 - I can not find anywhere what's
> default value for hbase.regionserver.lease.period? Is this parameter still
> called like this?
>
> On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yu...@gmail.com> wrote:
>
>> Please take a look at 98.3 under
>> http://hbase.apache.org/book.html#trouble.client
>>
>> BTW what's the value for hbase.hregion.max.filesize ?
>> Which split policy do you use ?
>>
>> Cheers
>>
>> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <de...@gmail.com>
>> wrote:
>>
>> > Basically how I came to this question - this happened super rarely, and
>> we
>> > narrowed it down to hotspotting. Map was timing out on three regions
>> which
>> > were 4-5 times bigger then other regions for the same table, and region
>> > split fixed this.
>> >
>> > However, was just thinking about if there are maybe some
>> recommendations or
>> > something about this, as it's also super hard to reproduce again same
>> > situation to retest it.
>> >
>> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
>> michael_segel@hotmail.com>
>> > wrote:
>> >
>> > > There is no single ‘right’ value.
>> > >
>> > > As you pointed out… some of your Mapper.map() iterations are taking
>> > longer
>> > > than 60 seconds.
>> > >
>> > > The first thing is to determine why that happens. (It could be
>> normal,
>> > or
>> > > it could be bad code on your developers part. We don’t know.)
>> > >
>> > > The other thing is that if you determine that your code is perfect
>> and it
>> > > does what you want it to do… and its a major part of your use case…
>> you
>> > > then increase your timeouts to 120 seconds.
>> > >
>> > > The reason why its a tough issue is that we don’t know what hardware
>> you
>> > > are using. How many nodes… code quality.. etc … too many factors.
>> > >
>> > >
>> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <de...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi,
>> > > >
>> > > > What's the best practice to calculate this value for your cluster,
>> if
>> > > there
>> > > > is some?
>> > > >
>> > > > In some situations we saw that some maps are taking more than
>> default
>> > 60
>> > > > seconds which was failing specific map job (as if it failed once, it
>> > > failed
>> > > > also every other time by number of configured retries).
>> > > >
>> > > > I would like to tune RPC parameters a bit, but googling and looking
>> > into
>> > > > HBase Book doesn't tell me how to calculate right values, and what
>> else
>> > > to
>> > > > take a look beside hbase.rpc.timeout.
>> > > >
>> > > > Thanks a lot,
>> > > > Dejan
>> > >
>> > >
>> >
>>
>