You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by chenwenhui <c7...@163.com> on 2014/08/21 12:01:11 UTC
What is the difference between "hbase.server.thread.wakefrequency"
and "hbase.hregion.majorcompaction"?
From the book named "Hbase: the Definitive Guide", i know that there is a condition to trigger major compaction, i.e. a thread controlled by "hbase.server.thread.wakefrequency(10 seconds by default)" and "hbase.server.thread.wakefrequency.multiplier" checks periodically whether major compaction.
But i also read that the other conditon, both "hbase.hregion.majorcompaction(7 days by default)" and "hbase.hregion.majorcompaction.jitter", can also trigger major compaction.
As my understanding, because of the former, the latter condition will never execute.
Maybe i mis-undertand the two conditions, anyone can explain the difference between the two condtions? Thanks!
Re: Re: What is the difference between "hbase.server.thread.wakefrequency"
and "hbase.hregion.majorcompaction"?
Posted by Ted Yu <yu...@gmail.com>.
Let's look at the code of MemStoreFlusher#FlushHandler#run() :
fqe = flushQueue.poll(threadWakeFrequency, TimeUnit.MILLISECONDS);
After FlushQueueEntry is retrieved, flushRegion(fre) is called which calls
this method:
private boolean flushRegion(final HRegion region, final boolean
emergencyFlush) {
where whether to trigger compaction is checked:
boolean shouldCompact = region.flushcache().isCompactionNeeded();
On the other hand, major compaction decision is made by
RatioBasedCompactionPolicy#isMajorCompaction() which considers
hbase.hregion.majorcompaction for time based major compaction - see
getNextMajorCompactTime()
Cheers
On Thu, Aug 21, 2014 at 6:21 PM, chenwenhui <c7...@163.com> wrote:
> Thanks.
> Maybe i can not present my question clearly.
> I indeed understand the mean of the combination
> condition("hbase.server.thread.wakefrequency" and
> "server.compactchecker.interval.multiplier"), i also know that its product
> is the period that check whether trigger compaction.
> For another combination condition("hbase.hregion.majorcompaction(7 days by
> default)" and "hbase.hregion.majorcompaction.jitter"), i understand it is
> also a condition to trigger compaction. right? if so, what is the relation
> between the two combination condition?
>
>
>
>
>
>
>
>
> At 2014-08-21 09:54:51, "Ted Yu" <yu...@gmail.com> wrote:
> >See explanation for the following parameter (from hbase refguide)
> >
> >hbase.server.compactchecker.interval.multiplier
> >The number that determines how often we scan to see if compaction is
> >necessary. Normally, compactions are done after some events (such as
> >memstore flush), but if region didn't receive a lot of writes for some
> >time, or due to different compaction policies, it may be necessary to
> check
> >it periodically. The interval between checks is hbase.
> >server.compactchecker.interval.multiplier multiplied by
> >hbase.server.thread.wakefrequency.
> >Default. 1000
> >Cheers
> >
> >
> >On Thu, Aug 21, 2014 at 3:01 AM, chenwenhui <c7...@163.com> wrote:
> >
> >> From the book named "Hbase: the Definitive Guide", i know that there is
> a
> >> condition to trigger major compaction, i.e. a thread controlled by
> >> "hbase.server.thread.wakefrequency(10 seconds by default)" and
> >> "hbase.server.thread.wakefrequency.multiplier" checks periodically
> whether
> >> major compaction.
> >> But i also read that the other conditon, both
> >> "hbase.hregion.majorcompaction(7 days by default)" and
> >> "hbase.hregion.majorcompaction.jitter", can also trigger major
> compaction.
> >> As my understanding, because of the former, the latter condition will
> >> never execute.
> >> Maybe i mis-undertand the two conditions, anyone can explain the
> >> difference between the two condtions? Thanks!
>
Re:Re: Re: What is the difference between
"hbase.server.thread.wakefrequency" and "hbase.hregion.majorcompaction"?
Posted by chenwenhui <c7...@163.com>.
Hi Ted,
Thank you!
I have read the source codes you guide. Now i understand the difference between the two group parameters.
Thanks again.
At 2014-08-22 12:28:16, "Ted Yu" <yu...@gmail.com> wrote:
>Let's look at the code of MemStoreFlusher#FlushHandler#run() :
>
> fqe = flushQueue.poll(threadWakeFrequency, TimeUnit.MILLISECONDS);
>
>After FlushQueueEntry is retrieved, flushRegion(fre) is called which calls
>this method:
>
> private boolean flushRegion(final HRegion region, final boolean
>emergencyFlush) {
>
>where whether to trigger compaction is checked:
>
> boolean shouldCompact = region.flushcache().isCompactionNeeded();
>
>
>On the other hand, major compaction decision is made by
>RatioBasedCompactionPolicy#isMajorCompaction() which considers
>hbase.hregion.majorcompaction for time based major compaction - see
>getNextMajorCompactTime()
>
>
>Cheers
>
>
>On Thu, Aug 21, 2014 at 6:21 PM, chenwenhui <c7...@163.com> wrote:
>
>> Thanks.
>> Maybe i can not present my question clearly.
>> I indeed understand the mean of the combination
>> condition("hbase.server.thread.wakefrequency" and
>> "server.compactchecker.interval.multiplier"), i also know that its product
>> is the period that check whether trigger compaction.
>> For another combination condition("hbase.hregion.majorcompaction(7 days by
>> default)" and "hbase.hregion.majorcompaction.jitter"), i understand it is
>> also a condition to trigger compaction. right? if so, what is the relation
>> between the two combination condition?
>>
>>
>>
>>
>>
>>
>>
>>
>> At 2014-08-21 09:54:51, "Ted Yu" <yu...@gmail.com> wrote:
>> >See explanation for the following parameter (from hbase refguide)
>> >
>> >hbase.server.compactchecker.interval.multiplier
>> >The number that determines how often we scan to see if compaction is
>> >necessary. Normally, compactions are done after some events (such as
>> >memstore flush), but if region didn't receive a lot of writes for some
>> >time, or due to different compaction policies, it may be necessary to
>> check
>> >it periodically. The interval between checks is hbase.
>> >server.compactchecker.interval.multiplier multiplied by
>> >hbase.server.thread.wakefrequency.
>> >Default. 1000
>> >Cheers
>> >
>> >
>> >On Thu, Aug 21, 2014 at 3:01 AM, chenwenhui <c7...@163.com> wrote:
>> >
>> >> From the book named "Hbase: the Definitive Guide", i know that there is
>> a
>> >> condition to trigger major compaction, i.e. a thread controlled by
>> >> "hbase.server.thread.wakefrequency(10 seconds by default)" and
>> >> "hbase.server.thread.wakefrequency.multiplier" checks periodically
>> whether
>> >> major compaction.
>> >> But i also read that the other conditon, both
>> >> "hbase.hregion.majorcompaction(7 days by default)" and
>> >> "hbase.hregion.majorcompaction.jitter", can also trigger major
>> compaction.
>> >> As my understanding, because of the former, the latter condition will
>> >> never execute.
>> >> Maybe i mis-undertand the two conditions, anyone can explain the
>> >> difference between the two condtions? Thanks!
>>
Re: Re: What is the difference between "hbase.server.thread.wakefrequency"
and "hbase.hregion.majorcompaction"?
Posted by Ted Yu <yu...@gmail.com>.
Let's look at the code of MemStoreFlusher#FlushHandler#run() :
fqe = flushQueue.poll(threadWakeFrequency, TimeUnit.MILLISECONDS);
After FlushQueueEntry is retrieved, flushRegion(fre) is called which calls
this method:
private boolean flushRegion(final HRegion region, final boolean
emergencyFlush) {
where whether to trigger compaction is checked:
boolean shouldCompact = region.flushcache().isCompactionNeeded();
On the other hand, major compaction decision is made by
RatioBasedCompactionPolicy#isMajorCompaction() which considers
hbase.hregion.majorcompaction for time based major compaction - see
getNextMajorCompactTime()
Cheers
On Thu, Aug 21, 2014 at 6:21 PM, chenwenhui <c7...@163.com> wrote:
> Thanks.
> Maybe i can not present my question clearly.
> I indeed understand the mean of the combination
> condition("hbase.server.thread.wakefrequency" and
> "server.compactchecker.interval.multiplier"), i also know that its product
> is the period that check whether trigger compaction.
> For another combination condition("hbase.hregion.majorcompaction(7 days by
> default)" and "hbase.hregion.majorcompaction.jitter"), i understand it is
> also a condition to trigger compaction. right? if so, what is the relation
> between the two combination condition?
>
>
>
>
>
>
>
>
> At 2014-08-21 09:54:51, "Ted Yu" <yu...@gmail.com> wrote:
> >See explanation for the following parameter (from hbase refguide)
> >
> >hbase.server.compactchecker.interval.multiplier
> >The number that determines how often we scan to see if compaction is
> >necessary. Normally, compactions are done after some events (such as
> >memstore flush), but if region didn't receive a lot of writes for some
> >time, or due to different compaction policies, it may be necessary to
> check
> >it periodically. The interval between checks is hbase.
> >server.compactchecker.interval.multiplier multiplied by
> >hbase.server.thread.wakefrequency.
> >Default. 1000
> >Cheers
> >
> >
> >On Thu, Aug 21, 2014 at 3:01 AM, chenwenhui <c7...@163.com> wrote:
> >
> >> From the book named "Hbase: the Definitive Guide", i know that there is
> a
> >> condition to trigger major compaction, i.e. a thread controlled by
> >> "hbase.server.thread.wakefrequency(10 seconds by default)" and
> >> "hbase.server.thread.wakefrequency.multiplier" checks periodically
> whether
> >> major compaction.
> >> But i also read that the other conditon, both
> >> "hbase.hregion.majorcompaction(7 days by default)" and
> >> "hbase.hregion.majorcompaction.jitter", can also trigger major
> compaction.
> >> As my understanding, because of the former, the latter condition will
> >> never execute.
> >> Maybe i mis-undertand the two conditions, anyone can explain the
> >> difference between the two condtions? Thanks!
>
Re:Re: What is the difference between
"hbase.server.thread.wakefrequency" and "hbase.hregion.majorcompaction"?
Posted by chenwenhui <c7...@163.com>.
Thanks.
Maybe i can not present my question clearly.
I indeed understand the mean of the combination condition("hbase.server.thread.wakefrequency" and "server.compactchecker.interval.multiplier"), i also know that its product is the period that check whether trigger compaction.
For another combination condition("hbase.hregion.majorcompaction(7 days by default)" and "hbase.hregion.majorcompaction.jitter"), i understand it is also a condition to trigger compaction. right? if so, what is the relation between the two combination condition?
At 2014-08-21 09:54:51, "Ted Yu" <yu...@gmail.com> wrote:
>See explanation for the following parameter (from hbase refguide)
>
>hbase.server.compactchecker.interval.multiplier
>The number that determines how often we scan to see if compaction is
>necessary. Normally, compactions are done after some events (such as
>memstore flush), but if region didn't receive a lot of writes for some
>time, or due to different compaction policies, it may be necessary to check
>it periodically. The interval between checks is hbase.
>server.compactchecker.interval.multiplier multiplied by
>hbase.server.thread.wakefrequency.
>Default. 1000
>Cheers
>
>
>On Thu, Aug 21, 2014 at 3:01 AM, chenwenhui <c7...@163.com> wrote:
>
>> From the book named "Hbase: the Definitive Guide", i know that there is a
>> condition to trigger major compaction, i.e. a thread controlled by
>> "hbase.server.thread.wakefrequency(10 seconds by default)" and
>> "hbase.server.thread.wakefrequency.multiplier" checks periodically whether
>> major compaction.
>> But i also read that the other conditon, both
>> "hbase.hregion.majorcompaction(7 days by default)" and
>> "hbase.hregion.majorcompaction.jitter", can also trigger major compaction.
>> As my understanding, because of the former, the latter condition will
>> never execute.
>> Maybe i mis-undertand the two conditions, anyone can explain the
>> difference between the two condtions? Thanks!
Re: What is the difference between "hbase.server.thread.wakefrequency"
and "hbase.hregion.majorcompaction"?
Posted by Ted Yu <yu...@gmail.com>.
See explanation for the following parameter (from hbase refguide)
hbase.server.compactchecker.interval.multiplier
The number that determines how often we scan to see if compaction is
necessary. Normally, compactions are done after some events (such as
memstore flush), but if region didn't receive a lot of writes for some
time, or due to different compaction policies, it may be necessary to check
it periodically. The interval between checks is hbase.
server.compactchecker.interval.multiplier multiplied by
hbase.server.thread.wakefrequency.
Default. 1000
Cheers
On Thu, Aug 21, 2014 at 3:01 AM, chenwenhui <c7...@163.com> wrote:
> From the book named "Hbase: the Definitive Guide", i know that there is a
> condition to trigger major compaction, i.e. a thread controlled by
> "hbase.server.thread.wakefrequency(10 seconds by default)" and
> "hbase.server.thread.wakefrequency.multiplier" checks periodically whether
> major compaction.
> But i also read that the other conditon, both
> "hbase.hregion.majorcompaction(7 days by default)" and
> "hbase.hregion.majorcompaction.jitter", can also trigger major compaction.
> As my understanding, because of the former, the latter condition will
> never execute.
> Maybe i mis-undertand the two conditions, anyone can explain the
> difference between the two condtions? Thanks!