You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Randy Fox <rf...@connexity.com> on 2015/10/07 00:07:31 UTC
Minor compacts seem blocked on major compacts
We just did a big leap forward from 0.94 to 1.0. We do our own major compacts manually every night. One of the first things we noticed is that when a major compact runs, no minors run. The compaction queue grows and when the major compact finishes the minors then run. I have not found any new knobs we should be setting. Any ideas?
Our config is:
<property>
<name>hbase.hregion.majorcompaction</name>
<value>0</value>
<final>true</final>
</property>
<property>
<name>hbase.hstore.compactionThreshold</name>
<value>3</value>
</property>
<property>
<name>hbase.hstore.compaction.max</name>
<value>9</value>
</property>
<property>
<name>hbase.regionserver.thread.compaction.throttle</name>
<value>2684354560</value>
</property>
Thanks in advance,
Randy
Re: Minor compacts seem blocked on major compacts
Posted by Ted Yu <yu...@gmail.com>.
See the following:
http://hbase.apache.org/book.html#build
If you continue to have problem building hbase, I suggest starting another
thread on user@ with errors encountered.
Cheers
On Thu, Oct 8, 2015 at 7:49 AM, Randy Fox <rf...@connexity.com> wrote:
> Thanks Ted, looks the right change. We are using CDH and it does not have
> that fix. Easy enough to put in, though I tried to build hbase and ran
> into many issues.
> Is there a doc somewhere on the prereqs for building hbase (1.0.0) in
> particular and the mvn flags?
>
> Thanks,
>
> Randy
>
>
>
> On 10/7/15, 1:34 PM, "Ted Yu" <yu...@gmail.com> wrote:
>
> >Have you seen the following code ?
> >
> > ThreadPoolExecutor pool = (selectNow &&
> >s.throttleCompaction(compaction.getRequest().getSize()))
> > ? longCompactions : shortCompactions;
> >
> >Looks like throttleCompaction() returned false.
> >Please see the following method in RatioBasedCompactionPolicy :
> >
> > public boolean throttleCompaction(long compactionSize) {
> >
> > return compactionSize > comConf.getThrottlePoint();
> >
> >FYI
> >
> >On Wed, Oct 7, 2015 at 11:07 AM, Randy Fox <rf...@connexity.com> wrote:
> >
> >> Looking at the source I don’t see how a manual major compact could get
> in
> >> the large compaction pool:
> >> If selectNow Is True the pool selector has to be shortCompactions. From
> >> the logs it sure seems that selectNow has to be true, else it would have
> >> logged “system” instead of
> >>
> "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
> >> ;”
> >>
> >>
> >> Sure seems like an hbase bug.
> >>
> >> Am I missing something?
> >>
> >> Thanks,
> >>
> >> Randy
> >>
> >> /**
> >> * @param r HRegion store belongs to
> >> * @param s Store to request compaction on
> >> * @param why Why compaction requested -- used in debug messages
> >> * @param priority override the default priority (NO_PRIORITY ==
> decide)
> >> * @param request custom compaction request. Can be <tt>null</tt> in
> >> which case a simple
> >> * compaction will be used.
> >> */
> >> private synchronized CompactionRequest requestCompactionInternal(final
> >> HRegion r, final Store s,
> >> final String why, int priority, CompactionRequest request, boolean
> >> selectNow)
> >> throws IOException {
> >> if (this.server.isStopped()
> >> || (r.getTableDesc() != null &&
> >> !r.getTableDesc().isCompactionEnabled())) {
> >> return null;
> >> }
> >>
> >> CompactionContext compaction = null;
> >> if (selectNow) {
> >> compaction = selectCompaction(r, s, priority, request);
> >> if (compaction == null) return null; // message logged inside
> >> }
> >>
> >> // We assume that most compactions are small. So, put system
> >> compactions into small
> >> // pool; we will do selection there, and move to large pool if
> >> necessary.
> >> long size = selectNow ? compaction.getRequest().getSize() : 0;
> >> ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
> >> ? longCompactions : shortCompactions;
> >> pool.execute(new CompactionRunner(s, r, compaction, pool));
> >> if (LOG.isDebugEnabled()) {
> >> String type = (pool == shortCompactions) ? "Small " : "Large ";
> >> LOG.debug(type + "Compaction requested: " + (selectNow ?
> >> compaction.toString() : "system")
> >> + (why != null && !why.isEmpty() ? "; Because: " + why : "") +
> >> "; " + this);
> >> }
> >> return selectNow ? compaction.getRequest() : null;
> >> }
> >>
> >>
> >>
> >>
> >>
> >> On 10/6/15, 9:58 PM, "Randy Fox" <rf...@connexity.com> wrote:
> >>
> >> >Sorry, I do not follow. I thought there was one thread per pool, one
> for
> >> large and one for small. Thus my major compact should have been in the
> >> large pool and not blocking the small pool. That is how it worked in
> 0.94.
> >> >From the logs it demoted my compact to the small pool.
> >> >
> >> >-randy
> >> >
> >> >
> >> >
> >> >On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vl...@gmail.com>
> wrote:
> >> >
> >> >>>> If I read this correct the size is 4.8G and the throttle is 2.5G
> so it
> >> >>should have been put into the Large compaction pool.
> >> >>
> >> >>You answered your question yourself. Minor compaction in the same
> pool (1
> >> >>thread default) will be waiting until major is finished.
> >> >>
> >> >>-Vlad
> >> >>
> >> >>
> >> >>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rf...@connexity.com> wrote:
> >> >>
> >> >>> 2015-10-06 14:50:35,644 INFO
> >> org.apache.hadoop.hbase.regionserver.HStore:
> >> >>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
> >> >>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49.
> into
> >> >>>
> >>
> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
> >> >>> totalSize=4.8 G
> >> >>> 2015-10-06 14:50:35,646 DEBUG
> >> >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small
> >> Compaction
> >> >>> requested:
> >> >>>
> >>
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
> >> ;
> >> >>> Because: User-triggered major compaction; compaction_queue=(0:0),
> >> >>> split_queue=0, merge_queue=0
> >> >>>
> >> >>>
> >> >>> If I read this correct the size is 4.8G and the throttle is 2.5G so
> it
> >> >>> should have been put into the Large compaction pool.
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
> >> >>>
> >> >>> >We just did a big leap forward from 0.94 to 1.0. We do our own
> major
> >> >>> compacts manually every night. One of the first things we noticed is
> >> that
> >> >>> when a major compact runs, no minors run. The compaction queue
> grows
> >> and
> >> >>> when the major compact finishes the minors then run. I have not
> >> found any
> >> >>> new knobs we should be setting. Any ideas?
> >> >>> >Our config is:
> >> >>> >
> >> >>> ><property>
> >> >>> > <name>hbase.hregion.majorcompaction</name>
> >> >>> > <value>0</value>
> >> >>> > <final>true</final>
> >> >>> ></property>
> >> >>> ><property>
> >> >>> > <name>hbase.hstore.compactionThreshold</name>
> >> >>> > <value>3</value>
> >> >>> ></property>
> >> >>> >
> >> >>> ><property>
> >> >>> > <name>hbase.hstore.compaction.max</name>
> >> >>> > <value>9</value>
> >> >>> ></property>
> >> >>> ><property>
> >> >>> > <name>hbase.regionserver.thread.compaction.throttle</name>
> >> >>> > <value>2684354560</value>
> >> >>> ></property>
> >> >>> >
> >> >>> >
> >> >>> >Thanks in advance,
> >> >>> >
> >> >>> >Randy
> >> >>>
> >>
>
Re: Minor compacts seem blocked on major compacts
Posted by Randy Fox <rf...@connexity.com>.
Thanks Ted, looks the right change. We are using CDH and it does not have that fix. Easy enough to put in, though I tried to build hbase and ran into many issues.
Is there a doc somewhere on the prereqs for building hbase (1.0.0) in particular and the mvn flags?
Thanks,
Randy
On 10/7/15, 1:34 PM, "Ted Yu" <yu...@gmail.com> wrote:
>Have you seen the following code ?
>
> ThreadPoolExecutor pool = (selectNow &&
>s.throttleCompaction(compaction.getRequest().getSize()))
> ? longCompactions : shortCompactions;
>
>Looks like throttleCompaction() returned false.
>Please see the following method in RatioBasedCompactionPolicy :
>
> public boolean throttleCompaction(long compactionSize) {
>
> return compactionSize > comConf.getThrottlePoint();
>
>FYI
>
>On Wed, Oct 7, 2015 at 11:07 AM, Randy Fox <rf...@connexity.com> wrote:
>
>> Looking at the source I don’t see how a manual major compact could get in
>> the large compaction pool:
>> If selectNow Is True the pool selector has to be shortCompactions. From
>> the logs it sure seems that selectNow has to be true, else it would have
>> logged “system” instead of
>> "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
>> ;”
>>
>>
>> Sure seems like an hbase bug.
>>
>> Am I missing something?
>>
>> Thanks,
>>
>> Randy
>>
>> /**
>> * @param r HRegion store belongs to
>> * @param s Store to request compaction on
>> * @param why Why compaction requested -- used in debug messages
>> * @param priority override the default priority (NO_PRIORITY == decide)
>> * @param request custom compaction request. Can be <tt>null</tt> in
>> which case a simple
>> * compaction will be used.
>> */
>> private synchronized CompactionRequest requestCompactionInternal(final
>> HRegion r, final Store s,
>> final String why, int priority, CompactionRequest request, boolean
>> selectNow)
>> throws IOException {
>> if (this.server.isStopped()
>> || (r.getTableDesc() != null &&
>> !r.getTableDesc().isCompactionEnabled())) {
>> return null;
>> }
>>
>> CompactionContext compaction = null;
>> if (selectNow) {
>> compaction = selectCompaction(r, s, priority, request);
>> if (compaction == null) return null; // message logged inside
>> }
>>
>> // We assume that most compactions are small. So, put system
>> compactions into small
>> // pool; we will do selection there, and move to large pool if
>> necessary.
>> long size = selectNow ? compaction.getRequest().getSize() : 0;
>> ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
>> ? longCompactions : shortCompactions;
>> pool.execute(new CompactionRunner(s, r, compaction, pool));
>> if (LOG.isDebugEnabled()) {
>> String type = (pool == shortCompactions) ? "Small " : "Large ";
>> LOG.debug(type + "Compaction requested: " + (selectNow ?
>> compaction.toString() : "system")
>> + (why != null && !why.isEmpty() ? "; Because: " + why : "") +
>> "; " + this);
>> }
>> return selectNow ? compaction.getRequest() : null;
>> }
>>
>>
>>
>>
>>
>> On 10/6/15, 9:58 PM, "Randy Fox" <rf...@connexity.com> wrote:
>>
>> >Sorry, I do not follow. I thought there was one thread per pool, one for
>> large and one for small. Thus my major compact should have been in the
>> large pool and not blocking the small pool. That is how it worked in 0.94.
>> >From the logs it demoted my compact to the small pool.
>> >
>> >-randy
>> >
>> >
>> >
>> >On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vl...@gmail.com> wrote:
>> >
>> >>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>> >>should have been put into the Large compaction pool.
>> >>
>> >>You answered your question yourself. Minor compaction in the same pool (1
>> >>thread default) will be waiting until major is finished.
>> >>
>> >>-Vlad
>> >>
>> >>
>> >>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rf...@connexity.com> wrote:
>> >>
>> >>> 2015-10-06 14:50:35,644 INFO
>> org.apache.hadoop.hbase.regionserver.HStore:
>> >>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
>> >>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
>> >>>
>> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
>> >>> totalSize=4.8 G
>> >>> 2015-10-06 14:50:35,646 DEBUG
>> >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small
>> Compaction
>> >>> requested:
>> >>>
>> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
>> ;
>> >>> Because: User-triggered major compaction; compaction_queue=(0:0),
>> >>> split_queue=0, merge_queue=0
>> >>>
>> >>>
>> >>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>> >>> should have been put into the Large compaction pool.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
>> >>>
>> >>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
>> >>> compacts manually every night. One of the first things we noticed is
>> that
>> >>> when a major compact runs, no minors run. The compaction queue grows
>> and
>> >>> when the major compact finishes the minors then run. I have not
>> found any
>> >>> new knobs we should be setting. Any ideas?
>> >>> >Our config is:
>> >>> >
>> >>> ><property>
>> >>> > <name>hbase.hregion.majorcompaction</name>
>> >>> > <value>0</value>
>> >>> > <final>true</final>
>> >>> ></property>
>> >>> ><property>
>> >>> > <name>hbase.hstore.compactionThreshold</name>
>> >>> > <value>3</value>
>> >>> ></property>
>> >>> >
>> >>> ><property>
>> >>> > <name>hbase.hstore.compaction.max</name>
>> >>> > <value>9</value>
>> >>> ></property>
>> >>> ><property>
>> >>> > <name>hbase.regionserver.thread.compaction.throttle</name>
>> >>> > <value>2684354560</value>
>> >>> ></property>
>> >>> >
>> >>> >
>> >>> >Thanks in advance,
>> >>> >
>> >>> >Randy
>> >>>
>>
Re: Minor compacts seem blocked on major compacts
Posted by Ted Yu <yu...@gmail.com>.
Have you seen the following code ?
ThreadPoolExecutor pool = (selectNow &&
s.throttleCompaction(compaction.getRequest().getSize()))
? longCompactions : shortCompactions;
Looks like throttleCompaction() returned false.
Please see the following method in RatioBasedCompactionPolicy :
public boolean throttleCompaction(long compactionSize) {
return compactionSize > comConf.getThrottlePoint();
FYI
On Wed, Oct 7, 2015 at 11:07 AM, Randy Fox <rf...@connexity.com> wrote:
> Looking at the source I don’t see how a manual major compact could get in
> the large compaction pool:
> If selectNow Is True the pool selector has to be shortCompactions. From
> the logs it sure seems that selectNow has to be true, else it would have
> logged “system” instead of
> "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
> ;”
>
>
> Sure seems like an hbase bug.
>
> Am I missing something?
>
> Thanks,
>
> Randy
>
> /**
> * @param r HRegion store belongs to
> * @param s Store to request compaction on
> * @param why Why compaction requested -- used in debug messages
> * @param priority override the default priority (NO_PRIORITY == decide)
> * @param request custom compaction request. Can be <tt>null</tt> in
> which case a simple
> * compaction will be used.
> */
> private synchronized CompactionRequest requestCompactionInternal(final
> HRegion r, final Store s,
> final String why, int priority, CompactionRequest request, boolean
> selectNow)
> throws IOException {
> if (this.server.isStopped()
> || (r.getTableDesc() != null &&
> !r.getTableDesc().isCompactionEnabled())) {
> return null;
> }
>
> CompactionContext compaction = null;
> if (selectNow) {
> compaction = selectCompaction(r, s, priority, request);
> if (compaction == null) return null; // message logged inside
> }
>
> // We assume that most compactions are small. So, put system
> compactions into small
> // pool; we will do selection there, and move to large pool if
> necessary.
> long size = selectNow ? compaction.getRequest().getSize() : 0;
> ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
> ? longCompactions : shortCompactions;
> pool.execute(new CompactionRunner(s, r, compaction, pool));
> if (LOG.isDebugEnabled()) {
> String type = (pool == shortCompactions) ? "Small " : "Large ";
> LOG.debug(type + "Compaction requested: " + (selectNow ?
> compaction.toString() : "system")
> + (why != null && !why.isEmpty() ? "; Because: " + why : "") +
> "; " + this);
> }
> return selectNow ? compaction.getRequest() : null;
> }
>
>
>
>
>
> On 10/6/15, 9:58 PM, "Randy Fox" <rf...@connexity.com> wrote:
>
> >Sorry, I do not follow. I thought there was one thread per pool, one for
> large and one for small. Thus my major compact should have been in the
> large pool and not blocking the small pool. That is how it worked in 0.94.
> >From the logs it demoted my compact to the small pool.
> >
> >-randy
> >
> >
> >
> >On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vl...@gmail.com> wrote:
> >
> >>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
> >>should have been put into the Large compaction pool.
> >>
> >>You answered your question yourself. Minor compaction in the same pool (1
> >>thread default) will be waiting until major is finished.
> >>
> >>-Vlad
> >>
> >>
> >>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rf...@connexity.com> wrote:
> >>
> >>> 2015-10-06 14:50:35,644 INFO
> org.apache.hadoop.hbase.regionserver.HStore:
> >>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
> >>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
> >>>
> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
> >>> totalSize=4.8 G
> >>> 2015-10-06 14:50:35,646 DEBUG
> >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small
> Compaction
> >>> requested:
> >>>
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
> ;
> >>> Because: User-triggered major compaction; compaction_queue=(0:0),
> >>> split_queue=0, merge_queue=0
> >>>
> >>>
> >>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
> >>> should have been put into the Large compaction pool.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
> >>>
> >>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
> >>> compacts manually every night. One of the first things we noticed is
> that
> >>> when a major compact runs, no minors run. The compaction queue grows
> and
> >>> when the major compact finishes the minors then run. I have not
> found any
> >>> new knobs we should be setting. Any ideas?
> >>> >Our config is:
> >>> >
> >>> ><property>
> >>> > <name>hbase.hregion.majorcompaction</name>
> >>> > <value>0</value>
> >>> > <final>true</final>
> >>> ></property>
> >>> ><property>
> >>> > <name>hbase.hstore.compactionThreshold</name>
> >>> > <value>3</value>
> >>> ></property>
> >>> >
> >>> ><property>
> >>> > <name>hbase.hstore.compaction.max</name>
> >>> > <value>9</value>
> >>> ></property>
> >>> ><property>
> >>> > <name>hbase.regionserver.thread.compaction.throttle</name>
> >>> > <value>2684354560</value>
> >>> ></property>
> >>> >
> >>> >
> >>> >Thanks in advance,
> >>> >
> >>> >Randy
> >>>
>
Re: Minor compacts seem blocked on major compacts
Posted by Randy Fox <rf...@connexity.com>.
Looking at the source I don’t see how a manual major compact could get in the large compaction pool:
If selectNow Is True the pool selector has to be shortCompactions. From the logs it sure seems that selectNow has to be true, else it would have logged “system” instead of "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8;”
Sure seems like an hbase bug.
Am I missing something?
Thanks,
Randy
/**
* @param r HRegion store belongs to
* @param s Store to request compaction on
* @param why Why compaction requested -- used in debug messages
* @param priority override the default priority (NO_PRIORITY == decide)
* @param request custom compaction request. Can be <tt>null</tt> in which case a simple
* compaction will be used.
*/
private synchronized CompactionRequest requestCompactionInternal(final HRegion r, final Store s,
final String why, int priority, CompactionRequest request, boolean selectNow)
throws IOException {
if (this.server.isStopped()
|| (r.getTableDesc() != null && !r.getTableDesc().isCompactionEnabled())) {
return null;
}
CompactionContext compaction = null;
if (selectNow) {
compaction = selectCompaction(r, s, priority, request);
if (compaction == null) return null; // message logged inside
}
// We assume that most compactions are small. So, put system compactions into small
// pool; we will do selection there, and move to large pool if necessary.
long size = selectNow ? compaction.getRequest().getSize() : 0;
ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
? longCompactions : shortCompactions;
pool.execute(new CompactionRunner(s, r, compaction, pool));
if (LOG.isDebugEnabled()) {
String type = (pool == shortCompactions) ? "Small " : "Large ";
LOG.debug(type + "Compaction requested: " + (selectNow ? compaction.toString() : "system")
+ (why != null && !why.isEmpty() ? "; Because: " + why : "") + "; " + this);
}
return selectNow ? compaction.getRequest() : null;
}
On 10/6/15, 9:58 PM, "Randy Fox" <rf...@connexity.com> wrote:
>Sorry, I do not follow. I thought there was one thread per pool, one for large and one for small. Thus my major compact should have been in the large pool and not blocking the small pool. That is how it worked in 0.94.
>From the logs it demoted my compact to the small pool.
>
>-randy
>
>
>
>On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vl...@gmail.com> wrote:
>
>>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>>should have been put into the Large compaction pool.
>>
>>You answered your question yourself. Minor compaction in the same pool (1
>>thread default) will be waiting until major is finished.
>>
>>-Vlad
>>
>>
>>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rf...@connexity.com> wrote:
>>
>>> 2015-10-06 14:50:35,644 INFO org.apache.hadoop.hbase.regionserver.HStore:
>>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
>>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
>>> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
>>> totalSize=4.8 G
>>> 2015-10-06 14:50:35,646 DEBUG
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction
>>> requested:
>>> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8;
>>> Because: User-triggered major compaction; compaction_queue=(0:0),
>>> split_queue=0, merge_queue=0
>>>
>>>
>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>>> should have been put into the Large compaction pool.
>>>
>>>
>>>
>>>
>>>
>>> On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
>>>
>>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
>>> compacts manually every night. One of the first things we noticed is that
>>> when a major compact runs, no minors run. The compaction queue grows and
>>> when the major compact finishes the minors then run. I have not found any
>>> new knobs we should be setting. Any ideas?
>>> >Our config is:
>>> >
>>> ><property>
>>> > <name>hbase.hregion.majorcompaction</name>
>>> > <value>0</value>
>>> > <final>true</final>
>>> ></property>
>>> ><property>
>>> > <name>hbase.hstore.compactionThreshold</name>
>>> > <value>3</value>
>>> ></property>
>>> >
>>> ><property>
>>> > <name>hbase.hstore.compaction.max</name>
>>> > <value>9</value>
>>> ></property>
>>> ><property>
>>> > <name>hbase.regionserver.thread.compaction.throttle</name>
>>> > <value>2684354560</value>
>>> ></property>
>>> >
>>> >
>>> >Thanks in advance,
>>> >
>>> >Randy
>>>
Re: Minor compacts seem blocked on major compacts
Posted by Randy Fox <rf...@connexity.com>.
Sorry, I do not follow. I thought there was one thread per pool, one for large and one for small. Thus my major compact should have been in the large pool and not blocking the small pool. That is how it worked in 0.94.
From the logs it demoted my compact to the small pool.
-randy
On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vl...@gmail.com> wrote:
>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>should have been put into the Large compaction pool.
>
>You answered your question yourself. Minor compaction in the same pool (1
>thread default) will be waiting until major is finished.
>
>-Vlad
>
>
>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rf...@connexity.com> wrote:
>
>> 2015-10-06 14:50:35,644 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
>> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
>> totalSize=4.8 G
>> 2015-10-06 14:50:35,646 DEBUG
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction
>> requested:
>> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8;
>> Because: User-triggered major compaction; compaction_queue=(0:0),
>> split_queue=0, merge_queue=0
>>
>>
>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>> should have been put into the Large compaction pool.
>>
>>
>>
>>
>>
>> On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
>>
>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
>> compacts manually every night. One of the first things we noticed is that
>> when a major compact runs, no minors run. The compaction queue grows and
>> when the major compact finishes the minors then run. I have not found any
>> new knobs we should be setting. Any ideas?
>> >Our config is:
>> >
>> ><property>
>> > <name>hbase.hregion.majorcompaction</name>
>> > <value>0</value>
>> > <final>true</final>
>> ></property>
>> ><property>
>> > <name>hbase.hstore.compactionThreshold</name>
>> > <value>3</value>
>> ></property>
>> >
>> ><property>
>> > <name>hbase.hstore.compaction.max</name>
>> > <value>9</value>
>> ></property>
>> ><property>
>> > <name>hbase.regionserver.thread.compaction.throttle</name>
>> > <value>2684354560</value>
>> ></property>
>> >
>> >
>> >Thanks in advance,
>> >
>> >Randy
>>
Re: Minor compacts seem blocked on major compacts
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
should have been put into the Large compaction pool.
You answered your question yourself. Minor compaction in the same pool (1
thread default) will be waiting until major is finished.
-Vlad
On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rf...@connexity.com> wrote:
> 2015-10-06 14:50:35,644 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
> totalSize=4.8 G
> 2015-10-06 14:50:35,646 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction
> requested:
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8;
> Because: User-triggered major compaction; compaction_queue=(0:0),
> split_queue=0, merge_queue=0
>
>
> If I read this correct the size is 4.8G and the throttle is 2.5G so it
> should have been put into the Large compaction pool.
>
>
>
>
>
> On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
>
> >We just did a big leap forward from 0.94 to 1.0. We do our own major
> compacts manually every night. One of the first things we noticed is that
> when a major compact runs, no minors run. The compaction queue grows and
> when the major compact finishes the minors then run. I have not found any
> new knobs we should be setting. Any ideas?
> >Our config is:
> >
> ><property>
> > <name>hbase.hregion.majorcompaction</name>
> > <value>0</value>
> > <final>true</final>
> ></property>
> ><property>
> > <name>hbase.hstore.compactionThreshold</name>
> > <value>3</value>
> ></property>
> >
> ><property>
> > <name>hbase.hstore.compaction.max</name>
> > <value>9</value>
> ></property>
> ><property>
> > <name>hbase.regionserver.thread.compaction.throttle</name>
> > <value>2684354560</value>
> ></property>
> >
> >
> >Thanks in advance,
> >
> >Randy
>
Re: Minor compacts seem blocked on major compacts
Posted by Randy Fox <rf...@connexity.com>.
2015-10-06 14:50:35,644 INFO org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 4 file(s) in L of PROD_audience4,\x00 \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp, totalSize=4.8 G
2015-10-06 14:50:35,646 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8; Because: User-triggered major compaction; compaction_queue=(0:0), split_queue=0, merge_queue=0
If I read this correct the size is 4.8G and the throttle is 2.5G so it should have been put into the Large compaction pool.
On 10/6/15, 3:07 PM, "Randy Fox" <rf...@connexity.com> wrote:
>We just did a big leap forward from 0.94 to 1.0. We do our own major compacts manually every night. One of the first things we noticed is that when a major compact runs, no minors run. The compaction queue grows and when the major compact finishes the minors then run. I have not found any new knobs we should be setting. Any ideas?
>Our config is:
>
><property>
> <name>hbase.hregion.majorcompaction</name>
> <value>0</value>
> <final>true</final>
></property>
><property>
> <name>hbase.hstore.compactionThreshold</name>
> <value>3</value>
></property>
>
><property>
> <name>hbase.hstore.compaction.max</name>
> <value>9</value>
></property>
><property>
> <name>hbase.regionserver.thread.compaction.throttle</name>
> <value>2684354560</value>
></property>
>
>
>Thanks in advance,
>
>Randy