You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Demai Ni <ni...@gmail.com> on 2014/03/06 19:28:45 UTC
turn off size-based Major Compaction?
hi, folks,
Is there a way to turn off Major Compaction, for sure?
We are using 96.0, and set hbase.hregion.majorcompaction is 0. The
intention was to only run major compaction manually. It turns out after
heavy load, major compaction still occur as shown in WebUI.
After a bit search, I read this discussion:
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html,
which indicated that there are three way to trigger major compaction:
1) user issued, (that is manually, such in hbase shell)
2) timed, (that is hbase.hregion.majorcompaction)
3) size-based.
Is there a way to prevent or configure the 3) method? I found
hbase.hstore.compaction.max,
but it is for minor compaction.
So the question is: what configuration for the *size-based* on
*Major*compaction?
Many thanks
Demai
Re: turn off size-based Major Compaction?
Posted by Demai Ni <ni...@gmail.com>.
hi, Lars and Ted,
thank you so much for the insight.
For some reason, I thought the 'hbase.hregion.majorcompaction' will turn
off all Major compaction, except for the manual one. Hence, the research
and questions. The discussion from Ted, Nicolas and Michael(
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html)
provided a much clearer picture.
Thank for the comments from Lars. It was a performance concern from my QA
team, which trigger this. Knowing the implication (no additional penalty)on
this area, will put it at easy.
Again, appreciate the quick response. You guys are great!
Demai
On Thu, Mar 6, 2014 at 10:54 AM, Ted Yu <yu...@gmail.com> wrote:
> Demai:
> You can find description of compaction-related parameters in:
>
> http://hbase.apache.org/book.html#compaction
>
> Cheers
>
>
> On Thu, Mar 6, 2014 at 10:44 AM, lars hofhansl <la...@apache.org> wrote:
>
> > The problem is not major compactions, but compactions that compact a lot
> > of files.
> >
> >
> > HBase will always compact in the background (minor compactions).
> > If a minor compaction needs to compact all files anyway it is promoted to
> > a major compaction.
> > There is no performance penalty to that, just that if all files are
> > compacted anyway might as well get rid of the delete markers in the
> process.
> >
> >
> > So rather than avoiding major compactions, what you want to avoid are
> > compactions that touch a lot of (or all) files. You'd need to tweak the
> > compaction setting - which is an art in itself. You could lower the
> > compaction ratio, or increase the number of store files before compations
> > start, pick a different compaction strategy, etc.
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Demai Ni <ni...@gmail.com>
> > To: "user@hbase.apache.org" <us...@hbase.apache.org>
> > Sent: Thursday, March 6, 2014 10:28 AM
> > Subject: turn off size-based Major Compaction?
> >
> >
> > hi, folks,
> >
> > Is there a way to turn off Major Compaction, for sure?
> >
> > We are using 96.0, and set hbase.hregion.majorcompaction is 0. The
> > intention was to only run major compaction manually. It turns out after
> > heavy load, major compaction still occur as shown in WebUI.
> >
> > After a bit search, I read this discussion:
> >
> >
> http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html
> > ,
> > which indicated that there are three way to trigger major compaction:
> > 1) user issued, (that is manually, such in hbase shell)
> > 2) timed, (that is hbase.hregion.majorcompaction)
> > 3) size-based.
> >
> > Is there a way to prevent or configure the 3) method? I found
> > hbase.hstore.compaction.max,
> > but it is for minor compaction.
> >
> > So the question is: what configuration for the *size-based* on
> > *Major*compaction?
> >
> > Many thanks
> >
> > Demai
>
Re: turn off size-based Major Compaction?
Posted by Ted Yu <yu...@gmail.com>.
Demai:
You can find description of compaction-related parameters in:
http://hbase.apache.org/book.html#compaction
Cheers
On Thu, Mar 6, 2014 at 10:44 AM, lars hofhansl <la...@apache.org> wrote:
> The problem is not major compactions, but compactions that compact a lot
> of files.
>
>
> HBase will always compact in the background (minor compactions).
> If a minor compaction needs to compact all files anyway it is promoted to
> a major compaction.
> There is no performance penalty to that, just that if all files are
> compacted anyway might as well get rid of the delete markers in the process.
>
>
> So rather than avoiding major compactions, what you want to avoid are
> compactions that touch a lot of (or all) files. You'd need to tweak the
> compaction setting - which is an art in itself. You could lower the
> compaction ratio, or increase the number of store files before compations
> start, pick a different compaction strategy, etc.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Demai Ni <ni...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Thursday, March 6, 2014 10:28 AM
> Subject: turn off size-based Major Compaction?
>
>
> hi, folks,
>
> Is there a way to turn off Major Compaction, for sure?
>
> We are using 96.0, and set hbase.hregion.majorcompaction is 0. The
> intention was to only run major compaction manually. It turns out after
> heavy load, major compaction still occur as shown in WebUI.
>
> After a bit search, I read this discussion:
>
> http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html
> ,
> which indicated that there are three way to trigger major compaction:
> 1) user issued, (that is manually, such in hbase shell)
> 2) timed, (that is hbase.hregion.majorcompaction)
> 3) size-based.
>
> Is there a way to prevent or configure the 3) method? I found
> hbase.hstore.compaction.max,
> but it is for minor compaction.
>
> So the question is: what configuration for the *size-based* on
> *Major*compaction?
>
> Many thanks
>
> Demai
Re: turn off size-based Major Compaction?
Posted by lars hofhansl <la...@apache.org>.
The problem is not major compactions, but compactions that compact a lot of files.
HBase will always compact in the background (minor compactions).
If a minor compaction needs to compact all files anyway it is promoted to a major compaction.
There is no performance penalty to that, just that if all files are compacted anyway might as well get rid of the delete markers in the process.
So rather than avoiding major compactions, what you want to avoid are compactions that touch a lot of (or all) files. You'd need to tweak the compaction setting - which is an art in itself. You could lower the compaction ratio, or increase the number of store files before compations start, pick a different compaction strategy, etc.
-- Lars
________________________________
From: Demai Ni <ni...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Thursday, March 6, 2014 10:28 AM
Subject: turn off size-based Major Compaction?
hi, folks,
Is there a way to turn off Major Compaction, for sure?
We are using 96.0, and set hbase.hregion.majorcompaction is 0. The
intention was to only run major compaction manually. It turns out after
heavy load, major compaction still occur as shown in WebUI.
After a bit search, I read this discussion:
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html,
which indicated that there are three way to trigger major compaction:
1) user issued, (that is manually, such in hbase shell)
2) timed, (that is hbase.hregion.majorcompaction)
3) size-based.
Is there a way to prevent or configure the 3) method? I found
hbase.hstore.compaction.max,
but it is for minor compaction.
So the question is: what configuration for the *size-based* on
*Major*compaction?
Many thanks
Demai