You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by branky <br...@gmail.com> on 2013/12/20 05:49:07 UTC
Did major compaction really happen?
In order to turn off major compaction, I set MAJOR_COMPACTION_PERIOD to 0.
However, I checked region server log, saw major compaction “happened" quite
often:
2013-12-17 01:44:45,085 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in cf1 of
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
b23af74c66d8477b963750feed19ef23, size=90.2m; total size for store is 90.2m
2013-12-17 01:45:33,762 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in cf1 of
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
5a9bcff84c9e4a96bc40c31d4ad80956, size=157.5m; total size for store is 179.7m
2013-12-17 01:46:29,629 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in cf1 of
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
8000d825d4fa4a9bb10d2aef2820e5c5, size=226.2m; total size for store is 248.4m
2013-12-17 01:47:26,814 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in cf1 of
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
724dd3f1c4ae4e7fb3925f70f878c724, size=294.8m; total size for store is 317.1m
2013-12-17 01:56:51,058 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in cf1 of
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
efc0238399bc4e40bf86cf76e5b55ed9, size=546.9m; total size for store is 613.8m
2013-12-17 02:05:49,823 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 5 file(s) in cf1 of
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
d296aa41ea6c4ddb8e552d8cb89b03dc, size=1.0g; total size for store is 1.1g
But then when I checked the detail log further, I were totally confused:
2013-12-17 01:55:48,930 DEBUG org.apache.hadoop.hbase.regionserver.Store:
a6b0df058bf97f2f0efe1ed75f445a2c - cf1: Initiating minorcompaction
2013-12-17 01:55:48,930 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on cf1 in region
my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.
2013-12-17 01:55:48,930 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Small Compaction requested:
regionName=my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.,
storeName=cf1, fileCount=4, fileSize=543.9m (294.8m, 204.7m, 22.2m, 22.3m),
priority=3, time=9701090312106379; Because: Thread-15; compaction_queue=(0:0),
split_queue=0
2013-12-17 01:55:48,930 INFO
org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 4 file(s) in
cf1 of my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into tmpdir=h
dfs://namenode.mycompany.com/hbase/my_table/a6b0df058bf97f2f0efe1ed75f445a2c/.tm
p, seqid=2261767047, totalSize=543.9m 2013-12-17 01:56:51,058 INFO
org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4
file(s) in cf1 of my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.
into efc0238399bc4e40bf86cf76e5b55ed9, size=546.9m; total size for store is
613.8m
2013-12-17 01:56:51,058 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed
compaction:
regionName=my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.,
storeName=cf1, fileCount=4, fileSize=543.9m, priority=3, time=9701090312106379;
duration=1mins, 2sec
It seems a minor compaction was initiated, but then turned to be identified as
major compaction when it’s finished. I am not sure whether this is a bug or not,
can anyone explain this situation? Thank you!
Re: Did major compaction really happen?
Posted by branky <br...@gmail.com>.
branky <br...@...> writes:
btw, the version is 0.94.6
>
> In order to turn off major compaction, I set MAJOR_COMPACTION_PERIOD to 0.
> However, I checked region server log, saw major compaction “happened" quite
> often:
Re: Did major compaction really happen?
Posted by Anoop John <an...@gmail.com>.
The minor compaction is selecting all files. So there is nothing harm there
by being promoted as major compaction (by the system) .. Are you finding
some issues with that or just the logs made you worry?
-Anoop-
On Fri, Dec 20, 2013 at 11:55 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> The main advantage you get by disabled major compaction is that the major
> compaction threads do not wake up to perform the major compaction and which
> is generally resource intensive as it used to work on a large set of files.
>
> Regards
> Ram
>
>
> On Fri, Dec 20, 2013 at 11:52 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > YEs in a way it is true. But do you have deletes in your usecase. If it
> > is not there then this does not do anything different other than what a
> > minor compaction would do.
> >
> > Regards
> > Ram
> >
> >
> > On Fri, Dec 20, 2013 at 11:24 AM, branky <br...@gmail.com> wrote:
> >
> >> So, we can't say there's a way to completely avoid major compaction,
> >> right?
> >>
> >>
> >>
> >>
> >
>
Re: Did major compaction really happen?
Posted by ramkrishna vasudevan <ra...@gmail.com>.
The main advantage you get by disabled major compaction is that the major
compaction threads do not wake up to perform the major compaction and which
is generally resource intensive as it used to work on a large set of files.
Regards
Ram
On Fri, Dec 20, 2013 at 11:52 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> YEs in a way it is true. But do you have deletes in your usecase. If it
> is not there then this does not do anything different other than what a
> minor compaction would do.
>
> Regards
> Ram
>
>
> On Fri, Dec 20, 2013 at 11:24 AM, branky <br...@gmail.com> wrote:
>
>> So, we can't say there's a way to completely avoid major compaction,
>> right?
>>
>>
>>
>>
>
Re: Did major compaction really happen?
Posted by ramkrishna vasudevan <ra...@gmail.com>.
YEs in a way it is true. But do you have deletes in your usecase. If it
is not there then this does not do anything different other than what a
minor compaction would do.
Regards
Ram
On Fri, Dec 20, 2013 at 11:24 AM, branky <br...@gmail.com> wrote:
> So, we can't say there's a way to completely avoid major compaction, right?
>
>
>
>
Re: Did major compaction really happen?
Posted by branky <br...@gmail.com>.
So, we can't say there's a way to completely avoid major compaction, right?
Re: Did major compaction really happen?
Posted by ramkrishna vasudevan <ra...@gmail.com>.
>From the code I see that there is no property for that. The system decides
that. From your logs one thing you can make sure is that see if every time
a minor compaction was upgraded to major compaction. Not always it would
have happened unless the above mentioned condition was met.
Regards
Ram
On Fri, Dec 20, 2013 at 10:54 AM, branky <br...@gmail.com> wrote:
>
> Thanks Ram. So is there any property about the number of files selected
> which
> controls whether promote a minor compact to major compaction?
> In my case 4 files triggered this behavior.
>
>
>
Re: Did major compaction really happen?
Posted by branky <br...@gmail.com>.
Thanks Ram. So is there any property about the number of files selected which
controls whether promote a minor compact to major compaction?
In my case 4 files triggered this behavior.
Re: Did major compaction really happen?
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Sometimes if the list of files selected for minor compaction is equal to
the total store files present at that time, such minor compactions are
promoted to work as major compaction. So this is what should have happened
in your system.
This is not a bug but a behaviour.
Regards
Ram
On Fri, Dec 20, 2013 at 10:19 AM, branky <br...@gmail.com> wrote:
> In order to turn off major compaction, I set MAJOR_COMPACTION_PERIOD to 0.
> However, I checked region server log, saw major compaction “happened"
> quite
> often:
>
> 2013-12-17 01:44:45,085 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 4 file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> b23af74c66d8477b963750feed19ef23, size=90.2m; total size for store is 90.2m
> 2013-12-17 01:45:33,762 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 4 file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> 5a9bcff84c9e4a96bc40c31d4ad80956, size=157.5m; total size for store is
> 179.7m
> 2013-12-17 01:46:29,629 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 4 file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> 8000d825d4fa4a9bb10d2aef2820e5c5, size=226.2m; total size for store is
> 248.4m
> 2013-12-17 01:47:26,814 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 4 file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> 724dd3f1c4ae4e7fb3925f70f878c724, size=294.8m; total size for store is
> 317.1m
> 2013-12-17 01:56:51,058 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 4 file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> efc0238399bc4e40bf86cf76e5b55ed9, size=546.9m; total size for store is
> 613.8m
> 2013-12-17 02:05:49,823 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed major compaction of 5 file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> d296aa41ea6c4ddb8e552d8cb89b03dc, size=1.0g; total size for store is 1.1g
>
> But then when I checked the detail log further, I were totally confused:
> 2013-12-17 01:55:48,930 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> a6b0df058bf97f2f0efe1ed75f445a2c - cf1: Initiating minorcompaction
> 2013-12-17 01:55:48,930 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on cf1 in region
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.
> 2013-12-17 01:55:48,930 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> Small Compaction requested:
> regionName=my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.,
> storeName=cf1, fileCount=4, fileSize=543.9m (294.8m, 204.7m, 22.2m, 22.3m),
> priority=3, time=9701090312106379; Because: Thread-15;
> compaction_queue=(0:0),
> split_queue=0
> 2013-12-17 01:55:48,930 INFO
> org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 4
> file(s) in
> cf1 of my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c. into
> tmpdir=h
> dfs://
> namenode.mycompany.com/hbase/my_table/a6b0df058bf97f2f0efe1ed75f445a2c/.tm
> p, seqid=2261767047, totalSize=543.9m 2013-12-17 01:56:51,058 INFO
> org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4
> file(s) in cf1 of
> my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.
> into efc0238399bc4e40bf86cf76e5b55ed9, size=546.9m; total size for store is
> 613.8m
> 2013-12-17 01:56:51,058 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
> completed
> compaction:
> regionName=my_table,75,1387269666255.a6b0df058bf97f2f0efe1ed75f445a2c.,
> storeName=cf1, fileCount=4, fileSize=543.9m, priority=3,
> time=9701090312106379;
> duration=1mins, 2sec
>
>
> It seems a minor compaction was initiated, but then turned to be
> identified as
> major compaction when it’s finished. I am not sure whether this is a bug
> or not,
> can anyone explain this situation? Thank you!
>
>
>