You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by yun peng <pe...@gmail.com> on 2013/04/11 16:52:33 UTC
On source code of File Section mechanism in minor compaction
Hi, All
I am asking the source code showing the file selection mechanism for minor
compaction in HBase. I am aware that HBase minor compaction selects file
based on size and generation time. But I can not find source file that
realizes this mechanism in HBase. I am interested in the algorithm works
behind it. Kindly point to me the source code and I appreciate it. Thanks.
Regards
Yun
Re: On source code of File Section mechanism in minor compaction
Posted by yun peng <pe...@gmail.com>.
Hi Ted, thanks a lot for the reference.
Ted and All, I have additional minor question though. Would a flush in
HBase cause cascaded minor compaction? To be more specific, when a minor
compaction finishes, the newly generated HFile may make the condition of
minor compaction met again, in this case, would HBase trigger another minor
compaction?
For example, say we have following files.
HFile index 1, 2, 3, 4, 5, 6
HFile size 1, 1, 3, 3, 9, 9
with hbase.hstore.compactionThreshold=3 (i.e., at least 3 files to merge
each time) and ratio is 1, then a flush into such file storage initially
generates a new HFile of size 1 (the file is indexed by number 0):
HFile index 0, 1, 2, 3, 4, 5, 6
HFile size 1, 1, 1, 3, 3, 9, 9
The compaction condition is met for HFile 0, 1, 2 and a minor compaction is
trigger to merge file 1 and 2 and 0 to a new file of size 3, as follows,
HFile index 2', 3, 4, 5, 6
HFile size 3, 3, 3, 9, 9
Then the condition for minor compaction met again for HFile 2',3,4, would
this be triggered and executed immediately or to say would compaction be
cascaded in HBase?
Regards,
Yun
On Thu, Apr 11, 2013 at 12:12 PM, Ted Yu <yu...@gmail.com> wrote:
> For 0.94, take a look at these files:
>
> 94-hbase tyu$ ls
> src/main/java/org/apache/hadoop/hbase/regionserver/compactions
> CompactSelection.java CompactionProgress.java CompactionRequest.java
>
> You can see the revision history for these files. I suggest you start with
> recent JIRAs that touched them.
>
> Cheers
>
> On Thu, Apr 11, 2013 at 8:04 AM, yun peng <pe...@gmail.com> wrote:
>
> > Sorry to not mention the version. I am looking at version 0.94.2 which
> > seems file RatioBasedCompactionPolicy is not yet in.
> > regards,
> > Yun
> >
> >
> > On Thu, Apr 11, 2013 at 11:00 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Which release are you targeting ?
> > >
> > > In trunk, take a look at RatioBasedCompactionPolicy which is the
> default:
> > >
> > > public class RatioBasedCompactionPolicy extends CompactionPolicy {
> > >
> > > Cheers
> > >
> > > On Thu, Apr 11, 2013 at 7:52 AM, yun peng <pe...@gmail.com>
> wrote:
> > >
> > > > Hi, All
> > > > I am asking the source code showing the file selection mechanism for
> > > minor
> > > > compaction in HBase. I am aware that HBase minor compaction selects
> > file
> > > > based on size and generation time. But I can not find source file
> that
> > > > realizes this mechanism in HBase. I am interested in the algorithm
> > works
> > > > behind it. Kindly point to me the source code and I appreciate it.
> > > Thanks.
> > > > Regards
> > > > Yun
> > > >
> > >
> >
>
Re: On source code of File Section mechanism in minor compaction
Posted by Ted Yu <yu...@gmail.com>.
For 0.94, take a look at these files:
94-hbase tyu$ ls
src/main/java/org/apache/hadoop/hbase/regionserver/compactions
CompactSelection.java CompactionProgress.java CompactionRequest.java
You can see the revision history for these files. I suggest you start with
recent JIRAs that touched them.
Cheers
On Thu, Apr 11, 2013 at 8:04 AM, yun peng <pe...@gmail.com> wrote:
> Sorry to not mention the version. I am looking at version 0.94.2 which
> seems file RatioBasedCompactionPolicy is not yet in.
> regards,
> Yun
>
>
> On Thu, Apr 11, 2013 at 11:00 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Which release are you targeting ?
> >
> > In trunk, take a look at RatioBasedCompactionPolicy which is the default:
> >
> > public class RatioBasedCompactionPolicy extends CompactionPolicy {
> >
> > Cheers
> >
> > On Thu, Apr 11, 2013 at 7:52 AM, yun peng <pe...@gmail.com> wrote:
> >
> > > Hi, All
> > > I am asking the source code showing the file selection mechanism for
> > minor
> > > compaction in HBase. I am aware that HBase minor compaction selects
> file
> > > based on size and generation time. But I can not find source file that
> > > realizes this mechanism in HBase. I am interested in the algorithm
> works
> > > behind it. Kindly point to me the source code and I appreciate it.
> > Thanks.
> > > Regards
> > > Yun
> > >
> >
>
Re: On source code of File Section mechanism in minor compaction
Posted by yun peng <pe...@gmail.com>.
Sorry to not mention the version. I am looking at version 0.94.2 which
seems file RatioBasedCompactionPolicy is not yet in.
regards,
Yun
On Thu, Apr 11, 2013 at 11:00 AM, Ted Yu <yu...@gmail.com> wrote:
> Which release are you targeting ?
>
> In trunk, take a look at RatioBasedCompactionPolicy which is the default:
>
> public class RatioBasedCompactionPolicy extends CompactionPolicy {
>
> Cheers
>
> On Thu, Apr 11, 2013 at 7:52 AM, yun peng <pe...@gmail.com> wrote:
>
> > Hi, All
> > I am asking the source code showing the file selection mechanism for
> minor
> > compaction in HBase. I am aware that HBase minor compaction selects file
> > based on size and generation time. But I can not find source file that
> > realizes this mechanism in HBase. I am interested in the algorithm works
> > behind it. Kindly point to me the source code and I appreciate it.
> Thanks.
> > Regards
> > Yun
> >
>
Re: On source code of File Section mechanism in minor compaction
Posted by Ted Yu <yu...@gmail.com>.
Which release are you targeting ?
In trunk, take a look at RatioBasedCompactionPolicy which is the default:
public class RatioBasedCompactionPolicy extends CompactionPolicy {
Cheers
On Thu, Apr 11, 2013 at 7:52 AM, yun peng <pe...@gmail.com> wrote:
> Hi, All
> I am asking the source code showing the file selection mechanism for minor
> compaction in HBase. I am aware that HBase minor compaction selects file
> based on size and generation time. But I can not find source file that
> realizes this mechanism in HBase. I am interested in the algorithm works
> behind it. Kindly point to me the source code and I appreciate it. Thanks.
> Regards
> Yun
>