You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by mukund murrali <mu...@gmail.com> on 2015/08/05 08:34:10 UTC
Major compaction skipping for older regions
Hi,
We wanted to have deleted data for a week. So we configured
MIN_VERSIONS => 1
KEEP_DELETED_CELLS => TTL
TTL => 1 week.
As per our understanding, after 1 week the deleted data becomes available
for major compaction and should be purged (correct if wrong). Since we have
time series data, we don't have any write operations in those regions after
a week . But major compaction never took place for any regions and our
overall size grew drastically though we have deletes happening. After
analyzing, we found that major compaction takes place if any one of the 2
condition is satisfied.
1. If the time interval between major compaction is greater than a week
(default config).
2. if the block locality index falls below a threshold.
In our case, since we have min_versions to be 1, the first case condition
fails. Time to verify is set to Long.Max value, if min versions is not 0.
Second is block locality. To check the block locality index we enabled
fine logs. And we found the block locality is always 1, and we got logs
stating "Skipping major compaction......".
So, in this case is manually triggering major compaction the only choice?
Thanks
Re: Major compaction skipping for older regions
Posted by mukund murrali <mu...@gmail.com>.
Thanks Vladimir will explore the possibility of this.
On Thursday 20 August 2015, Vladimir Rodionov <vl...@gmail.com>
wrote:
> Hey, this looks like a BUG. You may try
>
> this little hack (hbase-site.xml):
>
> hbase.hstore.min.locality.to.skip.major.compact=1.1
>
> Any value above 1.0 should work
>
> -Vlad
>
> On Thu, Aug 20, 2015 at 5:04 AM, mukund murrali <mukundmurrali9@gmail.com
> <javascript:;>>
> wrote:
>
> > Any update from anyone on this? We are invoking major compaction manually
> > and we see too many skipping of major compaction since we have
> MIN_VERSION
> > to 1.
> >
> > Thanks
> >
> > On Tue, Aug 11, 2015 at 5:09 PM, mukund murrali <
> mukundmurrali9@gmail.com <javascript:;>>
> > wrote:
> >
> > > Yes. We have min version to be 1, hence I suppose there wont be any
> sort
> > > of time based compaction. After triggering major compaction we had only
> > the
> > > following logs
> > >
> > > compactions.RatioBasedCompactionPolicy: Selecting compaction from 3
> store
> > > files, 0 compacting, 3 eligible, 50 blocking
> > >
> > > Thanks
> > >
> > > On Tue, Aug 11, 2015 at 3:25 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> > >
> > >> bq. oldestTime -9223370597787064221ms
> > >>
> > >> This is due to minTimestamp missing from store file:
> > >>
> > >> Long minTimestamp = sf.getMinimumTimestamp();
> > >>
> > >> long oldest = (minTimestamp == null)
> > >>
> > >> ? Long.MIN_VALUE
> > >>
> > >> : now - minTimestamp.longValue();
> > >>
> > >> Can you pastebin log from RatioBasedCompactionPolicy in region server
> > log
> > >> after the manual compaction ?
> > >>
> > >> Thanks
> > >>
> > >> On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <
> > mukundmurrali9@gmail.com <javascript:;>
> > >> >
> > >> wrote:
> > >>
> > >> > We are using hbase-1.0.0. The following logs appears for all regions
> > in
> > >> the
> > >> > regionserver
> > >> >
> > >> > 2015-08-08 14:01:51,586 DEBUG
> > [regionserver//R1:16020.compactionChecker]
> > >> > compactions.RatioBasedCompactionPolicy: Skipping major compaction of
> > >> >
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
> > >> > because one (major) compacted file only, oldestTime
> > >> -9223370597787064221ms
> > >> > is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
> > >> >
> > >> >
> > >> > Yes after manual triggering the deletes purged. But we don't want to
> > >> have
> > >> > it manual. Any other config to avoid such scenario?
> > >> >
> > >> > Thanks
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> > >> >
> > >> > > What release of hbase are you using ?
> > >> > >
> > >> > > Can you pastebin region server log with DEBUG logging ?
> > >> > >
> > >> > > I guess you have tried issuing manual command. Did it work ?
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <
> > >> > mukundmurrali9@gmail.com <javascript:;>>
> > >> > > wrote:
> > >> > >
> > >> > > > Any one help us in this :( Are we missing somewhere in the use
> > >> case?
> > >> > > None
> > >> > > > of the deleted cells are undergoing major compaction.
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
> > >> > > mukundmurrali9@gmail.com <javascript:;>>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > We wanted to have deleted data for a week. So we configured
> > >> > > > >
> > >> > > > > MIN_VERSIONS => 1
> > >> > > > > KEEP_DELETED_CELLS => TTL
> > >> > > > > TTL => 1 week.
> > >> > > > >
> > >> > > > > As per our understanding, after 1 week the deleted data
> becomes
> > >> > > available
> > >> > > > > for major compaction and should be purged (correct if wrong).
> > >> Since
> > >> > we
> > >> > > > have
> > >> > > > > time series data, we don't have any write operations in those
> > >> regions
> > >> > > > after
> > >> > > > > a week . But major compaction never took place for any regions
> > and
> > >> > our
> > >> > > > > overall size grew drastically though we have deletes
> happening.
> > >> After
> > >> > > > > analyzing, we found that major compaction takes place if any
> one
> > >> of
> > >> > > the 2
> > >> > > > > condition is satisfied.
> > >> > > > >
> > >> > > > > 1. If the time interval between major compaction is greater
> > than a
> > >> > week
> > >> > > > > (default config).
> > >> > > > > 2. if the block locality index falls below a threshold.
> > >> > > > >
> > >> > > > > In our case, since we have min_versions to be 1, the first
> case
> > >> > > condition
> > >> > > > > fails. Time to verify is set to Long.Max value, if min
> versions
> > is
> > >> > not
> > >> > > 0.
> > >> > > > >
> > >> > > > > Second is block locality. To check the block locality index
> we
> > >> > enabled
> > >> > > > > fine logs. And we found the block locality is always 1, and
> we
> > >> got
> > >> > > logs
> > >> > > > > stating "Skipping major compaction......".
> > >> > > > >
> > >> > > > > So, in this case is manually triggering major compaction the
> > only
> > >> > > choice?
> > >> > > > >
> > >> > > > > Thanks
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Major compaction skipping for older regions
Posted by Vladimir Rodionov <vl...@gmail.com>.
Hey, this looks like a BUG. You may try
this little hack (hbase-site.xml):
hbase.hstore.min.locality.to.skip.major.compact=1.1
Any value above 1.0 should work
-Vlad
On Thu, Aug 20, 2015 at 5:04 AM, mukund murrali <mu...@gmail.com>
wrote:
> Any update from anyone on this? We are invoking major compaction manually
> and we see too many skipping of major compaction since we have MIN_VERSION
> to 1.
>
> Thanks
>
> On Tue, Aug 11, 2015 at 5:09 PM, mukund murrali <mu...@gmail.com>
> wrote:
>
> > Yes. We have min version to be 1, hence I suppose there wont be any sort
> > of time based compaction. After triggering major compaction we had only
> the
> > following logs
> >
> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store
> > files, 0 compacting, 3 eligible, 50 blocking
> >
> > Thanks
> >
> > On Tue, Aug 11, 2015 at 3:25 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> bq. oldestTime -9223370597787064221ms
> >>
> >> This is due to minTimestamp missing from store file:
> >>
> >> Long minTimestamp = sf.getMinimumTimestamp();
> >>
> >> long oldest = (minTimestamp == null)
> >>
> >> ? Long.MIN_VALUE
> >>
> >> : now - minTimestamp.longValue();
> >>
> >> Can you pastebin log from RatioBasedCompactionPolicy in region server
> log
> >> after the manual compaction ?
> >>
> >> Thanks
> >>
> >> On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <
> mukundmurrali9@gmail.com
> >> >
> >> wrote:
> >>
> >> > We are using hbase-1.0.0. The following logs appears for all regions
> in
> >> the
> >> > regionserver
> >> >
> >> > 2015-08-08 14:01:51,586 DEBUG
> [regionserver//R1:16020.compactionChecker]
> >> > compactions.RatioBasedCompactionPolicy: Skipping major compaction of
> >> >
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
> >> > because one (major) compacted file only, oldestTime
> >> -9223370597787064221ms
> >> > is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
> >> >
> >> >
> >> > Yes after manual triggering the deletes purged. But we don't want to
> >> have
> >> > it manual. Any other config to avoid such scenario?
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> > > What release of hbase are you using ?
> >> > >
> >> > > Can you pastebin region server log with DEBUG logging ?
> >> > >
> >> > > I guess you have tried issuing manual command. Did it work ?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <
> >> > mukundmurrali9@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Any one help us in this :( Are we missing somewhere in the use
> >> case?
> >> > > None
> >> > > > of the deleted cells are undergoing major compaction.
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
> >> > > mukundmurrali9@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > We wanted to have deleted data for a week. So we configured
> >> > > > >
> >> > > > > MIN_VERSIONS => 1
> >> > > > > KEEP_DELETED_CELLS => TTL
> >> > > > > TTL => 1 week.
> >> > > > >
> >> > > > > As per our understanding, after 1 week the deleted data becomes
> >> > > available
> >> > > > > for major compaction and should be purged (correct if wrong).
> >> Since
> >> > we
> >> > > > have
> >> > > > > time series data, we don't have any write operations in those
> >> regions
> >> > > > after
> >> > > > > a week . But major compaction never took place for any regions
> and
> >> > our
> >> > > > > overall size grew drastically though we have deletes happening.
> >> After
> >> > > > > analyzing, we found that major compaction takes place if any one
> >> of
> >> > > the 2
> >> > > > > condition is satisfied.
> >> > > > >
> >> > > > > 1. If the time interval between major compaction is greater
> than a
> >> > week
> >> > > > > (default config).
> >> > > > > 2. if the block locality index falls below a threshold.
> >> > > > >
> >> > > > > In our case, since we have min_versions to be 1, the first case
> >> > > condition
> >> > > > > fails. Time to verify is set to Long.Max value, if min versions
> is
> >> > not
> >> > > 0.
> >> > > > >
> >> > > > > Second is block locality. To check the block locality index we
> >> > enabled
> >> > > > > fine logs. And we found the block locality is always 1, and we
> >> got
> >> > > logs
> >> > > > > stating "Skipping major compaction......".
> >> > > > >
> >> > > > > So, in this case is manually triggering major compaction the
> only
> >> > > choice?
> >> > > > >
> >> > > > > Thanks
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Major compaction skipping for older regions
Posted by mukund murrali <mu...@gmail.com>.
Any update from anyone on this? We are invoking major compaction manually
and we see too many skipping of major compaction since we have MIN_VERSION
to 1.
Thanks
On Tue, Aug 11, 2015 at 5:09 PM, mukund murrali <mu...@gmail.com>
wrote:
> Yes. We have min version to be 1, hence I suppose there wont be any sort
> of time based compaction. After triggering major compaction we had only the
> following logs
>
> compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store
> files, 0 compacting, 3 eligible, 50 blocking
>
> Thanks
>
> On Tue, Aug 11, 2015 at 3:25 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. oldestTime -9223370597787064221ms
>>
>> This is due to minTimestamp missing from store file:
>>
>> Long minTimestamp = sf.getMinimumTimestamp();
>>
>> long oldest = (minTimestamp == null)
>>
>> ? Long.MIN_VALUE
>>
>> : now - minTimestamp.longValue();
>>
>> Can you pastebin log from RatioBasedCompactionPolicy in region server log
>> after the manual compaction ?
>>
>> Thanks
>>
>> On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <mukundmurrali9@gmail.com
>> >
>> wrote:
>>
>> > We are using hbase-1.0.0. The following logs appears for all regions in
>> the
>> > regionserver
>> >
>> > 2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker]
>> > compactions.RatioBasedCompactionPolicy: Skipping major compaction of
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
>> > because one (major) compacted file only, oldestTime
>> -9223370597787064221ms
>> > is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
>> >
>> >
>> > Yes after manual triggering the deletes purged. But we don't want to
>> have
>> > it manual. Any other config to avoid such scenario?
>> >
>> > Thanks
>> >
>> >
>> >
>> > On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > What release of hbase are you using ?
>> > >
>> > > Can you pastebin region server log with DEBUG logging ?
>> > >
>> > > I guess you have tried issuing manual command. Did it work ?
>> > >
>> > > Thanks
>> > >
>> > > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <
>> > mukundmurrali9@gmail.com>
>> > > wrote:
>> > >
>> > > > Any one help us in this :( Are we missing somewhere in the use
>> case?
>> > > None
>> > > > of the deleted cells are undergoing major compaction.
>> > > >
>> > > > Thanks
>> > > >
>> > > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
>> > > mukundmurrali9@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > We wanted to have deleted data for a week. So we configured
>> > > > >
>> > > > > MIN_VERSIONS => 1
>> > > > > KEEP_DELETED_CELLS => TTL
>> > > > > TTL => 1 week.
>> > > > >
>> > > > > As per our understanding, after 1 week the deleted data becomes
>> > > available
>> > > > > for major compaction and should be purged (correct if wrong).
>> Since
>> > we
>> > > > have
>> > > > > time series data, we don't have any write operations in those
>> regions
>> > > > after
>> > > > > a week . But major compaction never took place for any regions and
>> > our
>> > > > > overall size grew drastically though we have deletes happening.
>> After
>> > > > > analyzing, we found that major compaction takes place if any one
>> of
>> > > the 2
>> > > > > condition is satisfied.
>> > > > >
>> > > > > 1. If the time interval between major compaction is greater than a
>> > week
>> > > > > (default config).
>> > > > > 2. if the block locality index falls below a threshold.
>> > > > >
>> > > > > In our case, since we have min_versions to be 1, the first case
>> > > condition
>> > > > > fails. Time to verify is set to Long.Max value, if min versions is
>> > not
>> > > 0.
>> > > > >
>> > > > > Second is block locality. To check the block locality index we
>> > enabled
>> > > > > fine logs. And we found the block locality is always 1, and we
>> got
>> > > logs
>> > > > > stating "Skipping major compaction......".
>> > > > >
>> > > > > So, in this case is manually triggering major compaction the only
>> > > choice?
>> > > > >
>> > > > > Thanks
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Major compaction skipping for older regions
Posted by mukund murrali <mu...@gmail.com>.
Yes. We have min version to be 1, hence I suppose there wont be any sort of
time based compaction. After triggering major compaction we had only the
following logs
compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store
files, 0 compacting, 3 eligible, 50 blocking
Thanks
On Tue, Aug 11, 2015 at 3:25 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. oldestTime -9223370597787064221ms
>
> This is due to minTimestamp missing from store file:
>
> Long minTimestamp = sf.getMinimumTimestamp();
>
> long oldest = (minTimestamp == null)
>
> ? Long.MIN_VALUE
>
> : now - minTimestamp.longValue();
>
> Can you pastebin log from RatioBasedCompactionPolicy in region server log
> after the manual compaction ?
>
> Thanks
>
> On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <mu...@gmail.com>
> wrote:
>
> > We are using hbase-1.0.0. The following logs appears for all regions in
> the
> > regionserver
> >
> > 2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker]
> > compactions.RatioBasedCompactionPolicy: Skipping major compaction of
> >
> >
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
> > because one (major) compacted file only, oldestTime
> -9223370597787064221ms
> > is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
> >
> >
> > Yes after manual triggering the deletes purged. But we don't want to have
> > it manual. Any other config to avoid such scenario?
> >
> > Thanks
> >
> >
> >
> > On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > What release of hbase are you using ?
> > >
> > > Can you pastebin region server log with DEBUG logging ?
> > >
> > > I guess you have tried issuing manual command. Did it work ?
> > >
> > > Thanks
> > >
> > > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <
> > mukundmurrali9@gmail.com>
> > > wrote:
> > >
> > > > Any one help us in this :( Are we missing somewhere in the use case?
> > > None
> > > > of the deleted cells are undergoing major compaction.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
> > > mukundmurrali9@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We wanted to have deleted data for a week. So we configured
> > > > >
> > > > > MIN_VERSIONS => 1
> > > > > KEEP_DELETED_CELLS => TTL
> > > > > TTL => 1 week.
> > > > >
> > > > > As per our understanding, after 1 week the deleted data becomes
> > > available
> > > > > for major compaction and should be purged (correct if wrong). Since
> > we
> > > > have
> > > > > time series data, we don't have any write operations in those
> regions
> > > > after
> > > > > a week . But major compaction never took place for any regions and
> > our
> > > > > overall size grew drastically though we have deletes happening.
> After
> > > > > analyzing, we found that major compaction takes place if any one of
> > > the 2
> > > > > condition is satisfied.
> > > > >
> > > > > 1. If the time interval between major compaction is greater than a
> > week
> > > > > (default config).
> > > > > 2. if the block locality index falls below a threshold.
> > > > >
> > > > > In our case, since we have min_versions to be 1, the first case
> > > condition
> > > > > fails. Time to verify is set to Long.Max value, if min versions is
> > not
> > > 0.
> > > > >
> > > > > Second is block locality. To check the block locality index we
> > enabled
> > > > > fine logs. And we found the block locality is always 1, and we got
> > > logs
> > > > > stating "Skipping major compaction......".
> > > > >
> > > > > So, in this case is manually triggering major compaction the only
> > > choice?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>
Re: Major compaction skipping for older regions
Posted by Ted Yu <yu...@gmail.com>.
bq. oldestTime -9223370597787064221ms
This is due to minTimestamp missing from store file:
Long minTimestamp = sf.getMinimumTimestamp();
long oldest = (minTimestamp == null)
? Long.MIN_VALUE
: now - minTimestamp.longValue();
Can you pastebin log from RatioBasedCompactionPolicy in region server log
after the manual compaction ?
Thanks
On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <mu...@gmail.com>
wrote:
> We are using hbase-1.0.0. The following logs appears for all regions in the
> regionserver
>
> 2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker]
> compactions.RatioBasedCompactionPolicy: Skipping major compaction of
>
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
> because one (major) compacted file only, oldestTime -9223370597787064221ms
> is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
>
>
> Yes after manual triggering the deletes purged. But we don't want to have
> it manual. Any other config to avoid such scenario?
>
> Thanks
>
>
>
> On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > What release of hbase are you using ?
> >
> > Can you pastebin region server log with DEBUG logging ?
> >
> > I guess you have tried issuing manual command. Did it work ?
> >
> > Thanks
> >
> > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <
> mukundmurrali9@gmail.com>
> > wrote:
> >
> > > Any one help us in this :( Are we missing somewhere in the use case?
> > None
> > > of the deleted cells are undergoing major compaction.
> > >
> > > Thanks
> > >
> > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
> > mukundmurrali9@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We wanted to have deleted data for a week. So we configured
> > > >
> > > > MIN_VERSIONS => 1
> > > > KEEP_DELETED_CELLS => TTL
> > > > TTL => 1 week.
> > > >
> > > > As per our understanding, after 1 week the deleted data becomes
> > available
> > > > for major compaction and should be purged (correct if wrong). Since
> we
> > > have
> > > > time series data, we don't have any write operations in those regions
> > > after
> > > > a week . But major compaction never took place for any regions and
> our
> > > > overall size grew drastically though we have deletes happening. After
> > > > analyzing, we found that major compaction takes place if any one of
> > the 2
> > > > condition is satisfied.
> > > >
> > > > 1. If the time interval between major compaction is greater than a
> week
> > > > (default config).
> > > > 2. if the block locality index falls below a threshold.
> > > >
> > > > In our case, since we have min_versions to be 1, the first case
> > condition
> > > > fails. Time to verify is set to Long.Max value, if min versions is
> not
> > 0.
> > > >
> > > > Second is block locality. To check the block locality index we
> enabled
> > > > fine logs. And we found the block locality is always 1, and we got
> > logs
> > > > stating "Skipping major compaction......".
> > > >
> > > > So, in this case is manually triggering major compaction the only
> > choice?
> > > >
> > > > Thanks
> > > >
> > >
> >
>
Re: Major compaction skipping for older regions
Posted by mukund murrali <mu...@gmail.com>.
We are using hbase-1.0.0. The following logs appears for all regions in the
regionserver
2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker]
compactions.RatioBasedCompactionPolicy: Skipping major compaction of
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
because one (major) compacted file only, oldestTime -9223370597787064221ms
is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
Yes after manual triggering the deletes purged. But we don't want to have
it manual. Any other config to avoid such scenario?
Thanks
On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yu...@gmail.com> wrote:
> What release of hbase are you using ?
>
> Can you pastebin region server log with DEBUG logging ?
>
> I guess you have tried issuing manual command. Did it work ?
>
> Thanks
>
> On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <mu...@gmail.com>
> wrote:
>
> > Any one help us in this :( Are we missing somewhere in the use case?
> None
> > of the deleted cells are undergoing major compaction.
> >
> > Thanks
> >
> > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
> mukundmurrali9@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > We wanted to have deleted data for a week. So we configured
> > >
> > > MIN_VERSIONS => 1
> > > KEEP_DELETED_CELLS => TTL
> > > TTL => 1 week.
> > >
> > > As per our understanding, after 1 week the deleted data becomes
> available
> > > for major compaction and should be purged (correct if wrong). Since we
> > have
> > > time series data, we don't have any write operations in those regions
> > after
> > > a week . But major compaction never took place for any regions and our
> > > overall size grew drastically though we have deletes happening. After
> > > analyzing, we found that major compaction takes place if any one of
> the 2
> > > condition is satisfied.
> > >
> > > 1. If the time interval between major compaction is greater than a week
> > > (default config).
> > > 2. if the block locality index falls below a threshold.
> > >
> > > In our case, since we have min_versions to be 1, the first case
> condition
> > > fails. Time to verify is set to Long.Max value, if min versions is not
> 0.
> > >
> > > Second is block locality. To check the block locality index we enabled
> > > fine logs. And we found the block locality is always 1, and we got
> logs
> > > stating "Skipping major compaction......".
> > >
> > > So, in this case is manually triggering major compaction the only
> choice?
> > >
> > > Thanks
> > >
> >
>
Re: Major compaction skipping for older regions
Posted by Ted Yu <yu...@gmail.com>.
What release of hbase are you using ?
Can you pastebin region server log with DEBUG logging ?
I guess you have tried issuing manual command. Did it work ?
Thanks
On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <mu...@gmail.com>
wrote:
> Any one help us in this :( Are we missing somewhere in the use case? None
> of the deleted cells are undergoing major compaction.
>
> Thanks
>
> On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <mu...@gmail.com>
> wrote:
>
> > Hi,
> >
> > We wanted to have deleted data for a week. So we configured
> >
> > MIN_VERSIONS => 1
> > KEEP_DELETED_CELLS => TTL
> > TTL => 1 week.
> >
> > As per our understanding, after 1 week the deleted data becomes available
> > for major compaction and should be purged (correct if wrong). Since we
> have
> > time series data, we don't have any write operations in those regions
> after
> > a week . But major compaction never took place for any regions and our
> > overall size grew drastically though we have deletes happening. After
> > analyzing, we found that major compaction takes place if any one of the 2
> > condition is satisfied.
> >
> > 1. If the time interval between major compaction is greater than a week
> > (default config).
> > 2. if the block locality index falls below a threshold.
> >
> > In our case, since we have min_versions to be 1, the first case condition
> > fails. Time to verify is set to Long.Max value, if min versions is not 0.
> >
> > Second is block locality. To check the block locality index we enabled
> > fine logs. And we found the block locality is always 1, and we got logs
> > stating "Skipping major compaction......".
> >
> > So, in this case is manually triggering major compaction the only choice?
> >
> > Thanks
> >
>
Re: Major compaction skipping for older regions
Posted by mukund murrali <mu...@gmail.com>.
Any one help us in this :( Are we missing somewhere in the use case? None
of the deleted cells are undergoing major compaction.
Thanks
On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <mu...@gmail.com>
wrote:
> Hi,
>
> We wanted to have deleted data for a week. So we configured
>
> MIN_VERSIONS => 1
> KEEP_DELETED_CELLS => TTL
> TTL => 1 week.
>
> As per our understanding, after 1 week the deleted data becomes available
> for major compaction and should be purged (correct if wrong). Since we have
> time series data, we don't have any write operations in those regions after
> a week . But major compaction never took place for any regions and our
> overall size grew drastically though we have deletes happening. After
> analyzing, we found that major compaction takes place if any one of the 2
> condition is satisfied.
>
> 1. If the time interval between major compaction is greater than a week
> (default config).
> 2. if the block locality index falls below a threshold.
>
> In our case, since we have min_versions to be 1, the first case condition
> fails. Time to verify is set to Long.Max value, if min versions is not 0.
>
> Second is block locality. To check the block locality index we enabled
> fine logs. And we found the block locality is always 1, and we got logs
> stating "Skipping major compaction......".
>
> So, in this case is manually triggering major compaction the only choice?
>
> Thanks
>