You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2012/12/12 21:15:08 UTC
How to know it's time for a major compaction?
Hello,
What are the main metrics/signals people use/look at to say "OK, I really
need to do a major compaction now", other than "hm, performance is not
good, let's try major compaction tonight"?
Thanks,
Otis
--
HBASE Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html
Re: How to know it's time for a major compaction?
Posted by Kevin O'dell <ke...@cloudera.com>.
Ted,
That is awesome. Thank you!
On Wed, Dec 12, 2012 at 3:36 PM, Ted Yu <yu...@gmail.com> wrote:
> Thanks confirmation, Kevin.
>
> I logged HBASE-7340 so that user doesn't have to script for compaction.
>
> On Wed, Dec 12, 2012 at 12:30 PM, Kevin O'dell <kevin.odell@cloudera.com
> >wrote:
>
> > Ted,
> >
> > Yes, it is all done through scripting.
> >
> > On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. they will major compact any region the balancer has moved.
> > >
> > > Interesting. Do they achieve this through scripting ?
> > >
> > > I think HBase can perform some follow-on action after region movement.
> > >
> > > Cheers
> > >
> > > On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <
> kevin.odell@cloudera.com
> > > >wrote:
> > >
> > > > Otis,
> > > >
> > > > I think it depends on your use case. Some use cases are extremely
> > > > dependent on locality and latency, so they will major compact any
> > region
> > > > the balancer has moved. Some people follow the HFile count and will
> > > > trigger a major when there are x amount of files. Others will just
> > > follow
> > > > best practices and do it daily. Some do very little writes and only
> > > > compact once a week.
> > > >
> > > > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > What are the main metrics/signals people use/look at to say "OK, I
> > > really
> > > > > need to do a major compaction now", other than "hm, performance is
> > not
> > > > > good, let's try major compaction tonight"?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > > > > Search Analytics - http://sematext.com/search-analytics/index.html
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kevin O'Dell
> > > > Customer Operations Engineer, Cloudera
> > > >
> > >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: How to know it's time for a major compaction?
Posted by Ted Yu <yu...@gmail.com>.
In RegionObserver, we already have the following hook:
/**
* Called after the region is reported as open to the master.
* @param c the environment provided by the region server
*/
void postOpen(final ObserverContext<RegionCoprocessorEnvironment> c);
Auto-compaction logic can be triggered through the above hook.
Cheers
On Wed, Dec 12, 2012 at 12:36 PM, Ted Yu <yu...@gmail.com> wrote:
> Thanks confirmation, Kevin.
>
> I logged HBASE-7340 so that user doesn't have to script for compaction.
>
>
> On Wed, Dec 12, 2012 at 12:30 PM, Kevin O'dell <ke...@cloudera.com>wrote:
>
>> Ted,
>>
>> Yes, it is all done through scripting.
>>
>> On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > bq. they will major compact any region the balancer has moved.
>> >
>> > Interesting. Do they achieve this through scripting ?
>> >
>> > I think HBase can perform some follow-on action after region movement.
>> >
>> > Cheers
>> >
>> > On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <
>> kevin.odell@cloudera.com
>> > >wrote:
>> >
>> > > Otis,
>> > >
>> > > I think it depends on your use case. Some use cases are extremely
>> > > dependent on locality and latency, so they will major compact any
>> region
>> > > the balancer has moved. Some people follow the HFile count and will
>> > > trigger a major when there are x amount of files. Others will just
>> > follow
>> > > best practices and do it daily. Some do very little writes and only
>> > > compact once a week.
>> > >
>> > > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
>> > > otis.gospodnetic@gmail.com> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > What are the main metrics/signals people use/look at to say "OK, I
>> > really
>> > > > need to do a major compaction now", other than "hm, performance is
>> not
>> > > > good, let's try major compaction tonight"?
>> > > >
>> > > > Thanks,
>> > > > Otis
>> > > > --
>> > > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
>> > > > Search Analytics - http://sematext.com/search-analytics/index.html
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kevin O'Dell
>> > > Customer Operations Engineer, Cloudera
>> > >
>> >
>>
>>
>>
>> --
>> Kevin O'Dell
>> Customer Operations Engineer, Cloudera
>>
>
>
Re: How to know it's time for a major compaction?
Posted by Ted Yu <yu...@gmail.com>.
Thanks confirmation, Kevin.
I logged HBASE-7340 so that user doesn't have to script for compaction.
On Wed, Dec 12, 2012 at 12:30 PM, Kevin O'dell <ke...@cloudera.com>wrote:
> Ted,
>
> Yes, it is all done through scripting.
>
> On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. they will major compact any region the balancer has moved.
> >
> > Interesting. Do they achieve this through scripting ?
> >
> > I think HBase can perform some follow-on action after region movement.
> >
> > Cheers
> >
> > On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <kevin.odell@cloudera.com
> > >wrote:
> >
> > > Otis,
> > >
> > > I think it depends on your use case. Some use cases are extremely
> > > dependent on locality and latency, so they will major compact any
> region
> > > the balancer has moved. Some people follow the HFile count and will
> > > trigger a major when there are x amount of files. Others will just
> > follow
> > > best practices and do it daily. Some do very little writes and only
> > > compact once a week.
> > >
> > > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > What are the main metrics/signals people use/look at to say "OK, I
> > really
> > > > need to do a major compaction now", other than "hm, performance is
> not
> > > > good, let's try major compaction tonight"?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > > > Search Analytics - http://sematext.com/search-analytics/index.html
> > > >
> > >
> > >
> > >
> > > --
> > > Kevin O'Dell
> > > Customer Operations Engineer, Cloudera
> > >
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>
Re: How to know it's time for a major compaction?
Posted by Kevin O'dell <ke...@cloudera.com>.
Ted,
Yes, it is all done through scripting.
On Wed, Dec 12, 2012 at 3:22 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. they will major compact any region the balancer has moved.
>
> Interesting. Do they achieve this through scripting ?
>
> I think HBase can perform some follow-on action after region movement.
>
> Cheers
>
> On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <kevin.odell@cloudera.com
> >wrote:
>
> > Otis,
> >
> > I think it depends on your use case. Some use cases are extremely
> > dependent on locality and latency, so they will major compact any region
> > the balancer has moved. Some people follow the HFile count and will
> > trigger a major when there are x amount of files. Others will just
> follow
> > best practices and do it daily. Some do very little writes and only
> > compact once a week.
> >
> > On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > What are the main metrics/signals people use/look at to say "OK, I
> really
> > > need to do a major compaction now", other than "hm, performance is not
> > > good, let's try major compaction tonight"?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > > Search Analytics - http://sematext.com/search-analytics/index.html
> > >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: How to know it's time for a major compaction?
Posted by Ted Yu <yu...@gmail.com>.
bq. they will major compact any region the balancer has moved.
Interesting. Do they achieve this through scripting ?
I think HBase can perform some follow-on action after region movement.
Cheers
On Wed, Dec 12, 2012 at 12:19 PM, Kevin O'dell <ke...@cloudera.com>wrote:
> Otis,
>
> I think it depends on your use case. Some use cases are extremely
> dependent on locality and latency, so they will major compact any region
> the balancer has moved. Some people follow the HFile count and will
> trigger a major when there are x amount of files. Others will just follow
> best practices and do it daily. Some do very little writes and only
> compact once a week.
>
> On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hello,
> >
> > What are the main metrics/signals people use/look at to say "OK, I really
> > need to do a major compaction now", other than "hm, performance is not
> > good, let's try major compaction tonight"?
> >
> > Thanks,
> > Otis
> > --
> > HBASE Performance Monitoring - http://sematext.com/spm/index.html
> > Search Analytics - http://sematext.com/search-analytics/index.html
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>
Re: How to know it's time for a major compaction?
Posted by Kevin O'dell <ke...@cloudera.com>.
Otis,
I think it depends on your use case. Some use cases are extremely
dependent on locality and latency, so they will major compact any region
the balancer has moved. Some people follow the HFile count and will
trigger a major when there are x amount of files. Others will just follow
best practices and do it daily. Some do very little writes and only
compact once a week.
On Wed, Dec 12, 2012 at 3:15 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hello,
>
> What are the main metrics/signals people use/look at to say "OK, I really
> need to do a major compaction now", other than "hm, performance is not
> good, let's try major compaction tonight"?
>
> Thanks,
> Otis
> --
> HBASE Performance Monitoring - http://sematext.com/spm/index.html
> Search Analytics - http://sematext.com/search-analytics/index.html
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera