You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by yanivG <ya...@gmail.com> on 2014/02/02 22:14:47 UTC
Major compaction issues in hbase
Hi,
I am using hbase 0.94.6 version. We are developing a java application and
using it as our storage for a high load web application.
The average ration is about 500 requests per second.
>From time to time I see high latency (more then 300 ms!!!) when trying to
store data object within hbase (using Put).
We disabled major compaction and set it to run manually once a week.
By examining the logs during those high latency , I see that major
compaction was executed.
I saw a post
(https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM) which
explains it.
The question is how can I avoid this? Should I ran the major compaction once
a day in low load hours? will it eliminate the sporadic major compaction?
Thanks,
Y
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Major compaction issues in hbase
Posted by Ted Yu <yu...@gmail.com>.
You can use off peak compaction: https://issues.apache.org/jira/browse/HBASE-4463
See also http://www.slideshare.net/cloudera/hbasecon-2013-compaction-improvements-in-apache-hbase
On Feb 2, 2014, at 1:14 PM, yanivG <ya...@gmail.com> wrote:
> Hi,
> I am using hbase 0.94.6 version. We are developing a java application and
> using it as our storage for a high load web application.
> The average ration is about 500 requests per second.
> From time to time I see high latency (more then 300 ms!!!) when trying to
> store data object within hbase (using Put).
>
> We disabled major compaction and set it to run manually once a week.
> By examining the logs during those high latency , I see that major
> compaction was executed.
> I saw a post
> (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM) which
> explains it.
> The question is how can I avoid this? Should I ran the major compaction once
> a day in low load hours? will it eliminate the sporadic major compaction?
> Thanks,
> Y
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Major compaction issues in hbase
Posted by Vladimir Rodionov <vl...@gmail.com>.
I think, the major issue here - not a sudden major compaction, but why
everything, literally, stalls when major compaction kicks off? Is there any
clear explanation, somewhere, how does major compaction can affect tput,
latency, availability of HBase? 300ms latency is possible, of course , as
an rare outlier, but if its 90% percentile - its totally different story.
-Vladimir
On Sun, Feb 2, 2014 at 1:49 PM, Kevin O'dell <ke...@cloudera.com>wrote:
> Hi Y,
>
> That may help, but a coulple quick questions:
>
> Are you doing any bulk loads?
>
> What is your avg flush size through out the day?
>
> After a day before how many storefiles have you built up?
>
> I would look at your memory sizing along with scheduling daily compactions.
>
>
> On Sun, Feb 2, 2014 at 4:14 PM, yanivG <ya...@gmail.com> wrote:
>
> > Hi,
> > I am using hbase 0.94.6 version. We are developing a java application and
> > using it as our storage for a high load web application.
> > The average ration is about 500 requests per second.
> > From time to time I see high latency (more then 300 ms!!!) when trying to
> > store data object within hbase (using Put).
> >
> > We disabled major compaction and set it to run manually once a week.
> > By examining the logs during those high latency , I see that major
> > compaction was executed.
> > I saw a post
> > (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM)
> > which
> > explains it.
> > The question is how can I avoid this? Should I ran the major compaction
> > once
> > a day in low load hours? will it eliminate the sporadic major compaction?
> > Thanks,
> > Y
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> > Sent from the HBase Developer mailing list archive at Nabble.com.
> >
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
>
Re: Major compaction issues in hbase
Posted by Nick Dimiduk <nd...@gmail.com>.
"Memory sizing" generally means how much RAM you have allocated to HBase.
The amount of heap, the total size of your memstores and the number of
them. This will impact the size of an individual memstore and thus the size
of the new HFiles flushed to disk. This impacts the frequency and character
of compactions. More column families => more memstores => (usually) smaller
flush sizes => more compactions.
On Mon, Feb 3, 2014 at 4:34 AM, yanivG <ya...@gmail.com> wrote:
> Hi Kevin,
> 1. No I am not doing bulk loads.
> 2. I don't know. Where can I see this metric? the load is about 200
> requests
> per second * 5K of data.
> 3. I don't know.
>
> Memory sizing? You mean heap size, etc?
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592p4055604.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
Re: Major compaction issues in hbase
Posted by yanivG <ya...@gmail.com>.
Hi Kevin,
1. No I am not doing bulk loads.
2. I don't know. Where can I see this metric? the load is about 200 requests
per second * 5K of data.
3. I don't know.
Memory sizing? You mean heap size, etc?
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592p4055604.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Major compaction issues in hbase
Posted by Kevin O'dell <ke...@cloudera.com>.
Hi Y,
That may help, but a coulple quick questions:
Are you doing any bulk loads?
What is your avg flush size through out the day?
After a day before how many storefiles have you built up?
I would look at your memory sizing along with scheduling daily compactions.
On Sun, Feb 2, 2014 at 4:14 PM, yanivG <ya...@gmail.com> wrote:
> Hi,
> I am using hbase 0.94.6 version. We are developing a java application and
> using it as our storage for a high load web application.
> The average ration is about 500 requests per second.
> From time to time I see high latency (more then 300 ms!!!) when trying to
> store data object within hbase (using Put).
>
> We disabled major compaction and set it to run manually once a week.
> By examining the logs during those high latency , I see that major
> compaction was executed.
> I saw a post
> (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM)
> which
> explains it.
> The question is how can I avoid this? Should I ran the major compaction
> once
> a day in low load hours? will it eliminate the sporadic major compaction?
> Thanks,
> Y
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
--
Kevin O'Dell
Systems Engineer, Cloudera
Re: Major compaction issues in hbase
Posted by yanivG <ya...@gmail.com>.
JM,
Can you elaborate on: "Also, this might be related to your design (too many
column families?),
your usage and your configuration (compaction triggers, memory
configuration, etc.). "?
We have 3 CF.
Compaction triggers?
memory configuration - do you mean GC settings?
Y
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592p4055605.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: Major compaction issues in hbase
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Yaniv,
When there is more than 3 files for a region (default setting), HBase will
trigger a minor compaction for this region. However, if all the regions
need to be compacted, then it will promote it as a major compaction. So
even by disabling major compaction so can still see some of it in the logs.
If you can run major compaction one a day it's better than once a week.
Also, this might be related to your design (too many column families?),
your usage and your configuration (compaction triggers, memory
configuration, etc.).
JM
2014-02-02 yanivG <ya...@gmail.com>:
> Hi,
> I am using hbase 0.94.6 version. We are developing a java application and
> using it as our storage for a high load web application.
> The average ration is about 500 requests per second.
> From time to time I see high latency (more then 300 ms!!!) when trying to
> store data object within hbase (using Put).
>
> We disabled major compaction and set it to run manually once a week.
> By examining the logs during those high latency , I see that major
> compaction was executed.
> I saw a post
> (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM)
> which
> explains it.
> The question is how can I avoid this? Should I ran the major compaction
> once
> a day in low load hours? will it eliminate the sporadic major compaction?
> Thanks,
> Y
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>