You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sumit Nigam <su...@yahoo.com.INVALID> on 2016/04/04 17:35:59 UTC
Major compaction
Hi,
Are there major overheads to running major compaction frequently? As much as I know, it produces one Hfile for a region and processes delete markers and version related drops. So, if this process has happened once say. a few mins back then another major compaction should ideally not cause much harm.
Why I am trying to understand this is because Hbase also sets it to 24 hour default (for time based compaction) and I am looking to lower it to say 20 mins to reduce stress by spreading the load.
Or am I completely off-track?
Thanks,Sumit
Re: Major compaction
Posted by Esteban Gutierrez <es...@cloudera.com>.
Why is RS is going down? are you hitting an OOME or just the RS goes into
heavy GC and you get a YouAreDeadException? my guess is that if you have
thousands of store files and you are major compacting probably you are low
in resources and the compaction queue goes out of control and triggers an
OOME but thats just an educated guess.
Also if you are generating 100s or 1000s of store files per day probably
the memstore settings are not the optimal for your workload and you are
triggering too many premature flushes. If you could upload the RS logs to
pastebin or another similar site we can take a look into and see if thats
the case.
thanks!
esteban.
--
Cloudera, Inc.
On Mon, Apr 4, 2016 at 9:29 PM, Sumit Nigam <su...@yahoo.com.invalid>
wrote:
> Hello all,
> Thanks a lot for your replies.
> @Frank - I will try the compactor you wrote and let you know how it goes.
> @Esteban - I am trying to understand how to reduce major compaction load.
> In my cluster whenever it happens, it takes down a region server or two. My
> setting are - disable time based compaction, make blocking files = 1000,
> min files to compact = 5, max files to compact = 10. Other settings are
> defaults. So, a question I had is if a major compaction of certain table is
> performed a few mins back, then would another major compaction of the same
> take a lot of time? If not, why not process delete markers/ version related
> expulsions more often by kicking in major compaction more often? Assuming,
> splits are not happening after every major compaction cycle, we should be
> able to manage major compaction better by spreading this load more evenly
> rather than doing once a day? Of course, I am missing something?
> @Vladimir - By same understanding that I mentioned above, would every
> major compaction really lead to more disk I/O or n/w or hbase process load?
> I assumed that the main things done by major compaction are deletes,
> creating a huge file per store and region splits. If this process has
> already been done say, a few mins back (or could be an hour back), then the
> next compaction should ideally have less to do. Not sure, if I am
> misunderstanding major compaction altogether.
> Please let me know your inputs.
> Best regards,Sumit
>
> From: Frank Luo <jl...@merkleinc.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc: Sumit Nigam <su...@yahoo.com>
> Sent: Monday, April 4, 2016 11:07 PM
> Subject: RE: Major compaction
>
> I wrote a small program to do MC in a "smart" way here:
> https://github.com/jinyeluo/smarthbasecompactor/
>
> Instead of blindly running MC on a table level, the program find a non-hot
> regions that has most store-files on a per region-server base, and run MC
> on them. Once done, it finds the next candidates... It just keeps on going
> until time is up.
>
> I am sure it has a lot area for improvement if something wants to go crazy
> on. But the code has been running for about half a year and it seems
> working well.
>
> -----Original Message-----
> From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
> Sent: Monday, April 04, 2016 12:15 PM
> To: user@hbase.apache.org
> Cc: Sumit Nigam <su...@yahoo.com>
> Subject: Re: Major compaction
>
> >> Why I am trying to understand this is because Hbase also sets it to
> >> 24
> hour default (for time based compaction) and I am looking to lower it to
> say >> 20 mins to reduce stress by spreading the load.
>
> The more frequently you run major compaction the more IO (disk/network)
> you consume.
>
> Usually, in production environment, periodic major compactions are
> disabled and run manually to avoid major compaction storms.
>
> To control major compaction completely you will also need to disable
> promotion minor compaction to major ones. You can do this, by setting
> maximum compaction size for minor compaction:
> *hbase.hstore.compaction.max.size*
>
> -Vlad
>
>
> On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > Hello Sumit,
> >
> > Ideally you shouldn't be triggering major compactions that frequently
> > since minor compactions should be taking care of reducing the number
> > of store files. The caveat of doing it more frequently is the
> > additional disk/network I/O.
> >
> > Can you please elaborate more on "reduce stress by spreading the
> > load." Is there anything else you are seeing in your cluster that is
> > suggesting to you to lower the period for major compactions?
> >
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam
> > <su...@yahoo.com.invalid>
> > wrote:
> >
> > > Hi,
> > > Are there major overheads to running major compaction frequently? As
> > > much as I know, it produces one Hfile for a region and processes
> > > delete
> > markers
> > > and version related drops. So, if this process has happened once
> > > say. a
> > few
> > > mins back then another major compaction should ideally not cause
> > > much
> > harm.
> > > Why I am trying to understand this is because Hbase also sets it to
> > > 24 hour default (for time based compaction) and I am looking to
> > > lower it to say 20 mins to reduce stress by spreading the load.
> > > Or am I completely off-track?
> > > Thanks,Sumit
> >
> Merkle was named a leader in Customer Insights Services Providers by
> Forrester Research
> <
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
>
> Forrester Research report names 500friends, a Merkle Company, a leader in
> customer Loyalty Solutions for Midsize Organizations<
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
>
>
>
Re: Major compaction
Posted by Sumit Nigam <su...@yahoo.com.INVALID>.
Hello all,
Thanks a lot for your replies.
@Frank - I will try the compactor you wrote and let you know how it goes.
@Esteban - I am trying to understand how to reduce major compaction load. In my cluster whenever it happens, it takes down a region server or two. My setting are - disable time based compaction, make blocking files = 1000, min files to compact = 5, max files to compact = 10. Other settings are defaults. So, a question I had is if a major compaction of certain table is performed a few mins back, then would another major compaction of the same take a lot of time? If not, why not process delete markers/ version related expulsions more often by kicking in major compaction more often? Assuming, splits are not happening after every major compaction cycle, we should be able to manage major compaction better by spreading this load more evenly rather than doing once a day? Of course, I am missing something?
@Vladimir - By same understanding that I mentioned above, would every major compaction really lead to more disk I/O or n/w or hbase process load? I assumed that the main things done by major compaction are deletes, creating a huge file per store and region splits. If this process has already been done say, a few mins back (or could be an hour back), then the next compaction should ideally have less to do. Not sure, if I am misunderstanding major compaction altogether.
Please let me know your inputs.
Best regards,Sumit
From: Frank Luo <jl...@merkleinc.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc: Sumit Nigam <su...@yahoo.com>
Sent: Monday, April 4, 2016 11:07 PM
Subject: RE: Major compaction
I wrote a small program to do MC in a "smart" way here: https://github.com/jinyeluo/smarthbasecompactor/
Instead of blindly running MC on a table level, the program find a non-hot regions that has most store-files on a per region-server base, and run MC on them. Once done, it finds the next candidates... It just keeps on going until time is up.
I am sure it has a lot area for improvement if something wants to go crazy on. But the code has been running for about half a year and it seems working well.
-----Original Message-----
From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
Sent: Monday, April 04, 2016 12:15 PM
To: user@hbase.apache.org
Cc: Sumit Nigam <su...@yahoo.com>
Subject: Re: Major compaction
>> Why I am trying to understand this is because Hbase also sets it to
>> 24
hour default (for time based compaction) and I am looking to lower it to say >> 20 mins to reduce stress by spreading the load.
The more frequently you run major compaction the more IO (disk/network) you consume.
Usually, in production environment, periodic major compactions are disabled and run manually to avoid major compaction storms.
To control major compaction completely you will also need to disable promotion minor compaction to major ones. You can do this, by setting maximum compaction size for minor compaction:
*hbase.hstore.compaction.max.size*
-Vlad
On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Hello Sumit,
>
> Ideally you shouldn't be triggering major compactions that frequently
> since minor compactions should be taking care of reducing the number
> of store files. The caveat of doing it more frequently is the
> additional disk/network I/O.
>
> Can you please elaborate more on "reduce stress by spreading the
> load." Is there anything else you are seeing in your cluster that is
> suggesting to you to lower the period for major compactions?
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam
> <su...@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > Are there major overheads to running major compaction frequently? As
> > much as I know, it produces one Hfile for a region and processes
> > delete
> markers
> > and version related drops. So, if this process has happened once
> > say. a
> few
> > mins back then another major compaction should ideally not cause
> > much
> harm.
> > Why I am trying to understand this is because Hbase also sets it to
> > 24 hour default (for time based compaction) and I am looking to
> > lower it to say 20 mins to reduce stress by spreading the load.
> > Or am I completely off-track?
> > Thanks,Sumit
>
Merkle was named a leader in Customer Insights Services Providers by Forrester Research
<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
Forrester Research report names 500friends, a Merkle Company, a leader in customer Loyalty Solutions for Midsize Organizations<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
答复: Major compaction
Posted by "Liu, Ming (Ming)" <mi...@esgyn.cn>.
Thanks Frank, this is something I am looking for. Would like to have a try with it.
Thanks,
Ming
-----邮件原件-----
发件人: Frank Luo [mailto:jluo@merkleinc.com]
发送时间: 2016年4月5日 1:38
收件人: user@hbase.apache.org
抄送: Sumit Nigam <su...@yahoo.com>
主题: RE: Major compaction
I wrote a small program to do MC in a "smart" way here: https://github.com/jinyeluo/smarthbasecompactor/
Instead of blindly running MC on a table level, the program find a non-hot regions that has most store-files on a per region-server base, and run MC on them. Once done, it finds the next candidates... It just keeps on going until time is up.
I am sure it has a lot area for improvement if something wants to go crazy on. But the code has been running for about half a year and it seems working well.
-----Original Message-----
From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
Sent: Monday, April 04, 2016 12:15 PM
To: user@hbase.apache.org
Cc: Sumit Nigam <su...@yahoo.com>
Subject: Re: Major compaction
>> Why I am trying to understand this is because Hbase also sets it to
>> 24
hour default (for time based compaction) and I am looking to lower it to say >> 20 mins to reduce stress by spreading the load.
The more frequently you run major compaction the more IO (disk/network) you consume.
Usually, in production environment, periodic major compactions are disabled and run manually to avoid major compaction storms.
To control major compaction completely you will also need to disable promotion minor compaction to major ones. You can do this, by setting maximum compaction size for minor compaction:
*hbase.hstore.compaction.max.size*
-Vlad
On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Hello Sumit,
>
> Ideally you shouldn't be triggering major compactions that frequently
> since minor compactions should be taking care of reducing the number
> of store files. The caveat of doing it more frequently is the
> additional disk/network I/O.
>
> Can you please elaborate more on "reduce stress by spreading the
> load." Is there anything else you are seeing in your cluster that is
> suggesting to you to lower the period for major compactions?
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam
> <su...@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > Are there major overheads to running major compaction frequently? As
> > much as I know, it produces one Hfile for a region and processes
> > delete
> markers
> > and version related drops. So, if this process has happened once
> > say. a
> few
> > mins back then another major compaction should ideally not cause
> > much
> harm.
> > Why I am trying to understand this is because Hbase also sets it to
> > 24 hour default (for time based compaction) and I am looking to
> > lower it to say 20 mins to reduce stress by spreading the load.
> > Or am I completely off-track?
> > Thanks,Sumit
>
Merkle was named a leader in Customer Insights Services Providers by Forrester Research <http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
Forrester Research report names 500friends, a Merkle Company, a leader in customer Loyalty Solutions for Midsize Organizations<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
RE: Major compaction
Posted by Frank Luo <jl...@merkleinc.com>.
I wrote a small program to do MC in a "smart" way here: https://github.com/jinyeluo/smarthbasecompactor/
Instead of blindly running MC on a table level, the program find a non-hot regions that has most store-files on a per region-server base, and run MC on them. Once done, it finds the next candidates... It just keeps on going until time is up.
I am sure it has a lot area for improvement if something wants to go crazy on. But the code has been running for about half a year and it seems working well.
-----Original Message-----
From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
Sent: Monday, April 04, 2016 12:15 PM
To: user@hbase.apache.org
Cc: Sumit Nigam <su...@yahoo.com>
Subject: Re: Major compaction
>> Why I am trying to understand this is because Hbase also sets it to
>> 24
hour default (for time based compaction) and I am looking to lower it to say >> 20 mins to reduce stress by spreading the load.
The more frequently you run major compaction the more IO (disk/network) you consume.
Usually, in production environment, periodic major compactions are disabled and run manually to avoid major compaction storms.
To control major compaction completely you will also need to disable promotion minor compaction to major ones. You can do this, by setting maximum compaction size for minor compaction:
*hbase.hstore.compaction.max.size*
-Vlad
On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Hello Sumit,
>
> Ideally you shouldn't be triggering major compactions that frequently
> since minor compactions should be taking care of reducing the number
> of store files. The caveat of doing it more frequently is the
> additional disk/network I/O.
>
> Can you please elaborate more on "reduce stress by spreading the
> load." Is there anything else you are seeing in your cluster that is
> suggesting to you to lower the period for major compactions?
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam
> <su...@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > Are there major overheads to running major compaction frequently? As
> > much as I know, it produces one Hfile for a region and processes
> > delete
> markers
> > and version related drops. So, if this process has happened once
> > say. a
> few
> > mins back then another major compaction should ideally not cause
> > much
> harm.
> > Why I am trying to understand this is because Hbase also sets it to
> > 24 hour default (for time based compaction) and I am looking to
> > lower it to say 20 mins to reduce stress by spreading the load.
> > Or am I completely off-track?
> > Thanks,Sumit
>
Merkle was named a leader in Customer Insights Services Providers by Forrester Research
<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
Forrester Research report names 500friends, a Merkle Company, a leader in customer Loyalty Solutions for Midsize Organizations<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Re: Major compaction
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> Why I am trying to understand this is because Hbase also sets it to 24
hour default (for time based compaction) and I am looking to lower it to
say >> 20 mins to reduce stress by spreading the load.
The more frequently you run major compaction the more IO (disk/network) you
consume.
Usually, in production environment, periodic major compactions are disabled
and run manually to avoid major compaction storms.
To control major compaction completely you will also need to disable
promotion minor compaction to major ones. You can do this, by setting
maximum compaction size for minor compaction:
*hbase.hstore.compaction.max.size*
-Vlad
On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Hello Sumit,
>
> Ideally you shouldn't be triggering major compactions that frequently since
> minor compactions should be taking care of reducing the number of store
> files. The caveat of doing it more frequently is the additional
> disk/network I/O.
>
> Can you please elaborate more on "reduce stress by spreading the load." Is
> there anything else you are seeing in your cluster that is suggesting to
> you to lower the period for major compactions?
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam <su...@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > Are there major overheads to running major compaction frequently? As much
> > as I know, it produces one Hfile for a region and processes delete
> markers
> > and version related drops. So, if this process has happened once say. a
> few
> > mins back then another major compaction should ideally not cause much
> harm.
> > Why I am trying to understand this is because Hbase also sets it to 24
> > hour default (for time based compaction) and I am looking to lower it to
> > say 20 mins to reduce stress by spreading the load.
> > Or am I completely off-track?
> > Thanks,Sumit
>
Re: Major compaction
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hello Sumit,
Ideally you shouldn't be triggering major compactions that frequently since
minor compactions should be taking care of reducing the number of store
files. The caveat of doing it more frequently is the additional
disk/network I/O.
Can you please elaborate more on "reduce stress by spreading the load." Is
there anything else you are seeing in your cluster that is suggesting to
you to lower the period for major compactions?
esteban.
--
Cloudera, Inc.
On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam <su...@yahoo.com.invalid>
wrote:
> Hi,
> Are there major overheads to running major compaction frequently? As much
> as I know, it produces one Hfile for a region and processes delete markers
> and version related drops. So, if this process has happened once say. a few
> mins back then another major compaction should ideally not cause much harm.
> Why I am trying to understand this is because Hbase also sets it to 24
> hour default (for time based compaction) and I am looking to lower it to
> say 20 mins to reduce stress by spreading the load.
> Or am I completely off-track?
> Thanks,Sumit