You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sumit Nigam <su...@yahoo.com.INVALID> on 2016/04/03 13:32:30 UTC
Compacting same table
Hi,
I have been informed that compacting (manual) the same hbase table takes same amount of time even when done in quick succession. This seems counter-intuitive because an already compacted table should not take same amount of time.
Also, what is the use of hbase.hstore.compaction.kv.max setting in compaction? I am unable to determine its implications on other compaction tuning factors? The default of 10 seems too less.
Thanks,Sumit
Re: Compacting same table
Posted by Sumit Nigam <su...@yahoo.com.INVALID>.
Hi Ted,
Thank you for your reply.
Yes, the source is an internal team which seemed to have done compaction of the same table twice in succession and made such an observation. To me, that is counter-intuitive because the Hbase code also has clear logic to show that it does not re-compact store files already compacted. However, I need to understand why the second compaction took so much time.
On "hbase.hstore.compaction.kv.max - The maximum number of KeyValues to read and then write in a batch when flushing or compacting"I had a question here.
Does this mean that from a given Hfile, as many KVs would be read at one time? So, if I increase memstore size, I should also increase this value because default value I believe is 10, which is probably too less.
Thanks,Sumit
From: Ted Yu <yu...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>; Sumit Nigam <su...@yahoo.com>
Sent: Sunday, April 3, 2016 7:15 PM
Subject: Re: Compacting same table
bq. I have been informed
Can you disclose the source of such information ?
For hbase.hstore.compaction.kv.max , hbase-default.xml has:
The maximum number of KeyValues to read and then write in a batch when
flushing or
compacting. Set this lower if you have big KeyValues and problems
with Out Of Memory
Exceptions Set this higher if you have wide, small rows.
Is the above description not clear ?
Thanks
On Sun, Apr 3, 2016 at 4:32 AM, Sumit Nigam <su...@yahoo.com.invalid>
wrote:
> Hi,
> I have been informed that compacting (manual) the same hbase table takes
> same amount of time even when done in quick succession. This seems
> counter-intuitive because an already compacted table should not take same
> amount of time.
> Also, what is the use of hbase.hstore.compaction.kv.max setting in
> compaction? I am unable to determine its implications on other compaction
> tuning factors? The default of 10 seems too less.
> Thanks,Sumit
Re: Compacting same table
Posted by Ted Yu <yu...@gmail.com>.
bq. I have been informed
Can you disclose the source of such information ?
For hbase.hstore.compaction.kv.max , hbase-default.xml has:
The maximum number of KeyValues to read and then write in a batch when
flushing or
compacting. Set this lower if you have big KeyValues and problems
with Out Of Memory
Exceptions Set this higher if you have wide, small rows.
Is the above description not clear ?
Thanks
On Sun, Apr 3, 2016 at 4:32 AM, Sumit Nigam <su...@yahoo.com.invalid>
wrote:
> Hi,
> I have been informed that compacting (manual) the same hbase table takes
> same amount of time even when done in quick succession. This seems
> counter-intuitive because an already compacted table should not take same
> amount of time.
> Also, what is the use of hbase.hstore.compaction.kv.max setting in
> compaction? I am unable to determine its implications on other compaction
> tuning factors? The default of 10 seems too less.
> Thanks,Sumit