[jira] [Comment Edited] (HBASE-11208) Remove the hbase.hstor.blockingStoreFiles setting

["Nicolas Liochon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004344#comment-14004344 ] 

Nicolas Liochon edited comment on HBASE-11208 at 5/21/14 4:46 AM:
------------------------------------------------------------------

bq. The only other mechanism is OOM'ing.
With this setting we're actually putting more pressure to the memory, by delaying the flush instead of writing the file. It lowers the pressure on the namenode and on the compaction (as we create less files), but we use more memory.

The gbif team has changed this parameter to 200 (see http://gbif.blogspot.fr/2012/07/optimizing-writes-in-hbase.html), and, as far as I can tell, it's not an uncommon practice. 

bq. HBASE-5162
bq.  How else do you purpose throttling clients?
We have two parameters for OOM:
- the block multiplier (not the ideal way imho, as said in HBASE-11209)
- the memory we allocate globally for the memstore: we block the writes if we reach this level.

I feel that we can remove this setting even if HBASE-5162 is not there... But of course, HBASE-5162  would be great to have. I will have a look as well.

bq. we'd then prevent condition where a region can get filled with hundreds of hfiles. Client push-back?
Should we not be able to do more compactions in parallel in such a case?

The issue with client push back is that it creates write latency.



was (Author: nkeywal):
bq. The only other mechanism is OOM'ing.
With this setting we're actually putting more pressure to the memory, by delaying the flush instead of writing the file. It lowers the pressure on the namenode and on the compaction (as we create less files), but we use more memory.

The gbif team has changed this parameter to 200 (see http://gbif.blogspot.fr/2012/07/optimizing-writes-in-hbase.html), and, as far as I can tell, it's not an uncommon practice. 

bq. HBASE-5162
bq.  How else do you purpose throttling clients?
We have two parameters for OOM:
- the block multiplier (not the ideal way imho, as said in HBASE-11209)
- the memory we allocate globally for the memstore: we block the writes if we reach this level.

I feel that we can remove this setting even if HBASE-5162 is not there... But of course, HBASE-5162  would be great to have. I will have a look as well.

bq. we'd then prevent condition where a region can get filled with hundreds of hfiles. Client push-back?
Should we not be able to do more compactions in parallel in such a case?

The issue with client push back is th


> Remove the hbase.hstor.blockingStoreFiles setting
> -------------------------------------------------
>
>                 Key: HBASE-11208
>                 URL: https://issues.apache.org/jira/browse/HBASE-11208
>             Project: HBase
>          Issue Type: Brainstorming
>          Components: Compaction, regionserver
>    Affects Versions: 0.99.0
>            Reporter: Nicolas Liochon
>            Assignee: Nicolas Liochon
>             Fix For: 0.99.0
>
>
> It's a little bit of a provocation, but the rational is:
>  - there are some bugs around the delayed flush. For example, if the periodic scheduler has asked for a delayed flush, and that we need to flush, we will have to wait
>  - if the number of WAL files increases, we won't flush immediately if the blockingFile number has been reached. This impacts the MTTR.
>  - We don't write to limit the compaction impact, but they are many cases where we would want to flush anyway, as the writes cannot wait.
>  - this obviously leads to huge write latency peaks.
> So I'm questioning this setting, it leads to multiple intricate cases, unpredictable write latency, and looks like a workaround for compaction performances. With all the work done on compaction, I think we can get rid of it.  A solution in the middle would be to deprecate it and to set it to a large value...
> Any opinion before I shoot :-) ? 



--
This message was sent by Atlassian JIRA
(v6.2#6252)