You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Biju Nair (JIRA)" <ji...@apache.org> on 2019/05/03 21:37:00 UTC
[jira] [Updated] (HBASE-21000) Default limits for
PressureAwareCompactionThroughputController are too low
[ https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Biju Nair updated HBASE-21000:
------------------------------
Component/s: Compaction
> Default limits for PressureAwareCompactionThroughputController are too low
> --------------------------------------------------------------------------
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
> Affects Versions: 1.5.0
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 1.5.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**
> * A throughput controller which uses the follow schema to limit throughput
> * <ul>
> * <li>If compaction pressure is greater than 1.0, no limitation.</li>
> * <li>In off peak hours, use a fixed throughput limitation
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}</li>
> * <li>In normal hours, the max throughput is tuned between
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the formula "lower +
> * (higher - lower) * compactionPressure", where compactionPressure is in range [0.0, 1.0]</li>
> * </ul>
> */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
> public static final String HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> "hbase.hstore.compaction.throughput.higher.bound";
> private static final long DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> 20L * 1024 * 1024;
> public static final String HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> "hbase.hstore.compaction.throughput.lower.bound";
> private static final long DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> 10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and write clients are backed off or failing due to RegionTooBusy exceptions. Although compaction throughput becomes unbounded after the store reaches the blocking file count, in the lead up to this the default settings do not provide enough bandwidth to stave off blocking. The defaults should be increased.
> I'm not sure what new good defaults make sense. We could start by doubling them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)