[jira] [Assigned] (HBASE-28672) Large batch requests can be blocked indefinitely by quotas

["Ray Mattingly (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-28672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ray Mattingly reassigned HBASE-28672:
-------------------------------------

    Assignee: Ray Mattingly

> Large batch requests can be blocked indefinitely by quotas
> ----------------------------------------------------------
>
>                 Key: HBASE-28672
>                 URL: https://issues.apache.org/jira/browse/HBASE-28672
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.6.0
>            Reporter: Ray Mattingly
>            Assignee: Ray Mattingly
>            Priority: Major
>
> At my day job we are trying to implement default quotas for a variety of access patterns. We began by introducing a default read IO limit per-user, per-machine — this has been very successful in reducing hotspots, even on clusters with thousands of distinct users.
> While implementing a default writes/second throttle, I realized that doing so would put us in a precarious situation where large-enough batches may never succeed. If your batch size is greater than your TimeLimiter's max throughput, then you will always fail in the quota estimation stage. Meanwhile [IO estimates are more optimistic|https://github.com/apache/hbase/blob/bdb3f216e864e20eb2b09352707a751a5cf7460f/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java#L192-L193], deliberately, which can let large requests do targeted oversubscription of an IO quota:
>  
> {code:java}
> // assume 1 block required for reads. this is probably a low estimate, which is okay
> readConsumed = numReads > 0 ? blockSizeBytes : 0;{code}
>  
> This is okay because the Limiter's availability will go negative and force a longer backoff on subsequent requests. I believe this is preferable UX compared to a doomed throttling loop.
> In my opinion, we should do something similar in batch request estimation, by estimating a batch request's workload at {{Math.min(batchSize, limiterMaxThroughput)}} rather than simply {{{}batchSize{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)