[jira] [Created] (HBASE-27704) Quotas can drastically overflow configured limit

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

Bryan Beaudreault created HBASE-27704:
-----------------------------------------

             Summary: Quotas can drastically overflow configured limit
                 Key: HBASE-27704
                 URL: https://issues.apache.org/jira/browse/HBASE-27704
             Project: HBase
          Issue Type: Bug
            Reporter: Bryan Beaudreault
         Attachments: Screenshot 2023-03-10 at 5.17.51 PM.png

The original implementation did not allow exceeding quota. For example, you specify a limit of 10 resource/sec and submit 20 resources, it takes 1.1 seconds to be able submit another request. This was covered by the [testOverconsumption in TestRateLimiter|https://github.com/apache/hbase/blame/587b0b4f20bdc0415b6541023e611b69c87dba15/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java#L97]. As an incidental part of HBASE-13686, that logic was changed. There is no mention of the reasoning behind the change in the issue comments or review board, I think it was missed. The goal of that issue was to add different refill strategies, but it also modified the over consumption. The testOverconsumption was [split out for both refill strategies|https://github.com/apache/hbase/blame/master/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java#L104-L159], but the core reasoning was lost. The comment says:
{code:java}
// 10 resources are available, but we need to consume 20 resources109    
// Verify that we have to wait at least 1.1sec to have 1 resource available {code}
But the actual test was updated to only require a new resource after 100ms. This is incorrect. 

The problem is, when consuming if you go negative it sets to 0 [here|https://github.com/apache/hbase/blame/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java#L187-L191]. Additionally, when refilling the new logic does a Math.max(0, refillAmount): [here|https://github.com/apache/hbase/blame/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java#L159-L163]. So it's really impossible to get below 0, which is impractical for a rate limiter. 

With this setup it's very easy to drastically overconsume the rate limiter. See attached screenshot, which shows two humps. The first one has the current logic, the second hump has my fix which removes both of those problems. The rate limit was set to 500mb/s, but I was easily able to go over 700 mb/s without the fix.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)