You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Dimitri BEKALE (Jira)" <ji...@apache.org> on 2026/10/20 08:50:00 UTC

[jira] [Created] (HBASE-30485) ExceedOperationQuota passes writeCapacityUnitConsumed as the readCapacityUnit argument of grabQuota

Dimitri BEKALE created HBASE-30485:
--------------------------------------

             Summary: ExceedOperationQuota passes writeCapacityUnitConsumed as the readCapacityUnit argument of grabQuota
                 Key: HBASE-30485
                 URL: https://issues.apache.org/jira/browse/HBASE-30485
             Project: HBase
          Issue Type: Bug
          Components: Quotas
    Affects Versions: 2.6.6, 3.0.0-beta-2
            Reporter: Dimitri BEKALE


When exceed-throttle is enabled (\{{Admin.exceedThrottleQuotaSwitch(true)}}),
{\{RegionServerRpcQuotaManager}} uses \{{ExceedOperationQuota}}. Its \{{checkQuota}} method debits
the limiters with \{{grabQuota}}, and both calls pass the write capacity-unit estimate in the read
capacity-unit position
([ExceedOperationQuota.java:97-104|https://github.com/apache/hbase/blob/09474ab77bfa8bede96131164fd4451f093a1a10/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/ExceedOperationQuota.java#L97-L104]):

{code:java}
regionServerLimiter.grabQuota(numWrites, writeConsumed, numReads + numScans, readConsumed,
  writeCapacityUnitConsumed, writeCapacityUnitConsumed, isAtomic, handlerUsageTimeConsumed);
...
limiter.grabQuota(numWrites, writeConsumed, numReads + numScans, readConsumed,
  writeCapacityUnitConsumed, writeCapacityUnitConsumed, isAtomic, 0L);
{code}

The signature is {{grabQuota(writeReqs, writeSize, readReqs, readSize, writeCapacityUnit,
readCapacityUnit, isAtomic, estimateHandlerThreadUsageMs)}} (\{{QuotaLimiter.java:70}}): the fifth
and sixth arguments are the write and read capacity units. The
other three call sites in the same class hierarchy pass \{{readCapacityUnitConsumed}} there:
{\{ExceedOperationQuota.checkQuota}} step 2 (line 82), \{{DefaultOperationQuota.checkQuota}} (line
151), and \{{ExceedOperationQuota.close()}} via \{{consumeRead}}.

h3. Effect

{\{TimeBasedLimiter.grabQuota}} only consumes \{{readCapacityUnitLimiter}} when the
{\{readCapacityUnit}} argument is positive, so with exceed-throttle on and capacity-unit quotas set
on the region server:

* A pure read (\{{writeCapacityUnitConsumed == 0}}) debits *zero* read capacity units. The quota is
  checked against the correct estimate in step 2 but never charged in step 4, so read CU throttling
  does not throttle. When \{{exceed}} is true (step 5) the same happens to the user/table/namespace
  limiters.
* A pure write debits the read CU limiter by the *write* CU estimate and the request CU limiter
  twice, so writes spuriously exhaust the read CU quota.

h3. Origin and affected versions

Introduced with the feature in HBASE-21783 (\{{9a55cbb2c1df}}, 2019-02-21, shipped in 2.2.0) — the
original hunk already had \{{writeCapacityUnitConsumed, writeCapacityUnitConsumed}}. HBASE-29231
reformatted the lines in 2025 and kept the duplicate. Present on \{{master}}, \{{branch-3}},
{\{branch-2}}, \{{branch-2.6}} and \{{rel/2.6.6}}.

{\{TestQuotaThrottle.testExceedThrottleQuota}} only exercises \{{READ_NUMBER}} / \{{WRITE_NUMBER}},
and the capacity-unit tests never enable exceed-throttle, which is presumably why this has not been
noticed.

h3. Reproducer

A unit test in the style of \{{TestDefaultOperationQuota}}: region-server limiter with
{\{READ_CAPACITY_UNIT}} soft limit 100/s and an unlimited \{{WRITE_CAPACITY_UNIT}}, an
{\{ExceedOperationQuota}} with no other limiters, two consecutive \{{checkBatchQuota(0, 1, false)}}
calls. One read is estimated at one 65536-byte block = 64 CU at the default 1024-byte CU size, so
the first read fits and should leave 36 CU; the second needs 64 and must throw
{\{RpcThrottlingException}}. On current master the limiter is debited 0 read CU and the second call
passes:

{noformat}
Tests run: 14, Failures: 1, Errors: 0, Skipped: 0
TestDefaultOperationQuota.testExceedQuotaGrabsReadCapacityUnits:318
  Expected org.apache.hadoop.hbase.quotas.RpcThrottlingException to be thrown, but nothing was thrown.
{noformat}

With the two-line fix below the same run is 14/14 green (the 13 existing tests pass in both runs).
Verified on \{{master}} at \{{09474ab77b}} with JDK 21.

{code:java}
@Test
public void testExceedQuotaGrabsReadCapacityUnits() throws RpcThrottlingException {
  QuotaProtos.Throttle rsThrottle = QuotaProtos.Throttle.newBuilder()
    .setReadCapacityUnit(QuotaProtos.TimedQuota.newBuilder().setSoftLimit(100)
      .setTimeUnit(HBaseProtos.TimeUnit.SECONDS).build())
    .setWriteCapacityUnit(QuotaProtos.TimedQuota.newBuilder().setSoftLimit(Long.MAX_VALUE)
      .setTimeUnit(HBaseProtos.TimeUnit.SECONDS).build())
    .build();
  QuotaLimiter rsLimiter = TimeBasedLimiter.fromThrottle(conf, rsThrottle);
  // no user/table/namespace limiters: only the region server quota applies
  ExceedOperationQuota quota =
    new ExceedOperationQuota(new Configuration(), 65536, DEFAULT_REQUESTS_PER_SECOND, rsLimiter);

  // one read = one 65536-byte block = 64 read CU; fits in 100 and must leave 36
  quota.checkBatchQuota(0, 1, false);

  // the second read needs 64 CU but only 36 remain, so it must be throttled
  assertThrows(RpcThrottlingException.class, () -> quota.checkBatchQuota(0, 1, false));
}
{code}

h3. Proposed fix

{code}
-        writeCapacityUnitConsumed, writeCapacityUnitConsumed, isAtomic, handlerUsageTimeConsumed);
+        writeCapacityUnitConsumed, readCapacityUnitConsumed, isAtomic, handlerUsageTimeConsumed);
...
-            writeCapacityUnitConsumed, writeCapacityUnitConsumed, isAtomic, 0L);
+            writeCapacityUnitConsumed, readCapacityUnitConsumed, isAtomic, 0L);
{code}

I ran only this test class, not the full suite or the integration test \{{TestQuotaThrottle}}.
Happy to open a PR with the fix and the test if that is welcome.

h3. Provenance

This was found during an academic study of LLM-assisted code review at Jeonbuk National University, Jeonju, South Korea. An automated LLM-based reviewer produced the initial analysis and I verified it by reading the source at \{{09474ab77bfa}} and the \{{QuotaLimiter}} signature, and by running the unit test above against the unpatched and patched class. I believe it is a genuine defect and will respect your triage either way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)