You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ajkh88 (via GitHub)" <gi...@apache.org> on 2025/05/19 13:48:00 UTC
[PR] HBASE-29231 - Thread handler usage throttling [hbase]
ajkh88 opened a new pull request, #7000:
URL: https://github.com/apache/hbase/pull/7000
[Upstream Issue](https://issues.apache.org/jira/browse/HBASE-29231)
This PR implements thread handler usage throttling support in HBase, enabling administrators to limit the amount of thread handler time that can be consumed across all threads.
## Features
- New throttle type: `REQUEST_HANDLER_USAGE_MS` that limits thread handler time usage
- Default throttle configuration via `hbase.quota.default.user.machine.request_handler_usage_ms`
- Throttling integration with existing quota infrastructure
## Implementation Details
- Enhanced `TimeBasedLimiter` to track and limit thread handler usage time
- Added handler usage time tracking to `DefaultOperationQuota`
- Added test cases to verify throttling behaviour for both reads and writes
This throttling capability helps prevent individual users or applications from monopolising RegionServer handler threads, improving overall service stability and responsiveness for HBase deployments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 Throttles should support limits based on handler thread usage time [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-3646873765
TestThreadHandlerUsageQuota fails for me locally most times, and it is also on the flaky dashboard.
Go deep into the logic, I found that I still can not understand the logic of calculateHandlerUsageTimeEstimate. In general, I think here we are trying to predicate how much time a request will be consumed, but why we just use `requestsPerSecond /1000`? How could this work? The unit of `requestsPerSecond / 1000` is `/ms`, but here what we want is `ms`? At least we should use reciprocal of this value?
Can someone explain more on this?
@rmdmattingly @ajkh88
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2119198564
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##########
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) {
return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond,
Review Comment:
Mind explain a bit more about the algorithm here? I do not fully understand what does 'handler usage time' mean...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "ajkh88 (via GitHub)" <gi...@apache.org>.
ajkh88 commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-2935256579
In addition to the tests in this PR, I have deployed this to a test cluster and ran a load test against it. Everything is functioning as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-2892801095
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 15s | | master passed |
| +1 :green_heart: | compile | 4m 30s | | master passed |
| +1 :green_heart: | checkstyle | 0m 58s | | master passed |
| +1 :green_heart: | spotbugs | 4m 35s | | master passed |
| +1 :green_heart: | spotless | 0m 46s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 4m 24s | | the patch passed |
| +1 :green_heart: | cc | 4m 24s | | the patch passed |
| +1 :green_heart: | javac | 4m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | spotbugs | 4m 48s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 2s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | hbaseprotoc | 1m 34s | | the patch passed |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not generate ASF License warnings. |
| | | 50m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7000 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc |
| uname | Linux 16300a5b6f0b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 549e24450d8d2e0273ed38c0d247c306a2972ba9 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 Throttles should support limits based on handler thread usage time [hbase]
Posted by "ajkh88 (via GitHub)" <gi...@apache.org>.
ajkh88 commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2128259789
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##########
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) {
return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond,
Review Comment:
Yes, that is pretty much it—we create the `OperationQuota` object and first 'check' it, meaning we estimate how much of the quota the operation will consume. The quota represents different resources for different throttle types (it could be time, space, etc.). Each quota type calculates its estimate differently, but they are all similar in that they don’t try anything too complicated; we just want a general idea of what will be used.
We compare our estimate to the current remaining quota, and if the estimate would consume it all, we throw an `RpcThrottlingException` (also calculating a wait interval here). This stops the request and backs off any retries.
If the estimate will not consume all the quota, then we 'grab', the quota - meaning we subtract the estimate from it.
Finally, when the `OperationQuota::close` method is called, we calculate the _actual_ time taken and update the quota accordingly, either adding to or subtracting from it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-2893146732
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 14s | | master passed |
| +1 :green_heart: | compile | 1m 51s | | master passed |
| +1 :green_heart: | javadoc | 0m 54s | | master passed |
| +1 :green_heart: | shadedjars | 5m 57s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 51s | | the patch passed |
| +1 :green_heart: | javac | 1m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 57s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 1m 34s | | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 210m 35s | | hbase-server in the patch passed. |
| | | 242m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7000 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 07b6ca684b0f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 549e24450d8d2e0273ed38c0d247c306a2972ba9 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/testReport/ |
| Max. process+thread count | 4882 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-client hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 Throttles should support limits based on handler thread usage time [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2125257856
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##########
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) {
return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond,
Review Comment:
I'm not very familiar with the quota implementation, so for every operation, we will create a OperationQuota object and calculate the estimate time, and then compare it with the actual time usage, and see whether we should stop it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "ajkh88 (via GitHub)" <gi...@apache.org>.
ajkh88 commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2121245360
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##########
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) {
return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond,
Review Comment:
In this context, `handler usage time` refers to the amount of time (in milliseconds) a handler thread will use.
We estimate the number of milliseconds we expect the thread to take to complete the request and deduct that amount from the quota. If there are enough milliseconds left in the quota, we proceed by taking that amount from the quota and continuing with the request. Otherwise, we throw a throttling exception.
When the request is complete, we calculate the actual amount of time used in the `close()` method and adjust the quota by either adding back time or further subtracting it, as appropriate.
Does this make things clearer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2095818581
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##########
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long estimateSize) {
return calculateReadCapacityUnit(actualSize) - calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double requestsPerSecond,
+ final int numHandlerThreads) {
+ if (requestsPerSecond <= numHandlerThreads) {
+ // If less than 1 request per second per handler thread, then we use the number of handler
+ // threads as a baseline to avoid incorrect estimations when the number of requests is very
+ // low.
+ return numHandlerThreads;
+ } else {
+ double requestsPerMillisecond = Math.ceil(requestsPerSecond / 1000);
Review Comment:
[nitpick] Consider adding documentation or comments to clarify the rationale for using Math.ceil(requestsPerSecond/1000) for estimating per-request handler usage time, ensuring it aligns with the intended behavior under low request rates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Posted by "rmdmattingly (via GitHub)" <gi...@apache.org>.
rmdmattingly commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-2891118070
lucky number 7000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29231 Throttles should support limits based on handler thread usage time [hbase]
Posted by "rmdmattingly (via GitHub)" <gi...@apache.org>.
rmdmattingly merged PR #7000:
URL: https://github.com/apache/hbase/pull/7000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org