You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "shangshu-qian (via GitHub)" <gi...@apache.org> on 2025/05/11 19:30:06 UTC
[PR] HBASE-29006. The region assignment retry logic in unconstrained and may cause workload amplification [hbase]
shangshu-qian opened a new pull request, #6983:
URL: https://github.com/apache/hbase/pull/6983
See [HBASE-29006](https://issues.apache.org/jira/browse/HBASE-29006)
We add a map to record the number of retried in the region assignment. If the retry count exceeds the predefined threshold (`MAX_RETRY_LIMIT`), the request will be blocked, preventing overloading the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29006. The region assignment retry logic in unconstrained and may cause workload amplification [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6983:
URL: https://github.com/apache/hbase/pull/6983#issuecomment-2870291141
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ branch-2.6 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 27s | | branch-2.6 passed |
| +1 :green_heart: | compile | 0m 51s | | branch-2.6 passed |
| +1 :green_heart: | javadoc | 0m 26s | | branch-2.6 passed |
| +1 :green_heart: | shadedjars | 6m 26s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 7s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 25s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 215m 21s | | hbase-server in the patch passed. |
| | | 243m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6983 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a86a4318e65f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.6 / cbc9126f8b9c009753146a1fb63dd2d070c13cfb |
| Default Java | Eclipse Adoptium-11.0.23+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/testReport/ |
| Max. process+thread count | 4361 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29006. The region assignment retry logic in unconstrained and may cause workload amplification [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6983:
URL: https://github.com/apache/hbase/pull/6983#issuecomment-2870296736
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 5s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ branch-2.6 Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 40s | | branch-2.6 passed |
| +1 :green_heart: | compile | 0m 42s | | branch-2.6 passed |
| +1 :green_heart: | javadoc | 0m 25s | | branch-2.6 passed |
| +1 :green_heart: | shadedjars | 5m 17s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 23s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 228m 54s | | hbase-server in the patch passed. |
| | | 252m 36s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6983 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 2d54f3610178 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.6 / cbc9126f8b9c009753146a1fb63dd2d070c13cfb |
| Default Java | Temurin-1.8.0_412-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/testReport/ |
| Max. process+thread count | 4235 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29006. The region assignment retry logic in unconstrained and may cause workload amplification [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6983:
URL: https://github.com/apache/hbase/pull/6983#issuecomment-2870181392
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 3m 10s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ branch-2.6 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 19s | | branch-2.6 passed |
| +1 :green_heart: | compile | 2m 59s | | branch-2.6 passed |
| +1 :green_heart: | checkstyle | 0m 38s | | branch-2.6 passed |
| +1 :green_heart: | spotbugs | 1m 33s | | branch-2.6 passed |
| +1 :green_heart: | spotless | 0m 46s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 5s | | the patch passed |
| +1 :green_heart: | compile | 2m 53s | | the patch passed |
| +1 :green_heart: | javac | 2m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 38s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 42s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 16m 58s | | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0. |
| -1 :x: | spotless | 0m 39s | | patch has 34 errors when running spotless:check, run spotless:apply to fix. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 40m 26s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6983 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux fef176c4734a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.6 / cbc9126f8b9c009753146a1fb63dd2d070c13cfb |
| Default Java | Eclipse Adoptium-11.0.23+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 77 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29006. The region assignment retry logic in unconstrained and may cause workload amplification [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6983:
URL: https://github.com/apache/hbase/pull/6983#discussion_r2086989879
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java:
##########
@@ -2483,7 +2487,21 @@ private void processAssignmentPlans(final HashMap<RegionInfo, RegionStateNode> r
acceptPlan(regions, balancer.retainAssignment(retainMap, servers));
} catch (HBaseIOException e) {
LOG.warn("unable to retain assignment", e);
- addToPendingAssignment(regions, retainMap.keySet());
+
+ for(RegionInfo hri : retainMap.keySet()) {
+ int retryCount = retryCountMap.getOrDefault(hri,0) + 1;
+ retryCountMap.put(hri, retryCount);
+
+ if(retryCount > MAX_RETRY_LIMIT) {
Review Comment:
Better add a backoff here? If we go with this PR, the region will be in RIT state forever and only restarting master can trigger the reassign...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29006. The region assignment retry logic in unconstrained and may cause workload amplification [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6983:
URL: https://github.com/apache/hbase/pull/6983#issuecomment-2870290134
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ branch-2.6 Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 16s | | branch-2.6 passed |
| +1 :green_heart: | compile | 1m 0s | | branch-2.6 passed |
| +1 :green_heart: | javadoc | 0m 28s | | branch-2.6 passed |
| +1 :green_heart: | shadedjars | 6m 14s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 9s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 35s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 214m 10s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 241m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6983 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 22be28260db7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.6 / cbc9126f8b9c009753146a1fb63dd2d070c13cfb |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/testReport/ |
| Max. process+thread count | 4507 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6983/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org