You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2024/11/13 11:31:50 UTC
[PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
virajjasani opened a new pull request, #6462:
URL: https://github.com/apache/hbase/pull/6462
Jira: HBASE-28638
Master initiated remote procedures are scheduled by RSProcedureDispatcher. If it encounters specific errors on first retry (e.g. CallQueueTooBigException or SaslException), it is guaranteed that the remote call has not reached the regionserver, therefore the remote call is marked failed prompting the parent procedure to select different target regionserver to resume the operation.
If the first attempt is successful, RSProcedureDispatcher continues with infinite retries. We can encounter valid case (e.g. ConnectionClosedException) which is halting the remote operation. Without manual intervention, it can cause significant delay upto several minutes or hours to the region-in-transition.
The purpose of this Jira is to impose retry limit for specific error types such that if the retry limit is reached, the master can recover the state of the ongoing remote call failure by initiating SCP (ServerCrashProcedure) on the target server. The SCP is going to override the TRSP (TransitRegionStateProcedure) if required. This can ensure that the target server has no region hosted online before we suspend the ongoing TRSP.
Scheduling SCP for the target server will always lead to the regionserver in stopped state. Either regionserver would be automatically stopped, or if the regionserver is able to send the region report to master, master will reject it, which will further lead to regionserver abort.
**Changes proposed:**
- Allow extending RSProcedureDispatcher
- RSProcedureDispatcher can impose retry limit for specific errors:
- CallQueueTooBigException
- SaslException
- ConnectionClosedException
- Default retry limit: 5
- If retry limit is exhausted, schedule recovery through server crash. Let SCP override current procedure state.
- Tests for ConnectionClosedException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2481057229
> Why do we need a new implementation of RSProcedureDispatcher?
It is done only for testing purpose. There is no new implementation as part of the source code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2473459872
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 9s | | master passed |
| +1 :green_heart: | compile | 3m 50s | | master passed |
| +1 :green_heart: | checkstyle | 0m 41s | | master passed |
| +1 :green_heart: | spotbugs | 1m 56s | | master passed |
| +1 :green_heart: | spotless | 0m 57s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 41s | | the patch passed |
| +1 :green_heart: | compile | 3m 45s | | the patch passed |
| +1 :green_heart: | javac | 3m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 47s | | the patch passed |
| +1 :green_heart: | spotbugs | 2m 16s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 13m 38s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 4s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | | The patch does not generate ASF License warnings. |
| | | 46m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux b9a355bfcdfd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c70332d9909c5f4df6f58d264c89ce527cd097ba |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1845340095
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:
##########
@@ -489,6 +491,15 @@ public class HMaster extends HBaseServerBase<MasterRpcServices> implements Maste
public static final String WARMUP_BEFORE_MOVE = "hbase.master.warmup.before.move";
private static final boolean DEFAULT_WARMUP_BEFORE_MOVE = true;
+ /**
+ * Use RSProcedureDispatcher instance to initiate master -> rs remote procedure execution. Use
+ * this config to provide customized RSProcedureDispatcher.
Review Comment:
With the current master's proc executor initialization, we are not able to provide a setter method for custom RSProcedureDispatcher without making significant changes.
Introducing config here to achieve the same is cleaner than setter method but the plan is only for test to use the custom implementation of RSProcedureDispatcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2552240851
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 44s | | master passed |
| +1 :green_heart: | compile | 2m 36s | | master passed |
| +1 :green_heart: | checkstyle | 0m 34s | | master passed |
| +1 :green_heart: | spotbugs | 1m 29s | | master passed |
| +1 :green_heart: | spotless | 0m 39s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 34s | | the patch passed |
| +1 :green_heart: | compile | 2m 32s | | the patch passed |
| +1 :green_heart: | javac | 2m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 32s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 9m 20s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 39s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 31m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux d5dc3e5987ef 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0628291c7459b0c72182bfeb9f1b8930f28154af |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/7/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1845339876
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:
##########
@@ -1833,7 +1844,11 @@ protected void stopServiceThreads() {
}
private void createProcedureExecutor() throws IOException {
- MasterProcedureEnv procEnv = new MasterProcedureEnv(this);
+ final String procedureDispatcherClassName =
+ conf.get(HBASE_MASTER_RSPROC_DISPATCHER_CLASS, DEFAULT_HBASE_MASTER_RSPROC_DISPATCHER_CLASS);
Review Comment:
This allows only extension of RSProcedureDispatcher, mainly to be used for testing purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Umeshkumar9414 (via GitHub)" <gi...@apache.org>.
Umeshkumar9414 commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1840761447
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:
##########
@@ -1833,7 +1844,11 @@ protected void stopServiceThreads() {
}
private void createProcedureExecutor() throws IOException {
- MasterProcedureEnv procEnv = new MasterProcedureEnv(this);
+ final String procedureDispatcherClassName =
+ conf.get(HBASE_MASTER_RSPROC_DISPATCHER_CLASS, DEFAULT_HBASE_MASTER_RSPROC_DISPATCHER_CLASS);
Review Comment:
Is this for extention of RSProcedureDispatcher or can there be a new implementation of RemoteProcedureDispatcher ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2479044372
Why do we need a new implementation of RSProcedureDispatcher?
I think here we can introduce a retry limit, by default it is -1, which means infinite retry, which keeps the old behavior, and if set to a positive value, we will schedule SCP when reaching the limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1896917709
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +404,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
Review Comment:
Sure, `isConnectionClosedError()` also follows the same pattern. Logically there is no difference but your suggestion will make it look cleaner so let's do it in separate Jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2529561237
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 35s | | master passed |
| +1 :green_heart: | compile | 0m 53s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 5m 38s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 39s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 20s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/5/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 43m 53s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 606bbca932d1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c644ba85c050431847768a9335212b4fceec20b8 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/5/testReport/ |
| Max. process+thread count | 2180 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/5/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1877056148
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,28 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+ ExecuteProceduresRequest executeProceduresRequest = request.build();
Review Comment:
Seems we do not need this change?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,28 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+ ExecuteProceduresRequest executeProceduresRequest = request.build();
+
+ // Check if the num of attempts have crossed the retry limit, and if the error type can
+ // fail-fast.
+ if (numberOfAttemptsSoFar >= failFastRetryLimit - 1 && isErrorTypeFailFast(e)) {
+ LOG
+ .warn("Number of retries {} exceeded limit {} for the given error type. Scheduling server"
+ + " crash for {}", numberOfAttemptsSoFar + 1, failFastRetryLimit, serverName, e);
+ // Expiring the server will schedule SCP and also reject the regionserver report from the
+ // regionserver if regionserver is somehow able to send the regionserver report to master.
+ // The master rejects the report by throwing YouAreDeadException, which would eventually
+ // result in the regionserver abort.
+ // This will also remove "serverName" from the ServerManager's onlineServers map.
+ master.getServerManager().expireServer(serverName);
+ }
// Always retry for other exception types if the region server is not dead yet.
if (!master.getServerManager().isServerOnline(serverName)) {
LOG.warn("Request to {} failed due to {}, try={} and the server is not online, give up",
serverName, e.toString(), numberOfAttemptsSoFar);
return false;
}
- if (e instanceof RegionServerAbortedException || e instanceof RegionServerStoppedException) {
+ if (e instanceof RegionServerStoppedException) {
Review Comment:
Why we remove the RegionServerAbortedException here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2474010657
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 26s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 59s | | master passed |
| +1 :green_heart: | compile | 0m 52s | | master passed |
| +1 :green_heart: | javadoc | 0m 26s | | master passed |
| +1 :green_heart: | shadedjars | 5m 37s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 36s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 230m 42s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 255m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 5c8093dec320 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c70332d9909c5f4df6f58d264c89ce527cd097ba |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/1/testReport/ |
| Max. process+thread count | 5022 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1845340095
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:
##########
@@ -489,6 +491,15 @@ public class HMaster extends HBaseServerBase<MasterRpcServices> implements Maste
public static final String WARMUP_BEFORE_MOVE = "hbase.master.warmup.before.move";
private static final boolean DEFAULT_WARMUP_BEFORE_MOVE = true;
+ /**
+ * Use RSProcedureDispatcher instance to initiate master -> rs remote procedure execution. Use
+ * this config to provide customized RSProcedureDispatcher.
Review Comment:
With the current master's proc executor initialization, its becoming very clumsy to provide a setter method for test to override using custom logic for RSProcedureDispatcher.
Introduced config here to achieve the same but the plan is only for test to use new extended implementation of RSProcedureDispatcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1877175828
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,28 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+ ExecuteProceduresRequest executeProceduresRequest = request.build();
+
+ // Check if the num of attempts have crossed the retry limit, and if the error type can
+ // fail-fast.
+ if (numberOfAttemptsSoFar >= failFastRetryLimit - 1 && isErrorTypeFailFast(e)) {
+ LOG
+ .warn("Number of retries {} exceeded limit {} for the given error type. Scheduling server"
+ + " crash for {}", numberOfAttemptsSoFar + 1, failFastRetryLimit, serverName, e);
+ // Expiring the server will schedule SCP and also reject the regionserver report from the
+ // regionserver if regionserver is somehow able to send the regionserver report to master.
+ // The master rejects the report by throwing YouAreDeadException, which would eventually
+ // result in the regionserver abort.
+ // This will also remove "serverName" from the ServerManager's onlineServers map.
+ master.getServerManager().expireServer(serverName);
+ }
// Always retry for other exception types if the region server is not dead yet.
if (!master.getServerManager().isServerOnline(serverName)) {
LOG.warn("Request to {} failed due to {}, try={} and the server is not online, give up",
serverName, e.toString(), numberOfAttemptsSoFar);
return false;
}
- if (e instanceof RegionServerAbortedException || e instanceof RegionServerStoppedException) {
+ if (e instanceof RegionServerStoppedException) {
Review Comment:
Because `RegionServerAbortedException extends RegionServerStoppedException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2530189662
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 52s | | master passed |
| +1 :green_heart: | compile | 3m 46s | | master passed |
| +1 :green_heart: | checkstyle | 0m 54s | | master passed |
| +1 :green_heart: | spotbugs | 1m 49s | | master passed |
| +1 :green_heart: | spotless | 0m 42s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 32s | | the patch passed |
| +1 :green_heart: | compile | 2m 52s | | the patch passed |
| +1 :green_heart: | javac | 2m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 39s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 10m 59s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 40s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 12s | | The patch does not generate ASF License warnings. |
| | | 38m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 1971d11ea77e 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 87b6585d895e27fe4604a9514fb4129361c9b144 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/6/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1845346649
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/util/RSProcDispatcher.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.AsyncRegionServerAdmin;
+import org.apache.hadoop.hbase.exceptions.ConnectionClosedException;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher;
+import org.apache.hadoop.hbase.regionserver.RegionServerStoppedException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos;
+
+/**
+ * Test implementation of RSProcedureDispatcher that throws desired errors for testing purpose.
+ */
+public class RSProcDispatcher extends RSProcedureDispatcher {
Review Comment:
This extension of RSProcedureDispatcher is under `hbase-server/src/test` package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1845479437
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +403,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
+ IOException unwrappedCause = unwrapException((IOException) cause);
+ if (unwrappedCause instanceof ConnectionClosedException) {
+ return true;
+ }
+ }
+ cause = cause.getCause();
+ if (cause == null) {
+ return false;
+ }
+ }
+ }
+
+ /**
+ * Returns true if the error is eligible for imposing retry limit.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error is eligible for imposing retry limit.
+ */
+ private boolean isErrorTypeEligibleForRetryLimit(IOException e) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2474541895
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 32s | | master passed |
| +1 :green_heart: | compile | 3m 8s | | master passed |
| +1 :green_heart: | checkstyle | 0m 36s | | master passed |
| +1 :green_heart: | spotbugs | 1m 40s | | master passed |
| +1 :green_heart: | spotless | 0m 47s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 43s | | the patch passed |
| +1 :green_heart: | compile | 3m 36s | | the patch passed |
| +1 :green_heart: | javac | 3m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 43s | | the patch passed |
| +1 :green_heart: | spotbugs | 2m 1s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 13m 27s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 57s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 13s | | The patch does not generate ASF License warnings. |
| | | 43m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux a0aa40543939 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a55b88f28f4db80eac07560afb60946d25c698ec |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/2/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2552475586
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 31s | | master passed |
| +1 :green_heart: | compile | 0m 56s | | master passed |
| +1 :green_heart: | javadoc | 0m 26s | | master passed |
| +1 :green_heart: | shadedjars | 5m 38s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 38s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 184m 25s | | hbase-server in the patch passed. |
| | | 209m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 8a782b85b501 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0628291c7459b0c72182bfeb9f1b8930f28154af |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/7/testReport/ |
| Max. process+thread count | 5029 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/7/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #6462:
URL: https://github.com/apache/hbase/pull/6462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2481267697
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 2m 46s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 56s | | master passed |
| +1 :green_heart: | compile | 0m 52s | | master passed |
| +1 :green_heart: | javadoc | 0m 26s | | master passed |
| +1 :green_heart: | shadedjars | 5m 39s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 37s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 211m 14s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/3/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 237m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 4863f8ad39a4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 6aa56e77566d24959e072dd8b452c6df3d27ebb3 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/3/testReport/ |
| Max. process+thread count | 5393 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/3/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2481392788
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 20s | | master passed |
| +1 :green_heart: | compile | 0m 52s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 5m 35s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 35s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 209m 25s | | hbase-server in the patch passed. |
| | | 233m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux e587864c4152 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 6aa56e77566d24959e072dd8b452c6df3d27ebb3 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/4/testReport/ |
| Max. process+thread count | 5628 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/4/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1840962664
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -249,6 +249,20 @@ protected class ExecuteProceduresRemoteCall implements RemoteProcedureResolver,
"hbase.regionserver.rpc.retry.interval";
private static final int DEFAULT_RS_RPC_RETRY_INTERVAL = 100;
+ /**
+ * Config to determine the retry limit while executing remote regionserver procedure. This retry
+ * limit applies to only specific errors. These errors could potentially get the remote
+ * procedure stuck for several minutes unless the retry limit is applied.
+ */
+ private static final String RS_REMOTE_PROC_RETRY_LIMIT =
+ "hbase.master.rs.remote.proc.retry.limit";
+ /**
+ * The default retry limit. Value = {@value}
+ */
+ private static final int DEFAULT_RS_REMOTE_PROC_RETRY_LIMIT = 5;
Review Comment:
Why 5? Why not 3?
Let's have a comment on justification for the default value.
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +403,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
+ IOException unwrappedCause = unwrapException((IOException) cause);
+ if (unwrappedCause instanceof ConnectionClosedException) {
+ return true;
+ }
+ }
+ cause = cause.getCause();
+ if (cause == null) {
+ return false;
+ }
+ }
+ }
+
+ /**
+ * Returns true if the error is eligible for imposing retry limit.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error is eligible for imposing retry limit.
+ */
+ private boolean isErrorTypeEligibleForRetryLimit(IOException e) {
Review Comment:
Instead of "eligible for retry limit" type wording, you should describe this as "fast fail" in method naming and comment text.
Ok, and then following this recommendation, the config `hbase.master.rs.remote.proc.retry.limit` might be better named `hbase.master.rs.remote.proc.fast.fail.limit`
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:
##########
@@ -489,6 +491,15 @@ public class HMaster extends HBaseServerBase<MasterRpcServices> implements Maste
public static final String WARMUP_BEFORE_MOVE = "hbase.master.warmup.before.move";
private static final boolean DEFAULT_WARMUP_BEFORE_MOVE = true;
+ /**
+ * Use RSProcedureDispatcher instance to initiate master -> rs remote procedure execution. Use
+ * this config to provide customized RSProcedureDispatcher.
Review Comment:
What are we planning here @virajjasani ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2475003578
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 2s | | master passed |
| +1 :green_heart: | compile | 0m 57s | | master passed |
| +1 :green_heart: | javadoc | 0m 31s | | master passed |
| +1 :green_heart: | shadedjars | 5m 27s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 59s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 18s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 233m 45s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 259m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 96ffd468c85d 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a55b88f28f4db80eac07560afb60946d25c698ec |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/2/testReport/ |
| Max. process+thread count | 5437 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1890409208
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,27 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+
+ // Check if the num of attempts have crossed the retry limit, and if the error type can
+ // fail-fast.
+ if (numberOfAttemptsSoFar >= failFastRetryLimit - 1 && isErrorTypeFailFast(e)) {
+ LOG
+ .warn("Number of retries {} exceeded limit {} for the given error type. Scheduling server"
+ + " crash for {}", numberOfAttemptsSoFar + 1, failFastRetryLimit, serverName, e);
+ // Expiring the server will schedule SCP and also reject the regionserver report from the
+ // regionserver if regionserver is somehow able to send the regionserver report to master.
+ // The master rejects the report by throwing YouAreDeadException, which would eventually
+ // result in the regionserver abort.
+ // This will also remove "serverName" from the ServerManager's onlineServers map.
+ master.getServerManager().expireServer(serverName);
Review Comment:
We don't need to return here because the next if condition will anyways take care of returning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1890212678
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,27 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+
+ // Check if the num of attempts have crossed the retry limit, and if the error type can
+ // fail-fast.
+ if (numberOfAttemptsSoFar >= failFastRetryLimit - 1 && isErrorTypeFailFast(e)) {
+ LOG
+ .warn("Number of retries {} exceeded limit {} for the given error type. Scheduling server"
+ + " crash for {}", numberOfAttemptsSoFar + 1, failFastRetryLimit, serverName, e);
+ // Expiring the server will schedule SCP and also reject the regionserver report from the
+ // regionserver if regionserver is somehow able to send the regionserver report to master.
+ // The master rejects the report by throwing YouAreDeadException, which would eventually
+ // result in the regionserver abort.
+ // This will also remove "serverName" from the ServerManager's onlineServers map.
+ master.getServerManager().expireServer(serverName);
Review Comment:
We do not return here? Just fall through to execute later logic?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +404,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
Review Comment:
I think we can only unwrap RemoteException, so here let's just test RemoteException directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1894924647
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,27 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+
+ // Check if the num of attempts have crossed the retry limit, and if the error type can
+ // fail-fast.
+ if (numberOfAttemptsSoFar >= failFastRetryLimit - 1 && isErrorTypeFailFast(e)) {
+ LOG
+ .warn("Number of retries {} exceeded limit {} for the given error type. Scheduling server"
+ + " crash for {}", numberOfAttemptsSoFar + 1, failFastRetryLimit, serverName, e);
+ // Expiring the server will schedule SCP and also reject the regionserver report from the
+ // regionserver if regionserver is somehow able to send the regionserver report to master.
+ // The master rejects the report by throwing YouAreDeadException, which would eventually
+ // result in the regionserver abort.
+ // This will also remove "serverName" from the ServerManager's onlineServers map.
+ master.getServerManager().expireServer(serverName);
Review Comment:
But it will log another line to say that because server is not online, which is confusing... We should always return earlier if possible.
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +404,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
Review Comment:
OK, I saw you have already used this style in isSaslError, I should stop it at the first place...
Let's file new issues to polish it. UnwrapException is not designed to be used here, it just want to put the instanceof test inside the method so we do not need to write it everywhere but here we already have the instanceof, so calling this method again does not make sense...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2481280293
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 2s | | master passed |
| +1 :green_heart: | compile | 2m 58s | | master passed |
| +1 :green_heart: | checkstyle | 0m 38s | | master passed |
| +1 :green_heart: | spotbugs | 1m 36s | | master passed |
| +1 :green_heart: | spotless | 0m 44s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 2m 55s | | the patch passed |
| +1 :green_heart: | javac | 2m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 38s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 10m 19s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 9s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 12s | | The patch does not generate ASF License warnings. |
| | | 35m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux d26ee9b19214 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 6aa56e77566d24959e072dd8b452c6df3d27ebb3 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/4/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2529502447
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 46s | | master passed |
| +1 :green_heart: | compile | 2m 35s | | master passed |
| +1 :green_heart: | checkstyle | 0m 34s | | master passed |
| +1 :green_heart: | spotbugs | 1m 33s | | master passed |
| +1 :green_heart: | spotless | 0m 42s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 37s | | the patch passed |
| +1 :green_heart: | compile | 2m 34s | | the patch passed |
| +1 :green_heart: | javac | 2m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 9m 27s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 40s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 32m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux c3c592a75746 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c644ba85c050431847768a9335212b4fceec20b8 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/5/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1891159167
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +404,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
Review Comment:
Is this fine @Apache9? I can change it if you have strong opinion on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1896917000
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -300,13 +318,27 @@ private boolean scheduleForRetry(IOException e) {
if (numberOfAttemptsSoFar == 0 && unableToConnectToServer(e)) {
return false;
}
+
+ // Check if the num of attempts have crossed the retry limit, and if the error type can
+ // fail-fast.
+ if (numberOfAttemptsSoFar >= failFastRetryLimit - 1 && isErrorTypeFailFast(e)) {
+ LOG
+ .warn("Number of retries {} exceeded limit {} for the given error type. Scheduling server"
+ + " crash for {}", numberOfAttemptsSoFar + 1, failFastRetryLimit, serverName, e);
+ // Expiring the server will schedule SCP and also reject the regionserver report from the
+ // regionserver if regionserver is somehow able to send the regionserver report to master.
+ // The master rejects the report by throwing YouAreDeadException, which would eventually
+ // result in the regionserver abort.
+ // This will also remove "serverName" from the ServerManager's onlineServers map.
+ master.getServerManager().expireServer(serverName);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2561470797
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 26s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 4s | | master passed |
| +1 :green_heart: | compile | 0m 58s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 5m 40s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 37s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 179m 33s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/8/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 204m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a683c22e0d9d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f8fd6b9c956017afc053381900574ba9fdaef3d0 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/8/testReport/ |
| Max. process+thread count | 5137 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/8/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1890687134
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -371,6 +404,39 @@ private boolean isSaslError(IOException e) {
}
}
+ /**
+ * Returns true if the error or its cause is of type ConnectionClosedException.
+ * @param e IOException thrown by the underlying rpc framework.
+ * @return True if the error or its cause is of type ConnectionClosedException.
+ */
+ private boolean isConnectionClosedError(IOException e) {
+ if (e instanceof ConnectionClosedException) {
+ return true;
+ }
+ Throwable cause = e;
+ while (true) {
+ if (cause instanceof IOException) {
Review Comment:
RemoteException is of type IOE and I kept this to be consistent with other exception check logic we have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2481133104
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 3m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 52s | | master passed |
| +1 :green_heart: | compile | 2m 56s | | master passed |
| +1 :green_heart: | checkstyle | 0m 38s | | master passed |
| +1 :green_heart: | spotbugs | 1m 33s | | master passed |
| +1 :green_heart: | spotless | 0m 46s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 2m 53s | | the patch passed |
| +1 :green_heart: | javac | 2m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 39s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 10m 27s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 43s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 38m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 44cda4e55ff2 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 6aa56e77566d24959e072dd8b452c6df3d27ebb3 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/3/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Impose retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6462:
URL: https://github.com/apache/hbase/pull/6462#discussion_r1845347100
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java:
##########
@@ -249,6 +249,20 @@ protected class ExecuteProceduresRemoteCall implements RemoteProcedureResolver,
"hbase.regionserver.rpc.retry.interval";
private static final int DEFAULT_RS_RPC_RETRY_INTERVAL = 100;
+ /**
+ * Config to determine the retry limit while executing remote regionserver procedure. This retry
+ * limit applies to only specific errors. These errors could potentially get the remote
+ * procedure stuck for several minutes unless the retry limit is applied.
+ */
+ private static final String RS_REMOTE_PROC_RETRY_LIMIT =
+ "hbase.master.rs.remote.proc.retry.limit";
+ /**
+ * The default retry limit. Value = {@value}
+ */
+ private static final int DEFAULT_RS_REMOTE_PROC_RETRY_LIMIT = 5;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2530193984
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 25s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 47s | | master passed |
| +1 :green_heart: | compile | 0m 53s | | master passed |
| +1 :green_heart: | javadoc | 0m 26s | | master passed |
| +1 :green_heart: | shadedjars | 5m 37s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 40s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 16s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/6/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 42m 51s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 910e25b0059c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 87b6585d895e27fe4604a9514fb4129361c9b144 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/6/testReport/ |
| Max. process+thread count | 2165 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/6/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28638 Fail-fast retry limit for specific errors to recover from remote procedure failure using server crash [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6462:
URL: https://github.com/apache/hbase/pull/6462#issuecomment-2561360221
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 2s | | master passed |
| +1 :green_heart: | compile | 2m 40s | | master passed |
| +1 :green_heart: | checkstyle | 0m 36s | | master passed |
| +1 :green_heart: | spotbugs | 1m 34s | | master passed |
| +1 :green_heart: | spotless | 0m 39s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 52s | | the patch passed |
| +1 :green_heart: | compile | 3m 5s | | the patch passed |
| +1 :green_heart: | javac | 3m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 34s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 9m 44s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 39s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 33m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6462 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 6971b5a7933a 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f8fd6b9c956017afc053381900574ba9fdaef3d0 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6462/8/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org