You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2025/04/16 05:44:11 UTC
[PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
virajjasani opened a new pull request, #6910:
URL: https://github.com/apache/hbase/pull/6910
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2816324236
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 18s | | master passed |
| +1 :green_heart: | compile | 3m 7s | | master passed |
| +1 :green_heart: | checkstyle | 0m 38s | | master passed |
| +1 :green_heart: | spotbugs | 1m 37s | | master passed |
| +1 :green_heart: | spotless | 0m 47s | | branch has no errors when running spotless:check. |
| -0 :warning: | patch | 0m 54s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 0s | | the patch passed |
| +1 :green_heart: | compile | 3m 7s | | the patch passed |
| +1 :green_heart: | javac | 3m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 39s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 52s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 43s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 38m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux b51abeb512a2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f4920793e66d5d58909ed4cb3ff9438d65502fd7 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/5/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2049819375
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
You need to define what you want to test, if you just want to make sure that if there is an exception you will call abort, it is very easy, and even do not need to bring up a cluster.
If you want to do something like a integration tests, you can extend HRegion, and there are bunch of ways to decide whether to throw an exception in batchMutate method. You can make the specific HRegion implementation an inner class of the testcase, and set a static field in the test class to control whether to throw an exception...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2051927386
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
Review Comment:
You mean for `tries == (maxRetriesForRegionUpdates - 1)` condition? Otherwise, it will do it anyways for all retries as per the above loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2813866851
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 19s | | master passed |
| +1 :green_heart: | compile | 3m 3s | | master passed |
| +1 :green_heart: | checkstyle | 0m 38s | | master passed |
| +1 :green_heart: | spotbugs | 1m 34s | | master passed |
| +1 :green_heart: | spotless | 0m 45s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 0s | | the patch passed |
| +1 :green_heart: | compile | 3m 3s | | the patch passed |
| +1 :green_heart: | javac | 3m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 38s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 52s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 45s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 38m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 621b2a36f17c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e77927a3323c03361ad23467b66a6a2ff93db857 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/2/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2049822824
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -147,9 +147,11 @@ public void update(UpdateMasterRegion action) throws IOException {
try {
action.update(region);
flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
+ } catch (IOException e) {
Review Comment:
Better add some comments here, to summary the discussion on the jira, and also give a pointer to the jira, to let later developpers know why here we will abort for any IOExceptions.
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMasterRegionMutation.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.SingleProcessHBaseCluster;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.master.hbck.HbckChore;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.region.MasterRegionFactory;
+import org.apache.hadoop.hbase.regionserver.HRegion;
+import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.OperationStatus;
+import org.apache.hadoop.hbase.regionserver.RegionServerServices;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos;
+
+/**
+ * MasterRegion related test that ensures the operations continue even when Procedure state update
+ * encounters IO errors.
+ */
+@Category({ MasterTests.class, LargeTests.class })
+public class TestMasterRegionMutation {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestMasterRegionMutation.class);
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestMasterRegionMutation.class);
+
+ @Rule
+ public TestName name = new TestName();
+
+ private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
+ private static ServerName rs0;
+
+ private static final AtomicBoolean ERROR_OUT = new AtomicBoolean(false);
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+ TEST_UTIL.getConfiguration().setClass(HConstants.REGION_IMPL, TestRegion.class, HRegion.class);
+ StartTestingClusterOption.Builder builder = StartTestingClusterOption.builder();
+ builder.numMasters(4).numRegionServers(3);
Review Comment:
We need 4 masters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2819964122
Thanks everyone for the reviews! Awaiting final build results before merging the PR, the jenkins build is still stuck scheduling the build to a VM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2046117878
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
Not a good way to test his, but since MasterRegion is final class, extending it is also not possible.
Though we can reproduce the exact issue with the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2048777439
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
And inside MasterRegion we just use HRegion, for HRegion, we have a way to inject specify implementation class. Please see HRegion.newHRegion
```
public static HRegion newHRegion(Path tableDir, WAL wal, FileSystem fs, Configuration conf,
RegionInfo regionInfo, final TableDescriptor htd, RegionServerServices rsServices) {
try {
@SuppressWarnings("unchecked")
Class<? extends HRegion> regionClass =
(Class<? extends HRegion>) conf.getClass(HConstants.REGION_IMPL, HRegion.class);
Constructor<? extends HRegion> c =
regionClass.getConstructor(Path.class, WAL.class, FileSystem.class, Configuration.class,
RegionInfo.class, TableDescriptor.class, RegionServerServices.class);
return c.newInstance(tableDir, wal, fs, conf, regionInfo, htd, rsServices);
} catch (Throwable e) {
// todo: what should I throw here?
throw new IllegalStateException("Could not instantiate a region instance.", e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #6910:
URL: https://github.com/apache/hbase/pull/6910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2815063439
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 6s | | master passed |
| +1 :green_heart: | compile | 1m 22s | | master passed |
| +1 :green_heart: | javadoc | 0m 35s | | master passed |
| +1 :green_heart: | shadedjars | 6m 50s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 20s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 416m 16s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/3/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 460m 39s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a738b7e6208d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c510142548a7bd8d165167ba5a88e6bffbb6d80a |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/3/testReport/ |
| Max. process+thread count | 4760 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/3/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2050008991
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMasterRegionMutation.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.SingleProcessHBaseCluster;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.master.hbck.HbckChore;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.region.MasterRegionFactory;
+import org.apache.hadoop.hbase.regionserver.HRegion;
+import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.OperationStatus;
+import org.apache.hadoop.hbase.regionserver.RegionServerServices;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos;
+
+/**
+ * MasterRegion related test that ensures the operations continue even when Procedure state update
+ * encounters IO errors.
+ */
+@Category({ MasterTests.class, LargeTests.class })
+public class TestMasterRegionMutation {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestMasterRegionMutation.class);
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestMasterRegionMutation.class);
+
+ @Rule
+ public TestName name = new TestName();
+
+ private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
+ private static ServerName rs0;
+
+ private static final AtomicBoolean ERROR_OUT = new AtomicBoolean(false);
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+ TEST_UTIL.getConfiguration().setClass(HConstants.REGION_IMPL, TestRegion.class, HRegion.class);
+ StartTestingClusterOption.Builder builder = StartTestingClusterOption.builder();
+ builder.numMasters(4).numRegionServers(3);
Review Comment:
3 would also work but i just kept one additional. I can keep it 3 if you are not fine with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Reidddddd (via GitHub)" <gi...@apache.org>.
Reidddddd commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2051997750
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
Review Comment:
I mean like this:
```
} catch (RegionTooBusyException e) {
flusherAndCompactor.onUpdate();
if (tries == (maxRetriesForRegionUpdates - 1)) {
***
}
}
```
The time interval hasn't been reached, but the `changesAfterLastFlush` threshold has been met, could happen, so when this exception caught. you need to trigger `flusherAndCompactor.onUpdate();`
Or otherwise, you need to switch the execution order
```
flusherAndCompactor.onUpdate(); // first
action.update(region); // after
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2047399476
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
I think it should be fine to remove `final` from MasterRegion class so that we can extend it for testing purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2049495727
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
Thanks @Apache9 @apurtell! Updated the test, now it's clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2815292952
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 38m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 48s | | master passed |
| +1 :green_heart: | compile | 4m 17s | | master passed |
| +1 :green_heart: | checkstyle | 0m 38s | | master passed |
| +1 :green_heart: | spotbugs | 1m 37s | | master passed |
| -1 :x: | spotless | 1m 3s | | branch has 1 errors when running spotless:check, run spotless:apply to fix. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 14s | | the patch passed |
| +1 :green_heart: | compile | 3m 10s | | the patch passed |
| +1 :green_heart: | javac | 3m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 20m 23s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 45s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 97m 44s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux ed4e926fe283 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c9dd767c05d1cae12d413bee1127119575d6e016 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/artifact/yetus-general-check/output/branch-spotless.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2817296241
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 42s | | master passed |
| +1 :green_heart: | compile | 3m 24s | | master passed |
| +1 :green_heart: | checkstyle | 0m 41s | | master passed |
| +1 :green_heart: | spotbugs | 1m 45s | | master passed |
| +1 :green_heart: | spotless | 0m 52s | | branch has no errors when running spotless:check. |
| -0 :warning: | patch | 0m 59s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 21s | | the patch passed |
| +1 :green_heart: | compile | 3m 8s | | the patch passed |
| +1 :green_heart: | javac | 3m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 38s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 46s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 57s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 46s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 40m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 647a0122cf6b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 7e66d31603aced161e0a4643c127924a75ebf837 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/6/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Reidddddd (via GitHub)" <gi...@apache.org>.
Reidddddd commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2051924226
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
Review Comment:
If RegionTooBusyException is caught, here can trigger `flusherAndCompactor.onUpdate();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2047755907
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
Let's do that and then make this test more clean.
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
action.update(region);
flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
+ } catch (IOException e) {
+ LOG.error(HBaseMarkers.FATAL, "MasterRegion mutation is not successful. Aborting server.");
+ server.abort("MasterRegion mutation is not successful", e);
Review Comment:
Aborting is a start.
The rest of my question here is addressed by the discussion on the JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2050011513
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -147,9 +147,11 @@ public void update(UpdateMasterRegion action) throws IOException {
try {
action.update(region);
flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
+ } catch (IOException e) {
Review Comment:
Done, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2816046172
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 59s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 34s | | master passed |
| +1 :green_heart: | compile | 3m 56s | | master passed |
| +1 :green_heart: | javadoc | 0m 58s | | master passed |
| +1 :green_heart: | shadedjars | 8m 28s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 59s | | the patch passed |
| +1 :green_heart: | javac | 0m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 30s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 470m 20s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 543m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux abd7a1693a8a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c9dd767c05d1cae12d413bee1127119575d6e016 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/testReport/ |
| Max. process+thread count | 4978 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/4/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Reidddddd (via GitHub)" <gi...@apache.org>.
Reidddddd commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2051923809
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
+ if (tries == (maxRetriesForRegionUpdates - 1)) {
Review Comment:
If `RegionTooBusyException` is caught, here can trigger `flusherAndCompactor.onUpdate();`.
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
+ if (tries == (maxRetriesForRegionUpdates - 1)) {
Review Comment:
If `RegionTooBusyException` is caught, here can trigger `flusherAndCompactor.onUpdate();`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2821370907
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 19s | | master passed |
| +1 :green_heart: | compile | 0m 56s | | master passed |
| +1 :green_heart: | javadoc | 0m 28s | | master passed |
| +1 :green_heart: | shadedjars | 5m 55s | | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | patch | 6m 3s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 52s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 211m 28s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/8/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 237m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a54f91b8d3a0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 47edd9ace2b40679b030cf884fb086c8fa44ca65 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/8/testReport/ |
| Max. process+thread count | 4947 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/8/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2049260141
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
The problem with HRegion is that, we don't want to have custom HRegion that doesn't work for all cases. We want to inject error for batchMutate() only when region moves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2049260141
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
The problem with HRegion is that, we don't want to have custom HRegion that doesn't work for all cases. We want to inject error for batchMutate() only when region moves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2046117878
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
Not a good way to test his, but since MasterRegion is final class, extending it is also not possible.
Using this, we can reproduce the exact issue with the test if we don't abort master with IOE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2048777439
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
And in side MasterRegion we just uses HRegion, for HRegion, we have a way to inject specify implementation class. Please see HRegion.newHRegion
```
public static HRegion newHRegion(Path tableDir, WAL wal, FileSystem fs, Configuration conf,
RegionInfo regionInfo, final TableDescriptor htd, RegionServerServices rsServices) {
try {
@SuppressWarnings("unchecked")
Class<? extends HRegion> regionClass =
(Class<? extends HRegion>) conf.getClass(HConstants.REGION_IMPL, HRegion.class);
Constructor<? extends HRegion> c =
regionClass.getConstructor(Path.class, WAL.class, FileSystem.class, Configuration.class,
RegionInfo.class, TableDescriptor.class, RegionServerServices.class);
return c.newInstance(tableDir, wal, fs, conf, regionInfo, htd, rsServices);
} catch (Throwable e) {
// todo: what should I throw here?
throw new IllegalStateException("Could not instantiate a region instance.", e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2048774787
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
MasterRegion has a private constructor, that's why we mark it as final.
Since UpdateMasterRegion is just a interface, I think it is very easy to verify the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2052953947
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
Review Comment:
Makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2817391422
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 14s | | master passed |
| +1 :green_heart: | compile | 0m 55s | | master passed |
| +1 :green_heart: | javadoc | 0m 28s | | master passed |
| +1 :green_heart: | shadedjars | 5m 53s | | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | patch | 6m 1s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 4s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 50s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 295m 2s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/6/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 321m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 5fe24167c835 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 7e66d31603aced161e0a4643c127924a75ebf837 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/6/testReport/ |
| Max. process+thread count | 4630 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/6/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Reidddddd (via GitHub)" <gi...@apache.org>.
Reidddddd commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2817590211
Will you add back-off feature in next ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2820256062
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 53s | | master passed |
| +1 :green_heart: | compile | 4m 5s | | master passed |
| +1 :green_heart: | checkstyle | 0m 51s | | master passed |
| +1 :green_heart: | spotbugs | 2m 1s | | master passed |
| +1 :green_heart: | spotless | 0m 59s | | branch has no errors when running spotless:check. |
| -0 :warning: | patch | 1m 7s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 7s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 45s | | the patch passed |
| +1 :green_heart: | spotbugs | 2m 15s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 21m 34s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| -1 :x: | spotless | 2m 11s | | patch has 1 errors when running spotless:check, run spotless:apply to fix. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not generate ASF License warnings. |
| | | 64m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux ba9ddefe2416 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 47edd9ace2b40679b030cf884fb086c8fa44ca65 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/7/artifact/yetus-general-check/output/patch-spotless.txt |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/7/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2820881127
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 40s | | master passed |
| +1 :green_heart: | compile | 3m 22s | | master passed |
| +1 :green_heart: | checkstyle | 0m 41s | | master passed |
| +1 :green_heart: | spotbugs | 1m 47s | | master passed |
| +1 :green_heart: | spotless | 0m 52s | | branch has no errors when running spotless:check. |
| -0 :warning: | patch | 0m 59s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 14s | | the patch passed |
| +1 :green_heart: | javac | 3m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 53s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 14m 33s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 56s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 44m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 0ad8e13010a8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 47edd9ace2b40679b030cf884fb086c8fa44ca65 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/8/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2816324670
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 17s | | master passed |
| +1 :green_heart: | compile | 1m 1s | | master passed |
| +1 :green_heart: | javadoc | 0m 29s | | master passed |
| +1 :green_heart: | shadedjars | 7m 41s | | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | patch | 7m 54s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 3s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 7s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 12m 43s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/5/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 39m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux bfc80ac03e17 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f4920793e66d5d58909ed4cb3ff9438d65502fd7 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/5/testReport/ |
| Max. process+thread count | 1834 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/5/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2049262132
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,13 +148,23 @@ private void shutdownWAL() {
}
}
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ public void setTestFailure() {
+ this.updateFailForTest = true;
+ }
+
public void update(UpdateMasterRegion action) throws IOException {
try {
+ if (updateFailForTest) {
+ // test for HBASE-29251
+ throw new IOException("Update failed");
+ }
Review Comment:
Any custom implementation of UpdateMasterRegion also needs custom hooks in Procedure executor classes, which make it more complicated to test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2814373261
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 29s | | master passed |
| +1 :green_heart: | compile | 3m 10s | | master passed |
| +1 :green_heart: | checkstyle | 0m 39s | | master passed |
| +1 :green_heart: | spotbugs | 2m 18s | | master passed |
| +1 :green_heart: | spotless | 1m 1s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 49s | | the patch passed |
| +1 :green_heart: | compile | 3m 27s | | the patch passed |
| +1 :green_heart: | javac | 3m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | spotbugs | 2m 13s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 14m 21s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 18s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 49m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux fad89586b5e1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c510142548a7bd8d165167ba5a88e6bffbb6d80a |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/3/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2814177863
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 18s | | master passed |
| +1 :green_heart: | compile | 0m 57s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 5m 54s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 2s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 52s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 213m 50s | | hbase-server in the patch passed. |
| | | 240m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux c5f0531b2fde 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e77927a3323c03361ad23467b66a6a2ff93db857 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/2/testReport/ |
| Max. process+thread count | 4996 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2050008991
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMasterRegionMutation.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.util;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.SingleProcessHBaseCluster;
+import org.apache.hadoop.hbase.StartTestingClusterOption;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.master.hbck.HbckChore;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.region.MasterRegionFactory;
+import org.apache.hadoop.hbase.regionserver.HRegion;
+import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.OperationStatus;
+import org.apache.hadoop.hbase.regionserver.RegionServerServices;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.wal.WAL;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos;
+
+/**
+ * MasterRegion related test that ensures the operations continue even when Procedure state update
+ * encounters IO errors.
+ */
+@Category({ MasterTests.class, LargeTests.class })
+public class TestMasterRegionMutation {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestMasterRegionMutation.class);
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestMasterRegionMutation.class);
+
+ @Rule
+ public TestName name = new TestName();
+
+ private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
+ private static ServerName rs0;
+
+ private static final AtomicBoolean ERROR_OUT = new AtomicBoolean(false);
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+ TEST_UTIL.getConfiguration().setClass(HConstants.REGION_IMPL, TestRegion.class, HRegion.class);
+ StartTestingClusterOption.Builder builder = StartTestingClusterOption.builder();
+ builder.numMasters(4).numRegionServers(3);
Review Comment:
3 would also work (2 are aborted) but i just kept one additional. I can keep it 3 if you are not fine with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6910:
URL: https://github.com/apache/hbase/pull/6910#discussion_r2051930476
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java:
##########
@@ -143,17 +154,54 @@ private void shutdownWAL() {
}
}
+ /**
+ * Performs the mutation to the master region using UpdateMasterRegion update action.
+ * @param action Update region action.
+ * @throws IOException IO error that causes active master to abort.
+ */
public void update(UpdateMasterRegion action) throws IOException {
- try {
- action.update(region);
- flusherAndCompactor.onUpdate();
- } catch (WALSyncTimeoutIOException e) {
- LOG.error(HBaseMarkers.FATAL, "WAL sync timeout. Aborting server.");
- server.abort("WAL sync timeout", e);
- throw e;
+ for (int tries = 0; tries < maxRetriesForRegionUpdates; tries++) {
+ try {
+ // If the update is successful, return immediately.
+ action.update(region);
+ flusherAndCompactor.onUpdate();
+ return;
+ } catch (RegionTooBusyException e) {
+ // RegionTooBusyException is the type of IOException for which we can retry
+ // for few times before aborting the active master. The master region might
+ // have genuine case for delayed flushes and/or some procedure bug causing
+ // heavy pressure on the memstore.
+ if (tries == (maxRetriesForRegionUpdates - 1)) {
+ abortServer(e);
+ }
+ LOG.info("Master region {} is too busy... retry attempt: {}", region, tries);
+ Threads.sleep(ConnectionUtils.getPauseTime(regionUpdateRetryPauseTime, tries));
Review Comment:
@Reidddddd exponential backoff is added here. I think I should add comment here, because single line is not readable enough, let me do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2808519187
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 22s | | master passed |
| +1 :green_heart: | compile | 3m 4s | | master passed |
| +1 :green_heart: | checkstyle | 0m 36s | | master passed |
| +1 :green_heart: | spotbugs | 1m 37s | | master passed |
| +1 :green_heart: | spotless | 0m 47s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 56s | | the patch passed |
| +1 :green_heart: | compile | 3m 4s | | the patch passed |
| +1 :green_heart: | javac | 3m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 39s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 54s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 38m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 02f493be32f8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / da1200662540b9cdd7a2a5552b338b0e976bf940 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29251 Procedure gets stuck if the procedure state cannot be persisted [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6910:
URL: https://github.com/apache/hbase/pull/6910#issuecomment-2809034837
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 44s | | master passed |
| +1 :green_heart: | compile | 1m 18s | | master passed |
| +1 :green_heart: | javadoc | 0m 29s | | master passed |
| +1 :green_heart: | shadedjars | 6m 5s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 6s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 51s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 212m 26s | | hbase-server in the patch passed. |
| | | 240m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6910 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 9ceaf9bdf235 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / da1200662540b9cdd7a2a5552b338b0e976bf940 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/1/testReport/ |
| Max. process+thread count | 4724 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6910/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org