You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/09/07 03:59:36 UTC
[GitHub] [hbase] Apache9 opened a new pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Apache9 opened a new pull request #2354:
URL: https://github.com/apache/hbase/pull/2354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688016524
Because we reference the related methods in splitRegion and mergeRegions methods of MetaTableAccessor, also moved these two methods to RegionStateStore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688029963
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 4m 1s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 38s | master passed |
| +1 :green_heart: | checkstyle | 1m 46s | master passed |
| +1 :green_heart: | spotbugs | 3m 22s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 22s | the patch passed |
| -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 8 new + 113 unchanged - 4 fixed = 121 total (was 117) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 3m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
| | | 42m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux cacc0a737e42 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#discussion_r484240582
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/GCMergedRegionsProcedure.java
##########
@@ -88,7 +87,7 @@ protected Flow executeFromState(MasterProcedureEnv env, GCMergedRegionsState sta
setNextState(GCMergedRegionsState.GC_REGION_EDIT_METADATA);
break;
case GC_REGION_EDIT_METADATA:
- MetaTableAccessor.deleteMergeQualifiers(env.getMasterServices().getConnection(), mergedChild);
+ env.getAssignmentManager().getRegionStateStore().deleteMergeQualifiers(mergedChild);
Review comment:
Is it possible NPE in this long call chain?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688159897
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 28s | master passed |
| +1 :green_heart: | checkstyle | 1m 42s | master passed |
| +1 :green_heart: | spotbugs | 3m 27s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 45s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 28s | The patch passed checkstyle in hbase-client |
| +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-balancer |
| +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 99 unchanged - 18 fixed = 99 total (was 117) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 4m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
| | | 42m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 46f936793662 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688102870
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 27s | master passed |
| +1 :green_heart: | checkstyle | 1m 42s | master passed |
| +1 :green_heart: | spotbugs | 3m 26s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 21s | the patch passed |
| -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 111 unchanged - 6 fixed = 112 total (was 117) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 4m 7s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 38m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux e3829f9e5dd7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688074181
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 1m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 10s | master passed |
| +1 :green_heart: | compile | 1m 53s | master passed |
| +1 :green_heart: | shadedjars | 6m 37s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 19s | hbase-balancer in master failed. |
| -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. |
| -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 57s | the patch passed |
| +1 :green_heart: | compile | 1m 51s | the patch passed |
| +1 :green_heart: | javac | 1m 51s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. |
| -0 :warning: | javadoc | 0m 17s | hbase-balancer in the patch failed. |
| -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 0m 23s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 131m 28s | hbase-server in the patch passed. |
| | | 166m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 368d5ce2633f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-balancer.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-balancer.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/testReport/ |
| Max. process+thread count | 4852 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache9 merged pull request #2354:
URL: https://github.com/apache/hbase/pull/2354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#discussion_r484257892
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationBarrierFamilyFormat.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.RegionState.State;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Helper class for storing replication barriers in family 'rep_barrier' of meta table.
+ * <p/>
+ * See SerialReplicationChecker on how to make use of the barriers.
+ */
+@InterfaceAudience.Private
+public final class ReplicationBarrierFamilyFormat {
+
+ @VisibleForTesting
+ public static final byte[] REPLICATION_PARENT_QUALIFIER = Bytes.toBytes("parent");
+
+ private static final byte ESCAPE_BYTE = (byte) 0xFF;
+
+ private static final byte SEPARATED_BYTE = 0x00;
+
+ private ReplicationBarrierFamilyFormat() {
+ }
+
+ public static void addReplicationBarrier(Put put, long openSeqNum) throws IOException {
+ put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
+ .setFamily(HConstants.REPLICATION_BARRIER_FAMILY).setQualifier(HConstants.SEQNUM_QUALIFIER)
+ .setTimestamp(put.getTimestamp()).setType(Type.Put).setValue(Bytes.toBytes(openSeqNum))
+ .build());
+ }
+
+ private static void writeRegionName(ByteArrayOutputStream out, byte[] regionName) {
+ for (byte b : regionName) {
+ if (b == ESCAPE_BYTE) {
+ out.write(ESCAPE_BYTE);
+ }
+ out.write(b);
+ }
+ }
+
+ @VisibleForTesting
+ public static byte[] getParentsBytes(List<RegionInfo> parents) {
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ Iterator<RegionInfo> iter = parents.iterator();
+ writeRegionName(bos, iter.next().getRegionName());
+ while (iter.hasNext()) {
+ bos.write(ESCAPE_BYTE);
+ bos.write(SEPARATED_BYTE);
+ writeRegionName(bos, iter.next().getRegionName());
+ }
+ return bos.toByteArray();
+ }
+
+ private static List<byte[]> parseParentsBytes(byte[] bytes) {
+ List<byte[]> parents = new ArrayList<>();
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ for (int i = 0; i < bytes.length; i++) {
+ if (bytes[i] == ESCAPE_BYTE) {
+ i++;
+ if (bytes[i] == SEPARATED_BYTE) {
+ parents.add(bos.toByteArray());
+ bos.reset();
+ continue;
+ }
+ // fall through to append the byte
+ }
+ bos.write(bytes[i]);
+ }
+ if (bos.size() > 0) {
+ parents.add(bos.toByteArray());
+ }
+ return parents;
+ }
+
+ public static void addReplicationParent(Put put, List<RegionInfo> parents) throws IOException {
+ byte[] value = getParentsBytes(parents);
+ put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
+ .setFamily(HConstants.REPLICATION_BARRIER_FAMILY).setQualifier(REPLICATION_PARENT_QUALIFIER)
+ .setTimestamp(put.getTimestamp()).setType(Type.Put).setValue(value).build());
+ }
+
+ public static Put makePutForReplicationBarrier(RegionInfo regionInfo, long openSeqNum, long ts)
+ throws IOException {
+ Put put = new Put(regionInfo.getRegionName(), ts);
+ addReplicationBarrier(put, openSeqNum);
+ return put;
+ }
+
+ public static final class ReplicationBarrierResult {
+ private final long[] barriers;
+ private final RegionState.State state;
+ private final List<byte[]> parentRegionNames;
+
+ ReplicationBarrierResult(long[] barriers, State state, List<byte[]> parentRegionNames) {
+ this.barriers = barriers;
+ this.state = state;
+ this.parentRegionNames = parentRegionNames;
+ }
+
+ public long[] getBarriers() {
+ return barriers;
+ }
+
+ public RegionState.State getState() {
+ return state;
+ }
+
+ public List<byte[]> getParentRegionNames() {
+ return parentRegionNames;
+ }
+
+ @Override
+ public String toString() {
+ return "ReplicationBarrierResult [barriers=" + Arrays.toString(barriers) + ", state=" +
+ state + ", parentRegionNames=" +
+ parentRegionNames.stream().map(Bytes::toStringBinary).collect(Collectors.joining(", ")) +
+ "]";
+ }
+ }
+
+ private static long getReplicationBarrier(Cell c) {
+ return Bytes.toLong(c.getValueArray(), c.getValueOffset(), c.getValueLength());
+ }
+
+ public static long[] getReplicationBarriers(Result result) {
+ return result.getColumnCells(HConstants.REPLICATION_BARRIER_FAMILY, HConstants.SEQNUM_QUALIFIER)
+ .stream().mapToLong(ReplicationBarrierFamilyFormat::getReplicationBarrier).sorted().distinct()
+ .toArray();
+ }
+
+ private static ReplicationBarrierResult getReplicationBarrierResult(Result result) {
+ long[] barriers = getReplicationBarriers(result);
+ byte[] stateBytes = result.getValue(HConstants.CATALOG_FAMILY, HConstants.STATE_QUALIFIER);
+ RegionState.State state =
+ stateBytes != null ? RegionState.State.valueOf(Bytes.toString(stateBytes)) : null;
+ byte[] parentRegionsBytes =
+ result.getValue(HConstants.REPLICATION_BARRIER_FAMILY, REPLICATION_PARENT_QUALIFIER);
+ List<byte[]> parentRegionNames =
+ parentRegionsBytes != null ? parseParentsBytes(parentRegionsBytes) : Collections.emptyList();
+ return new ReplicationBarrierResult(barriers, state, parentRegionNames);
+ }
+
+ public static ReplicationBarrierResult getReplicationBarrierResult(Connection conn,
+ TableName tableName, byte[] row, byte[] encodedRegionName) throws IOException {
+ byte[] metaStartKey = RegionInfo.createRegionName(tableName, row, HConstants.NINES, false);
+ byte[] metaStopKey =
+ RegionInfo.createRegionName(tableName, HConstants.EMPTY_START_ROW, "", false);
+ Scan scan = new Scan().withStartRow(metaStartKey).withStopRow(metaStopKey)
+ .addColumn(HConstants.CATALOG_FAMILY, HConstants.STATE_QUALIFIER)
+ .addFamily(HConstants.REPLICATION_BARRIER_FAMILY).readAllVersions().setReversed(true)
+ .setCaching(10);
+ try (Table table = conn.getTable(TableName.META_TABLE_NAME);
+ ResultScanner scanner = table.getScanner(scan)) {
+ for (Result result;;) {
+ result = scanner.next();
+ if (result == null) {
+ return new ReplicationBarrierResult(new long[0], null, Collections.emptyList());
+ }
+ byte[] regionName = result.getRow();
+ // TODO: we may look up a region which has already been split or merged so we need to check
+ // whether the encoded name matches. Need to find a way to quit earlier when there is no
+ // record for the given region, for now it will scan to the end of the table.
+ if (!Bytes.equals(encodedRegionName,
+ Bytes.toBytes(RegionInfo.encodeRegionName(regionName)))) {
+ continue;
+ }
+ return getReplicationBarrierResult(result);
+ }
+ }
+ }
+
+ public static long[] getReplicationBarrier(Connection conn, byte[] regionName)
Review comment:
getReplicationBarrier => getReplicationBarriers? As this method return a arrary too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#discussion_r484264480
##########
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/CatalogFamilyFormat.java
##########
@@ -346,4 +347,64 @@ public static TableState getTableState(Result r) throws IOException {
throw new IOException(e);
}
}
+
+ /**
+ * @return Deserialized values of <qualifier,regioninfo> pairs taken from column values that
+ * match the regex 'info:merge.*' in array of <code>cells</code>.
+ */
+ @Nullable
+ public static Map<String, RegionInfo> getMergeRegionsWithName(Cell[] cells) {
+ if (cells == null) {
+ return null;
+ }
+ Map<String, RegionInfo> regionsToMerge = null;
+ for (Cell cell : cells) {
+ if (!isMergeQualifierPrefix(cell)) {
+ continue;
+ }
+ // Ok. This cell is that of a info:merge* column.
+ RegionInfo ri = RegionInfo.parseFromOrNull(cell.getValueArray(), cell.getValueOffset(),
+ cell.getValueLength());
+ if (ri != null) {
+ if (regionsToMerge == null) {
+ regionsToMerge = new LinkedHashMap<>();
+ }
+ regionsToMerge.put(Bytes.toString(CellUtil.cloneQualifier(cell)), ri);
+ }
+ }
+ return regionsToMerge;
+ }
+
+ /**
+ * @return Deserialized regioninfo values taken from column values that match the regex
+ * 'info:merge.*' in array of <code>cells</code>.
+ */
+ @Nullable
+ public static List<RegionInfo> getMergeRegions(Cell[] cells) {
+ Map<String, RegionInfo> mergeRegionsWithName = getMergeRegionsWithName(cells);
+ return (mergeRegionsWithName == null) ? null : new ArrayList<>(mergeRegionsWithName.values());
+ }
+
+ /**
+ * @return True if any merge regions present in <code>cells</code>; i.e. the column in
+ * <code>cell</code> matches the regex 'info:merge.*'.
+ */
+ public static boolean hasMergeRegions(Cell[] cells) {
Review comment:
Just copy-paste. Let me change the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688272395
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 23s | master passed |
| +1 :green_heart: | compile | 1m 41s | master passed |
| +1 :green_heart: | shadedjars | 6m 29s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 24s | the patch passed |
| +1 :green_heart: | compile | 1m 40s | the patch passed |
| +1 :green_heart: | javac | 1m 40s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 16s | hbase-balancer generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| -0 :warning: | javadoc | 0m 36s | hbase-server generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 5s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 0m 21s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 139m 37s | hbase-server in the patch passed. |
| | | 170m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 625b81c999fd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| Default Java | 1.8.0_232 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-balancer.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/testReport/ |
| Max. process+thread count | 4466 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688274622
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 59s | master passed |
| +1 :green_heart: | compile | 1m 51s | master passed |
| +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. |
| -0 :warning: | javadoc | 0m 17s | hbase-balancer in master failed. |
| -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 58s | the patch passed |
| +1 :green_heart: | compile | 1m 51s | the patch passed |
| +1 :green_heart: | javac | 1m 51s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 41s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. |
| -0 :warning: | javadoc | 0m 16s | hbase-balancer in the patch failed. |
| -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 7s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 142m 18s | hbase-server in the patch passed. |
| | | 174m 50s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 6dfd86d4d679 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-balancer.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-balancer.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/testReport/ |
| Max. process+thread count | 4089 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688076268
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 38s | master passed |
| +1 :green_heart: | compile | 1m 42s | master passed |
| +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 15s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed |
| +1 :green_heart: | compile | 1m 37s | the patch passed |
| +1 :green_heart: | javac | 1m 37s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 16s | hbase-balancer generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| -0 :warning: | javadoc | 0m 36s | hbase-server generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 6s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | hbase-balancer in the patch passed. |
| +1 :green_heart: | unit | 138m 28s | hbase-server in the patch passed. |
| | | 170m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 12f48376f831 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| Default Java | 1.8.0_232 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-balancer.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/testReport/ |
| Max. process+thread count | 4397 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#discussion_r484237351
##########
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/CatalogFamilyFormat.java
##########
@@ -346,4 +347,64 @@ public static TableState getTableState(Result r) throws IOException {
throw new IOException(e);
}
}
+
+ /**
+ * @return Deserialized values of <qualifier,regioninfo> pairs taken from column values that
+ * match the regex 'info:merge.*' in array of <code>cells</code>.
+ */
+ @Nullable
+ public static Map<String, RegionInfo> getMergeRegionsWithName(Cell[] cells) {
+ if (cells == null) {
+ return null;
+ }
+ Map<String, RegionInfo> regionsToMerge = null;
+ for (Cell cell : cells) {
+ if (!isMergeQualifierPrefix(cell)) {
+ continue;
+ }
+ // Ok. This cell is that of a info:merge* column.
+ RegionInfo ri = RegionInfo.parseFromOrNull(cell.getValueArray(), cell.getValueOffset(),
+ cell.getValueLength());
+ if (ri != null) {
+ if (regionsToMerge == null) {
+ regionsToMerge = new LinkedHashMap<>();
+ }
+ regionsToMerge.put(Bytes.toString(CellUtil.cloneQualifier(cell)), ri);
+ }
+ }
+ return regionsToMerge;
+ }
+
+ /**
+ * @return Deserialized regioninfo values taken from column values that match the regex
+ * 'info:merge.*' in array of <code>cells</code>.
+ */
+ @Nullable
+ public static List<RegionInfo> getMergeRegions(Cell[] cells) {
+ Map<String, RegionInfo> mergeRegionsWithName = getMergeRegionsWithName(cells);
+ return (mergeRegionsWithName == null) ? null : new ArrayList<>(mergeRegionsWithName.values());
+ }
+
+ /**
+ * @return True if any merge regions present in <code>cells</code>; i.e. the column in
+ * <code>cell</code> matches the regex 'info:merge.*'.
+ */
+ public static boolean hasMergeRegions(Cell[] cells) {
Review comment:
Minor suggestion. Can reduce one line code. :-)
for (Cell cell : cells) {
if (isMergeQualifierPrefix(cell)) {
return true;
}
}
return false;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#discussion_r484268135
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationBarrierFamilyFormat.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.RegionState.State;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Helper class for storing replication barriers in family 'rep_barrier' of meta table.
+ * <p/>
+ * See SerialReplicationChecker on how to make use of the barriers.
+ */
+@InterfaceAudience.Private
+public final class ReplicationBarrierFamilyFormat {
+
+ @VisibleForTesting
+ public static final byte[] REPLICATION_PARENT_QUALIFIER = Bytes.toBytes("parent");
+
+ private static final byte ESCAPE_BYTE = (byte) 0xFF;
+
+ private static final byte SEPARATED_BYTE = 0x00;
+
+ private ReplicationBarrierFamilyFormat() {
+ }
+
+ public static void addReplicationBarrier(Put put, long openSeqNum) throws IOException {
+ put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
+ .setFamily(HConstants.REPLICATION_BARRIER_FAMILY).setQualifier(HConstants.SEQNUM_QUALIFIER)
+ .setTimestamp(put.getTimestamp()).setType(Type.Put).setValue(Bytes.toBytes(openSeqNum))
+ .build());
+ }
+
+ private static void writeRegionName(ByteArrayOutputStream out, byte[] regionName) {
+ for (byte b : regionName) {
+ if (b == ESCAPE_BYTE) {
+ out.write(ESCAPE_BYTE);
+ }
+ out.write(b);
+ }
+ }
+
+ @VisibleForTesting
+ public static byte[] getParentsBytes(List<RegionInfo> parents) {
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ Iterator<RegionInfo> iter = parents.iterator();
+ writeRegionName(bos, iter.next().getRegionName());
+ while (iter.hasNext()) {
+ bos.write(ESCAPE_BYTE);
+ bos.write(SEPARATED_BYTE);
+ writeRegionName(bos, iter.next().getRegionName());
+ }
+ return bos.toByteArray();
+ }
+
+ private static List<byte[]> parseParentsBytes(byte[] bytes) {
+ List<byte[]> parents = new ArrayList<>();
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ for (int i = 0; i < bytes.length; i++) {
+ if (bytes[i] == ESCAPE_BYTE) {
+ i++;
+ if (bytes[i] == SEPARATED_BYTE) {
+ parents.add(bos.toByteArray());
+ bos.reset();
+ continue;
+ }
+ // fall through to append the byte
+ }
+ bos.write(bytes[i]);
+ }
+ if (bos.size() > 0) {
+ parents.add(bos.toByteArray());
+ }
+ return parents;
+ }
+
+ public static void addReplicationParent(Put put, List<RegionInfo> parents) throws IOException {
+ byte[] value = getParentsBytes(parents);
+ put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
+ .setFamily(HConstants.REPLICATION_BARRIER_FAMILY).setQualifier(REPLICATION_PARENT_QUALIFIER)
+ .setTimestamp(put.getTimestamp()).setType(Type.Put).setValue(value).build());
+ }
+
+ public static Put makePutForReplicationBarrier(RegionInfo regionInfo, long openSeqNum, long ts)
+ throws IOException {
+ Put put = new Put(regionInfo.getRegionName(), ts);
+ addReplicationBarrier(put, openSeqNum);
+ return put;
+ }
+
+ public static final class ReplicationBarrierResult {
+ private final long[] barriers;
+ private final RegionState.State state;
+ private final List<byte[]> parentRegionNames;
+
+ ReplicationBarrierResult(long[] barriers, State state, List<byte[]> parentRegionNames) {
+ this.barriers = barriers;
+ this.state = state;
+ this.parentRegionNames = parentRegionNames;
+ }
+
+ public long[] getBarriers() {
+ return barriers;
+ }
+
+ public RegionState.State getState() {
+ return state;
+ }
+
+ public List<byte[]> getParentRegionNames() {
+ return parentRegionNames;
+ }
+
+ @Override
+ public String toString() {
+ return "ReplicationBarrierResult [barriers=" + Arrays.toString(barriers) + ", state=" +
+ state + ", parentRegionNames=" +
+ parentRegionNames.stream().map(Bytes::toStringBinary).collect(Collectors.joining(", ")) +
+ "]";
+ }
+ }
+
+ private static long getReplicationBarrier(Cell c) {
+ return Bytes.toLong(c.getValueArray(), c.getValueOffset(), c.getValueLength());
+ }
+
+ public static long[] getReplicationBarriers(Result result) {
+ return result.getColumnCells(HConstants.REPLICATION_BARRIER_FAMILY, HConstants.SEQNUM_QUALIFIER)
+ .stream().mapToLong(ReplicationBarrierFamilyFormat::getReplicationBarrier).sorted().distinct()
+ .toArray();
+ }
+
+ private static ReplicationBarrierResult getReplicationBarrierResult(Result result) {
+ long[] barriers = getReplicationBarriers(result);
+ byte[] stateBytes = result.getValue(HConstants.CATALOG_FAMILY, HConstants.STATE_QUALIFIER);
+ RegionState.State state =
+ stateBytes != null ? RegionState.State.valueOf(Bytes.toString(stateBytes)) : null;
+ byte[] parentRegionsBytes =
+ result.getValue(HConstants.REPLICATION_BARRIER_FAMILY, REPLICATION_PARENT_QUALIFIER);
+ List<byte[]> parentRegionNames =
+ parentRegionsBytes != null ? parseParentsBytes(parentRegionsBytes) : Collections.emptyList();
+ return new ReplicationBarrierResult(barriers, state, parentRegionNames);
+ }
+
+ public static ReplicationBarrierResult getReplicationBarrierResult(Connection conn,
+ TableName tableName, byte[] row, byte[] encodedRegionName) throws IOException {
+ byte[] metaStartKey = RegionInfo.createRegionName(tableName, row, HConstants.NINES, false);
+ byte[] metaStopKey =
+ RegionInfo.createRegionName(tableName, HConstants.EMPTY_START_ROW, "", false);
+ Scan scan = new Scan().withStartRow(metaStartKey).withStopRow(metaStopKey)
+ .addColumn(HConstants.CATALOG_FAMILY, HConstants.STATE_QUALIFIER)
+ .addFamily(HConstants.REPLICATION_BARRIER_FAMILY).readAllVersions().setReversed(true)
+ .setCaching(10);
+ try (Table table = conn.getTable(TableName.META_TABLE_NAME);
+ ResultScanner scanner = table.getScanner(scan)) {
+ for (Result result;;) {
+ result = scanner.next();
+ if (result == null) {
+ return new ReplicationBarrierResult(new long[0], null, Collections.emptyList());
+ }
+ byte[] regionName = result.getRow();
+ // TODO: we may look up a region which has already been split or merged so we need to check
+ // whether the encoded name matches. Need to find a way to quit earlier when there is no
+ // record for the given region, for now it will scan to the end of the table.
+ if (!Bytes.equals(encodedRegionName,
+ Bytes.toBytes(RegionInfo.encodeRegionName(regionName)))) {
+ continue;
+ }
+ return getReplicationBarrierResult(result);
+ }
+ }
+ }
+
+ public static long[] getReplicationBarrier(Connection conn, byte[] regionName)
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#issuecomment-688203680
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 56s | master passed |
| +1 :green_heart: | checkstyle | 1m 48s | master passed |
| +1 :green_heart: | spotbugs | 3m 37s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 50s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 28s | The patch passed checkstyle in hbase-client |
| +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-balancer |
| +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 99 unchanged - 18 fixed = 99 total (was 117) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 4m 32s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
| | | 43m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2354 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 7ca065b414fb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1e8db480b3 |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-client hbase-balancer hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2354/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2354: HBASE-24986 Move ReplicationBarrier related methods to a separated class
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2354:
URL: https://github.com/apache/hbase/pull/2354#discussion_r484263948
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/GCMergedRegionsProcedure.java
##########
@@ -88,7 +87,7 @@ protected Flow executeFromState(MasterProcedureEnv env, GCMergedRegionsState sta
setNextState(GCMergedRegionsState.GC_REGION_EDIT_METADATA);
break;
case GC_REGION_EDIT_METADATA:
- MetaTableAccessor.deleteMergeQualifiers(env.getMasterServices().getConnection(), mergedChild);
+ env.getAssignmentManager().getRegionStateStore().deleteMergeQualifiers(mergedChild);
Review comment:
In real deploy, no. In tests we could have null assignmentmanager or regionstatestore, but as the above pre commit is fine so I do not think it is a problem here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org