You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/03/01 07:15:55 UTC
[GitHub] [hbase] sunhelly opened a new pull request #3001: HBASE-25518 Support separate child regions to different region servers
sunhelly opened a new pull request #3001:
URL: https://github.com/apache/hbase/pull/3001
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
saintstack merged pull request #3001:
URL: https://github.com/apache/hbase/pull/3001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-787833190
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 53s | master passed |
| +1 :green_heart: | compile | 1m 46s | master passed |
| +1 :green_heart: | shadedjars | 8m 2s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 13s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 51s | the patch passed |
| +1 :green_heart: | compile | 1m 47s | the patch passed |
| +1 :green_heart: | javac | 1m 47s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 13s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 146m 49s | hbase-server in the patch passed. |
| | | 183m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux f502a037ca15 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 34bd1bda6f |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/testReport/ |
| Max. process+thread count | 3995 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800639694
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 8s | master passed |
| +1 :green_heart: | compile | 1m 43s | master passed |
| +1 :green_heart: | shadedjars | 8m 26s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 25s | the patch passed |
| +1 :green_heart: | compile | 1m 34s | the patch passed |
| +1 :green_heart: | javac | 1m 34s | the patch passed |
| -1 :x: | shadedjars | 6m 22s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. |
| -1 :x: | unit | 226m 36s | hbase-server in the patch failed. |
| | | 261m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8cb737d4bec5 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/testReport/ |
| Max. process+thread count | 3136 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800504302
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 46s | master passed |
| +1 :green_heart: | compile | 3m 57s | master passed |
| +1 :green_heart: | checkstyle | 1m 31s | master passed |
| +1 :green_heart: | spotbugs | 2m 49s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 36s | the patch passed |
| +1 :green_heart: | compile | 4m 2s | the patch passed |
| +1 :green_heart: | javac | 4m 2s | the patch passed |
| -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 3 new + 1 unchanged - 1 fixed = 4 total (was 2) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 10s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 27s | The patch generated 1 ASF License warnings. |
| | | 51m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux bff6acf337ea 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800481912
Shout if you need input on how to fix the licensing issue.
The test failure looks related. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r594754063
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManagerUtil.java
##########
@@ -189,6 +190,67 @@ private static void unlock(List<RegionStateNode> regionNodes) {
return ArrayUtils.addAll(primaryRegionProcs, replicaRegionAssignProcs);
}
+ /**
+ * Create round robin assign procedures for the give regions,
Review comment:
s/give/given/
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##########
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
- assertEquals("Table region not correct.", 2,
- UTIL.getHBaseCluster().getRegions(tableName).size());
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+ cleanupTest();
+ //restart master
+ UTIL.getConfiguration().setBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS, true);
+ setupCluster();
+
+ final TableName tableName = TableName.valueOf(name.getMethodName());
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+
+ RegionInfo[] regions =
+ MasterProcedureTestingUtility.createTable(procExec, tableName, null, ColumnFamilyName);
+ insertData(tableName);
+ int splitRowNum = startRowNum + rowCount / 2;
+ byte[] splitKey = Bytes.toBytes("" + splitRowNum);
+
+ assertTrue("not able to find a splittable region", regions != null);
+ assertTrue("not able to find a splittable region", regions.length == 1);
+
+ // Split region of the table
+ long procId = procExec.submitProcedure(
+ new SplitTableRegionProcedure(procExec.getEnvironment(), regions[0], splitKey));
+ // Wait the completion
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+ ProcedureTestingUtility.assertProcNotFailed(procExec, procId);
+
+ assertTrue("not able to split table", UTIL.getHBaseCluster().getRegions(tableName).size() == 2);
+
+ //disable table
+ UTIL.getAdmin().disableTable(tableName);
+ Thread.sleep(500);
+
+ //stop master
+ UTIL.getHBaseCluster().stopMaster(0);
+ UTIL.getHBaseCluster().waitOnMaster(0);
+ Thread.sleep(500);
+
+ //restart master
+ JVMClusterUtil.MasterThread t = UTIL.getHBaseCluster().startMaster();
+ Thread.sleep(500);
+
+ UTIL.invalidateConnection();
+ // enable table
+ UTIL.getAdmin().enableTable(tableName);
+ Thread.sleep(500);
+
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertNotEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
Review comment:
How can you be sure the roundrobin will not assign both daughters to same server?
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##########
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
- assertEquals("Table region not correct.", 2,
- UTIL.getHBaseCluster().getRegions(tableName).size());
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+ cleanupTest();
Review comment:
You should make a new test suite with this test in it rather than do this cleanupTest and setupCluster... These methods are for before the test class and after... not for calling in the test as here.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManagerUtil.java
##########
@@ -189,6 +190,67 @@ private static void unlock(List<RegionStateNode> regionNodes) {
return ArrayUtils.addAll(primaryRegionProcs, replicaRegionAssignProcs);
}
+ /**
+ * Create round robin assign procedures for the give regions,
+ * according to the {@code regionReplication}.
+ * <p/>
+ * For rolling back, we will submit procedures directly to the {@code ProcedureExecutor}, so it is
+ * possible that we persist the newly scheduled procedures, and then crash before persisting the
+ * rollback state, so when we arrive here the second time, it is possible that some regions have
+ * already been associated with a TRSP.
+ * @param ignoreIfInTransition if true, will skip creating TRSP for the given region if it is
+ * already in transition, otherwise we will add an assert that it should not in
+ * transition.
+ */
+ private static TransitRegionStateProcedure[] createRoundRobinAssignProcedures(
+ MasterProcedureEnv env, List<RegionInfo> regions, int regionReplication,
+ List<ServerName> serversToExclude, boolean ignoreIfInTransition) {
+ List<RegionInfo> regionsAndReplicas = new ArrayList<>(regions);
+ if (regionReplication != DEFAULT_REGION_REPLICA) {
+
+ // collect the replica region infos
+ List<RegionInfo> replicaRegionInfos =
+ new ArrayList<RegionInfo>(regions.size() * (regionReplication - 1));
+ for (RegionInfo hri : regions) {
+ // start the index from 1
+ for (int i = 1; i < regionReplication; i++) {
+ replicaRegionInfos.add(RegionReplicaUtil.getRegionInfoForReplica(hri, i));
+ }
+ }
+ regionsAndReplicas.addAll(replicaRegionInfos);
+ }
+ if (ignoreIfInTransition) {
+ for (RegionInfo region : regionsAndReplicas) {
+ if (env.getAssignmentManager().getRegionStates().getOrCreateRegionStateNode(region)
+ .isInTransition()) {
+ return null;
+ }
+ }
+ }
+ // create round robin procs. Note that we exclude the primary region's target server
+ return env.getAssignmentManager()
+ .createRoundRobinAssignProcedures(regionsAndReplicas, serversToExclude);
+ }
+
+ static TransitRegionStateProcedure[] createAssignProceduresForSplitDaughters(
+ MasterProcedureEnv env, List<RegionInfo> daughters, int regionReplication,
+ ServerName parentServer) {
+ if(env.getMasterConfiguration().getBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS,
+ false)){
+ // keep daughter one on the parent RS
Review comment:
s/keep daughter one/keep one daughter/
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##########
@@ -156,6 +156,10 @@
/** Default value for the balancer period */
public static final int DEFAULT_HBASE_BALANCER_PERIOD = 300000;
+ /** Config for support separate child regions */
Review comment:
I think this config needs more explaination about what it enables. Does it need a default setting too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800299322
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 48s | master passed |
| +1 :green_heart: | compile | 4m 3s | master passed |
| +1 :green_heart: | checkstyle | 1m 28s | master passed |
| +1 :green_heart: | spotbugs | 2m 51s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 27s | the patch passed |
| +1 :green_heart: | compile | 3m 59s | the patch passed |
| +1 :green_heart: | javac | 3m 59s | the patch passed |
| -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 2 new + 1 unchanged - 1 fixed = 3 total (was 2) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 11s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 27s | The patch generated 1 ASF License warnings. |
| | | 52m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b9a1b3e0e0a8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800778549
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 33s | master passed |
| +1 :green_heart: | compile | 4m 2s | master passed |
| +1 :green_heart: | checkstyle | 1m 29s | master passed |
| +1 :green_heart: | spotbugs | 2m 46s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 30s | the patch passed |
| +1 :green_heart: | compile | 4m 5s | the patch passed |
| +1 :green_heart: | javac | 4m 5s | the patch passed |
| -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 3 new + 1 unchanged - 1 fixed = 4 total (was 2) |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 22s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 50m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4a355c869731 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
sunhelly commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r595111408
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##########
@@ -156,6 +156,10 @@
/** Default value for the balancer period */
public static final int DEFAULT_HBASE_BALANCER_PERIOD = 300000;
+ /** Config for support separate child regions */
Review comment:
Thanks, I'll add more explanation and a default value for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800857771
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 26s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 52s | master passed |
| +1 :green_heart: | compile | 1m 23s | master passed |
| +1 :green_heart: | shadedjars | 7m 19s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 56s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 26s | the patch passed |
| +1 :green_heart: | javac | 1m 26s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 22s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 209m 44s | hbase-server in the patch passed. |
| | | 243m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 96b01f72a5c5 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/testReport/ |
| Max. process+thread count | 3855 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r595411259
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##########
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
- assertEquals("Table region not correct.", 2,
- UTIL.getHBaseCluster().getRegions(tableName).size());
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+ cleanupTest();
+ //restart master
+ UTIL.getConfiguration().setBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS, true);
+ setupCluster();
+
+ final TableName tableName = TableName.valueOf(name.getMethodName());
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+
+ RegionInfo[] regions =
+ MasterProcedureTestingUtility.createTable(procExec, tableName, null, ColumnFamilyName);
+ insertData(tableName);
+ int splitRowNum = startRowNum + rowCount / 2;
+ byte[] splitKey = Bytes.toBytes("" + splitRowNum);
+
+ assertTrue("not able to find a splittable region", regions != null);
+ assertTrue("not able to find a splittable region", regions.length == 1);
+
+ // Split region of the table
+ long procId = procExec.submitProcedure(
+ new SplitTableRegionProcedure(procExec.getEnvironment(), regions[0], splitKey));
+ // Wait the completion
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+ ProcedureTestingUtility.assertProcNotFailed(procExec, procId);
+
+ assertTrue("not able to split table", UTIL.getHBaseCluster().getRegions(tableName).size() == 2);
+
+ //disable table
+ UTIL.getAdmin().disableTable(tableName);
+ Thread.sleep(500);
+
+ //stop master
+ UTIL.getHBaseCluster().stopMaster(0);
+ UTIL.getHBaseCluster().waitOnMaster(0);
+ Thread.sleep(500);
+
+ //restart master
+ JVMClusterUtil.MasterThread t = UTIL.getHBaseCluster().startMaster();
+ Thread.sleep(500);
+
+ UTIL.invalidateConnection();
+ // enable table
+ UTIL.getAdmin().enableTable(tableName);
+ Thread.sleep(500);
+
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertNotEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800584080
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 7s | master passed |
| +1 :green_heart: | compile | 1m 34s | master passed |
| +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 4s | the patch passed |
| +1 :green_heart: | compile | 1m 32s | the patch passed |
| +1 :green_heart: | javac | 1m 32s | the patch passed |
| -1 :x: | shadedjars | 5m 28s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hbase-common in the patch passed. |
| -1 :x: | unit | 144m 38s | hbase-server in the patch failed. |
| | | 175m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a58b905691b4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/testReport/ |
| Max. process+thread count | 4936 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
sunhelly commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r595118863
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##########
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
- assertEquals("Table region not correct.", 2,
- UTIL.getHBaseCluster().getRegions(tableName).size());
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+ cleanupTest();
+ //restart master
+ UTIL.getConfiguration().setBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS, true);
+ setupCluster();
+
+ final TableName tableName = TableName.valueOf(name.getMethodName());
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+
+ RegionInfo[] regions =
+ MasterProcedureTestingUtility.createTable(procExec, tableName, null, ColumnFamilyName);
+ insertData(tableName);
+ int splitRowNum = startRowNum + rowCount / 2;
+ byte[] splitKey = Bytes.toBytes("" + splitRowNum);
+
+ assertTrue("not able to find a splittable region", regions != null);
+ assertTrue("not able to find a splittable region", regions.length == 1);
+
+ // Split region of the table
+ long procId = procExec.submitProcedure(
+ new SplitTableRegionProcedure(procExec.getEnvironment(), regions[0], splitKey));
+ // Wait the completion
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+ ProcedureTestingUtility.assertProcNotFailed(procExec, procId);
+
+ assertTrue("not able to split table", UTIL.getHBaseCluster().getRegions(tableName).size() == 2);
+
+ //disable table
+ UTIL.getAdmin().disableTable(tableName);
+ Thread.sleep(500);
+
+ //stop master
+ UTIL.getHBaseCluster().stopMaster(0);
+ UTIL.getHBaseCluster().waitOnMaster(0);
+ Thread.sleep(500);
+
+ //restart master
+ JVMClusterUtil.MasterThread t = UTIL.getHBaseCluster().startMaster();
+ Thread.sleep(500);
+
+ UTIL.invalidateConnection();
+ // enable table
+ UTIL.getAdmin().enableTable(tableName);
+ Thread.sleep(500);
+
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertNotEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
Review comment:
Because the test cluster has three region servers, and before the round robin assignment, one daughter will be kept on the parent server, which will be excluded from the round robin servers set. At last, the other daughter and its replicas will be assigned to the other two servers. More than one servers in the destination server set will make round robin work well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-787830120
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 48s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 42s | master passed |
| +1 :green_heart: | compile | 1m 21s | master passed |
| +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 33s | the patch passed |
| +1 :green_heart: | compile | 1m 23s | the patch passed |
| +1 :green_heart: | javac | 1m 23s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 144m 28s | hbase-server in the patch passed. |
| | | 178m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5bc455c47078 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 34bd1bda6f |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/testReport/ |
| Max. process+thread count | 4183 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800462769
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 2s | master passed |
| +1 :green_heart: | compile | 1m 22s | master passed |
| +1 :green_heart: | shadedjars | 7m 10s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 52s | the patch passed |
| +1 :green_heart: | compile | 1m 22s | the patch passed |
| +1 :green_heart: | javac | 1m 22s | the patch passed |
| -1 :x: | shadedjars | 5m 44s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 56s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. |
| -1 :x: | unit | 211m 45s | hbase-server in the patch failed. |
| | | 242m 35s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 912b2ddb45b0 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/testReport/ |
| Max. process+thread count | 3902 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800823727
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 14s | master passed |
| +1 :green_heart: | compile | 1m 33s | master passed |
| +1 :green_heart: | shadedjars | 6m 50s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 1m 33s | the patch passed |
| +1 :green_heart: | javac | 1m 33s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 139m 25s | hbase-server in the patch passed. |
| | | 171m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux b08d954de0e1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/testReport/ |
| Max. process+thread count | 4517 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] sunhelly commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
sunhelly commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r595114912
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManagerUtil.java
##########
@@ -189,6 +190,67 @@ private static void unlock(List<RegionStateNode> regionNodes) {
return ArrayUtils.addAll(primaryRegionProcs, replicaRegionAssignProcs);
}
+ /**
+ * Create round robin assign procedures for the give regions,
Review comment:
Thanks, I'll change it.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManagerUtil.java
##########
@@ -189,6 +190,67 @@ private static void unlock(List<RegionStateNode> regionNodes) {
return ArrayUtils.addAll(primaryRegionProcs, replicaRegionAssignProcs);
}
+ /**
+ * Create round robin assign procedures for the give regions,
+ * according to the {@code regionReplication}.
+ * <p/>
+ * For rolling back, we will submit procedures directly to the {@code ProcedureExecutor}, so it is
+ * possible that we persist the newly scheduled procedures, and then crash before persisting the
+ * rollback state, so when we arrive here the second time, it is possible that some regions have
+ * already been associated with a TRSP.
+ * @param ignoreIfInTransition if true, will skip creating TRSP for the given region if it is
+ * already in transition, otherwise we will add an assert that it should not in
+ * transition.
+ */
+ private static TransitRegionStateProcedure[] createRoundRobinAssignProcedures(
+ MasterProcedureEnv env, List<RegionInfo> regions, int regionReplication,
+ List<ServerName> serversToExclude, boolean ignoreIfInTransition) {
+ List<RegionInfo> regionsAndReplicas = new ArrayList<>(regions);
+ if (regionReplication != DEFAULT_REGION_REPLICA) {
+
+ // collect the replica region infos
+ List<RegionInfo> replicaRegionInfos =
+ new ArrayList<RegionInfo>(regions.size() * (regionReplication - 1));
+ for (RegionInfo hri : regions) {
+ // start the index from 1
+ for (int i = 1; i < regionReplication; i++) {
+ replicaRegionInfos.add(RegionReplicaUtil.getRegionInfoForReplica(hri, i));
+ }
+ }
+ regionsAndReplicas.addAll(replicaRegionInfos);
+ }
+ if (ignoreIfInTransition) {
+ for (RegionInfo region : regionsAndReplicas) {
+ if (env.getAssignmentManager().getRegionStates().getOrCreateRegionStateNode(region)
+ .isInTransition()) {
+ return null;
+ }
+ }
+ }
+ // create round robin procs. Note that we exclude the primary region's target server
+ return env.getAssignmentManager()
+ .createRoundRobinAssignProcedures(regionsAndReplicas, serversToExclude);
+ }
+
+ static TransitRegionStateProcedure[] createAssignProceduresForSplitDaughters(
+ MasterProcedureEnv env, List<RegionInfo> daughters, int regionReplication,
+ ServerName parentServer) {
+ if(env.getMasterConfiguration().getBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS,
+ false)){
+ // keep daughter one on the parent RS
Review comment:
Thanks, I'll change it.
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##########
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
- assertEquals("Table region not correct.", 2,
- UTIL.getHBaseCluster().getRegions(tableName).size());
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+ cleanupTest();
+ //restart master
+ UTIL.getConfiguration().setBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS, true);
+ setupCluster();
+
+ final TableName tableName = TableName.valueOf(name.getMethodName());
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+
+ RegionInfo[] regions =
+ MasterProcedureTestingUtility.createTable(procExec, tableName, null, ColumnFamilyName);
+ insertData(tableName);
+ int splitRowNum = startRowNum + rowCount / 2;
+ byte[] splitKey = Bytes.toBytes("" + splitRowNum);
+
+ assertTrue("not able to find a splittable region", regions != null);
+ assertTrue("not able to find a splittable region", regions.length == 1);
+
+ // Split region of the table
+ long procId = procExec.submitProcedure(
+ new SplitTableRegionProcedure(procExec.getEnvironment(), regions[0], splitKey));
+ // Wait the completion
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+ ProcedureTestingUtility.assertProcNotFailed(procExec, procId);
+
+ assertTrue("not able to split table", UTIL.getHBaseCluster().getRegions(tableName).size() == 2);
+
+ //disable table
+ UTIL.getAdmin().disableTable(tableName);
+ Thread.sleep(500);
+
+ //stop master
+ UTIL.getHBaseCluster().stopMaster(0);
+ UTIL.getHBaseCluster().waitOnMaster(0);
+ Thread.sleep(500);
+
+ //restart master
+ JVMClusterUtil.MasterThread t = UTIL.getHBaseCluster().startMaster();
+ Thread.sleep(500);
+
+ UTIL.invalidateConnection();
+ // enable table
+ UTIL.getAdmin().enableTable(tableName);
+ Thread.sleep(500);
+
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertNotEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
Review comment:
Because the test cluster has three region servers, and before the round robin assignment, one daughter will be kept on the parent server, which will be excluded from the round robin servers set. At last, the other daughter ant its replicas will be assigned to the other two servers. More than one servers in the destination server set will make round robin work well.
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##########
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
- assertEquals("Table region not correct.", 2,
- UTIL.getHBaseCluster().getRegions(tableName).size());
+ List<HRegion> tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+ assertEquals("Table region not correct.", 2, tableRegions.size());
+ Map<RegionInfo, ServerName> regionInfoMap = UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+ assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+ cleanupTest();
Review comment:
Thanks, I'll change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-800466111
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 40s | master passed |
| +1 :green_heart: | compile | 1m 38s | master passed |
| +1 :green_heart: | shadedjars | 7m 30s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 43s | the patch passed |
| +1 :green_heart: | compile | 1m 40s | the patch passed |
| +1 :green_heart: | javac | 1m 40s | the patch passed |
| -1 :x: | shadedjars | 5m 58s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. |
| -1 :x: | unit | 212m 37s | hbase-server in the patch failed. |
| | | 246m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux c0c01a7502bf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d74ae15fa7 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/testReport/ |
| Max. process+thread count | 3546 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#issuecomment-787744842
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 46s | master passed |
| +1 :green_heart: | compile | 4m 1s | master passed |
| +1 :green_heart: | checkstyle | 1m 29s | master passed |
| +1 :green_heart: | spotbugs | 2m 42s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 29s | the patch passed |
| +1 :green_heart: | compile | 4m 0s | the patch passed |
| +1 :green_heart: | javac | 4m 0s | the patch passed |
| -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| | | 50m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3001 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a26ad2dd6d09 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 34bd1bda6f |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org