You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/08/25 23:31:14 UTC
[GitHub] [hbase] bharathv opened a new pull request #2315: HBASE-24765: Dynamic master discovery
bharathv opened a new pull request #2315:
URL: https://github.com/apache/hbase/pull/2315
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <nd...@apache.org>
Signed-off-by: Viraj Jasani <vj...@apache.org>
Signed-off-by: Duo Zhang <zh...@apache.org>
(cherry picked from commit 275a38e1533eafa1d4bd1d50c13bcecd9a397ea8)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2315: HBASE-24765: Dynamic master discovery
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2315:
URL: https://github.com/apache/hbase/pull/2315#issuecomment-680649628
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 35s | branch-2.3 passed |
| +1 :green_heart: | compile | 2m 49s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 6m 19s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 30s | hbase-client in branch-2.3 failed. |
| -0 :warning: | javadoc | 0m 38s | hbase-server in branch-2.3 failed. |
| -0 :warning: | javadoc | 0m 17s | hbase-zookeeper in branch-2.3 failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 8s | the patch passed |
| +1 :green_heart: | compile | 2m 47s | the patch passed |
| +1 :green_heart: | javac | 2m 47s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 49s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. |
| -0 :warning: | javadoc | 0m 15s | hbase-zookeeper in the patch failed. |
| -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 53s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 25s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 0m 41s | hbase-zookeeper in the patch passed. |
| +1 :green_heart: | unit | 133m 10s | hbase-server in the patch passed. |
| | | 170m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2315 |
| JIRA Issue | HBASE-24765 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9a5638596f5c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / 83b8dfe947 |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-zookeeper.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/testReport/ |
| Max. process+thread count | 3884 (vs. ulimit of 12500) |
| modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2315: HBASE-24765: Dynamic master discovery
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2315:
URL: https://github.com/apache/hbase/pull/2315#discussion_r477157027
##########
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
##########
@@ -259,17 +282,54 @@ private RegionLocations transformMetaRegionLocations(GetMetaRegionLocationsRespo
.thenApply(GetClusterIdResponse::getClusterId);
}
- private ServerName transformServerName(GetActiveMasterResponse resp) {
- return ProtobufUtil.toServerName(resp.getServerName());
+ private static boolean hasActiveMaster(GetMastersResponse resp) {
+ List<GetMastersResponseEntry> activeMasters =
+ resp.getMasterServersList().stream().filter(GetMastersResponseEntry::getIsActive).collect(
+ Collectors.toList());
+ return activeMasters.size() == 1;
+ }
+
+ private static ServerName filterActiveMaster(GetMastersResponse resp) throws IOException {
+ List<GetMastersResponseEntry> activeMasters =
+ resp.getMasterServersList().stream().filter(GetMastersResponseEntry::getIsActive).collect(
+ Collectors.toList());
+ if (activeMasters.size() != 1) {
+ throw new IOException(String.format("Incorrect number of active masters encountered." +
+ " Expected: 1 found: %d. Content: %s", activeMasters.size(), activeMasters));
+ }
+ return ProtobufUtil.toServerName(activeMasters.get(0).getServerName());
}
@Override
public CompletableFuture<ServerName> getActiveMaster() {
+ CompletableFuture<ServerName> future = new CompletableFuture<>();
+ addListener(call((c, s, d) -> s.getMasters(c, GetMastersRequest.getDefaultInstance(), d),
Review comment:
As said on jira, for branch-2.3, we'd better still use the getActiveMaster call here to keep the wire compatibility.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2315: HBASE-24765: Dynamic master discovery
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2315:
URL: https://github.com/apache/hbase/pull/2315#issuecomment-680672937
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 14s | branch-2.3 passed |
| +1 :green_heart: | compile | 2m 44s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 5m 32s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 38s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 51s | the patch passed |
| +1 :green_heart: | compile | 2m 33s | the patch passed |
| +1 :green_heart: | javac | 2m 33s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 23s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 30s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 2m 31s | hbase-client in the patch passed. |
| +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. |
| +1 :green_heart: | unit | 175m 56s | hbase-server in the patch passed. |
| | | 212m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2315 |
| JIRA Issue | HBASE-24765 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 4875d4ad63f4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / 83b8dfe947 |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/testReport/ |
| Max. process+thread count | 3524 (vs. ulimit of 12500) |
| modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #2315: HBASE-24765: Dynamic master discovery
Posted by GitBox <gi...@apache.org>.
bharathv commented on pull request #2315:
URL: https://github.com/apache/hbase/pull/2315#issuecomment-680978736
Abandoning per discussion in the jira.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2315: HBASE-24765: Dynamic master discovery
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2315:
URL: https://github.com/apache/hbase/pull/2315#issuecomment-680489162
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 43s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2.3 Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 58s | branch-2.3 passed |
| +1 :green_heart: | checkstyle | 1m 58s | branch-2.3 passed |
| +1 :green_heart: | spotbugs | 6m 7s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-protocol-shaded |
| +1 :green_heart: | checkstyle | 0m 32s | The patch passed checkstyle in hbase-client |
| +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-zookeeper |
| +1 :green_heart: | checkstyle | 1m 9s | hbase-server: The patch generated 0 new + 111 unchanged - 2 fixed = 111 total (was 113) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 54s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. |
| +1 :green_heart: | hbaseprotoc | 2m 39s | the patch passed |
| +1 :green_heart: | spotbugs | 7m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 51s | The patch does not generate ASF License warnings. |
| | | 58m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2315 |
| JIRA Issue | HBASE-24765 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool |
| uname | Linux 5694f04c3124 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / 83b8dfe947 |
| Max. process+thread count | 94 (vs. ulimit of 12500) |
| modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2315/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv closed pull request #2315: HBASE-24765: Dynamic master discovery
Posted by GitBox <gi...@apache.org>.
bharathv closed pull request #2315:
URL: https://github.com/apache/hbase/pull/2315
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org