You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/10/22 07:25:32 UTC
[GitHub] [hbase] ddupg opened a new pull request #2579: HBASE-24999 Master manages ReplicationServers
ddupg opened a new pull request #2579:
URL: https://github.com/apache/hbase/pull/2579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ddupg commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
ddupg commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r512399178
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##########
@@ -3402,4 +3406,33 @@ public ListReplicationSinkServersResponse listReplicationSinkServers(
}
return builder.build();
}
+
+ @Override
+ public RegionServerReportResponse replicationServerReport(RpcController controller,
+ RegionServerReportRequest request) throws ServiceException {
+ try {
+ master.checkServiceStarted();
+ int versionNumber = 0;
+ String version = "0.0.0";
+ VersionInfo versionInfo = VersionInfoUtil.getCurrentClientVersionInfo();
+ if (versionInfo != null) {
+ version = versionInfo.getVersion();
+ versionNumber = VersionInfoUtil.getVersionNumber(versionInfo);
+ }
+ ClusterStatusProtos.ServerLoad sl = request.getLoad();
+ ServerName serverName = ProtobufUtil.toServerName(request.getServer());
+ ServerMetrics oldMetrics = master.getReplicationServerManager().getServerMetrics(serverName);
+ ServerMetrics newMetrics =
+ ServerMetricsBuilder.toServerMetrics(serverName, versionNumber, version, sl);
+ master.getReplicationServerManager().serverReport(serverName, newMetrics);
+ if (sl != null && master.metricsMaster != null) {
+ // Up our metrics.
+ master.metricsMaster.incrementRequests(sl.getTotalNumberOfRequests()
Review comment:
RegionServer increments request count metric in all RPC method, including `replicateWALEntry`, so the request count metric should mean more than just reading and writing metric?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
##########
@@ -388,4 +446,152 @@ protected ReplicationServerRpcServices createRpcServices() throws IOException {
protected boolean setAbortRequested() {
return abortRequested.compareAndSet(false, true);
}
+
+ private void tryReplicationServerReport(long reportStartTime, long reportEndTime)
+ throws IOException {
+ ReplicationServerStatusService.BlockingInterface rss = rssStub;
+ if (rss == null) {
+ createReplicationServerStatusStub(true);
+ rss = rssStub;
+ if (rss == null) {
+ return;
+ }
+ }
+ ClusterStatusProtos.ServerLoad sl = buildServerLoad(reportStartTime, reportEndTime);
+ try {
+ RegionServerReportRequest.Builder request = RegionServerReportRequest
+ .newBuilder();
+ request.setServer(ProtobufUtil.toServerName(this.serverName));
+ request.setLoad(sl);
+ rss.replicationServerReport(null, request.build());
+ } catch (ServiceException se) {
+ IOException ioe = ProtobufUtil.getRemoteException(se);
+ if (ioe instanceof YouAreDeadException) {
+ // This will be caught and handled as a fatal error in run()
+ throw ioe;
+ }
+ if (rssStub == rss) {
+ rssStub = null;
+ }
+ // Couldn't connect to the master, get location from zk and reconnect
+ // Method blocks until new master is found or we are stopped
+ createReplicationServerStatusStub(true);
+ }
+ }
+
+ private ClusterStatusProtos.ServerLoad buildServerLoad(long reportStartTime, long reportEndTime) {
+ long usedMemory = -1L;
+ long maxMemory = -1L;
+ final MemoryUsage usage = MemorySizeUtil.safeGetHeapMemoryUsage();
+ if (usage != null) {
+ usedMemory = usage.getUsed();
+ maxMemory = usage.getMax();
+ }
+
+ ClusterStatusProtos.ServerLoad.Builder serverLoad = ClusterStatusProtos.ServerLoad.newBuilder();
+ serverLoad.setTotalNumberOfRequests(rpcServices.requestCount.sum());
+ serverLoad.setUsedHeapMB((int) (usedMemory / 1024 / 1024));
+ serverLoad.setMaxHeapMB((int) (maxMemory / 1024 / 1024));
+
+ serverLoad.setReportStartTime(reportStartTime);
+ serverLoad.setReportEndTime(reportEndTime);
+
+ // for the replicationLoad purpose. Only need to get from one executorService
+ // either source or sink will get the same info
+ ReplicationSinkService sinks = getReplicationSinkService();
+
+ if (sinks != null) {
+ // always refresh first to get the latest value
+ ReplicationLoad rLoad = sinks.refreshAndGetReplicationLoad();
+ if (rLoad != null) {
+ serverLoad.setReplLoadSink(rLoad.getReplicationLoadSink());
+ }
+ }
+ return serverLoad.build();
+ }
+
+ /**
+ * Get the current master from ZooKeeper and open the RPC connection to it. To get a fresh
+ * connection, the current rssStub must be null. Method will block until a master is available.
+ * You can break from this block by requesting the server stop.
+ * @param refresh If true then master address will be read from ZK, otherwise use cached data
+ * @return master + port, or null if server has been stopped
+ */
+ private synchronized ServerName createReplicationServerStatusStub(boolean refresh) {
Review comment:
It should be useful, let me fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio edited a comment on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio edited a comment on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716902255
> Thanks @infraio for reviewing. I have fixed as comments.
>
> > Add master UI to show ReplicationServer?
>
> Do it in this PR? I was going to do it in a new PR?
Ok. Let's do it in follow-on PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r510244366
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##########
@@ -288,39 +258,26 @@ public boolean isAborted() {
return false;
}
- /**
- * Get the connection to peer cluster
- * @return connection to peer cluster
- * @throws IOException If anything goes wrong connecting
- */
- private synchronized AsyncClusterConnection getPeerConnection() throws IOException {
- if (peerConnection == null) {
- Configuration conf = ctx.getConfiguration();
- peerConnection = ClusterConnectionFactory.createAsyncClusterConnection(conf, null,
- UserProvider.instantiate(conf).getCurrent());
- }
- return peerConnection;
- }
-
/**
* Get the list of all the servers that are responsible for replication sink
* from the specified peer master
* @return list of server addresses or an empty list if the slave is unavailable
Review comment:
Is it true that an empty list would be returned if we can't reach slave master? Looks like we either throw an IOException or a RemoteException.
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/ReplicationServerManager.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.ServerMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The ServerManager class manages info about replication servers.
+ * <p>
+ * Maintains lists of online and dead servers.
+ * <p>
+ * Servers are distinguished in two different ways. A given server has a
+ * location, specified by hostname and port, and of which there can only be one
+ * online at any given time. A server instance is specified by the location
+ * (hostname and port) as well as the startcode (timestamp from when the server
+ * was started). This is used to differentiate a restarted instance of a given
+ * server from the original instance.
+ */
+@InterfaceAudience.Private
+public class ReplicationServerManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReplicationServerManager.class);
+
+ public static final String ONLINE_SERVER_REFRESH_INTERVAL =
+ "hbase.master.replication.server.refresh.interval";
+ public static final int ONLINE_SERVER_REFRESH_INTERVAL_DEFAULT = 60 * 1000; // 1 mins
+
+ private final MasterServices master;
+
+ /** Map of registered servers to their current load */
+ private final ConcurrentNavigableMap<ServerName, ServerMetrics> onlineServers =
+ new ConcurrentSkipListMap<>();
+
+ private OnlineServerRefresher onlineServerRefresher;
+ private int refreshPeriod;
+
+ /**
+ * Constructor.
+ */
+ public ReplicationServerManager(final MasterServices master) {
+ this.master = master;
+ }
+
+ /**
+ * start chore in ServerManager
+ */
+ public void startChore() {
+ Configuration conf = master.getConfiguration();
+ refreshPeriod = conf.getInt(ONLINE_SERVER_REFRESH_INTERVAL,
+ ONLINE_SERVER_REFRESH_INTERVAL_DEFAULT);
+ onlineServerRefresher = new OnlineServerRefresher("ReplicationServerRefresher", refreshPeriod);
+ master.getChoreService().scheduleChore(onlineServerRefresher);
+ }
+
+ /**
+ * Stop the ServerManager.
+ */
+ public void stop() {
+ if (onlineServerRefresher != null) {
+ onlineServerRefresher.cancel();
+ }
+ }
+
+ public void serverReport(ServerName sn, ServerMetrics sl) {
+ if (null == this.onlineServers.replace(sn, sl)) {
+ if (!checkAndRecordNewServer(sn, sl)) {
+ LOG.info("ReplicationServerReport ignored, could not record the server: {}", sn);
+ }
+ }
+ }
+
+ /**
+ * Check is a server of same host and port already exists,
+ * if not, or the existed one got a smaller start code, record it.
+ *
+ * @param serverName the server to check and record
+ * @param sl the server load on the server
+ * @return true if the server is recorded, otherwise, false
+ */
+ boolean checkAndRecordNewServer(final ServerName serverName, final ServerMetrics sl) {
+ ServerName existingServer = null;
+ synchronized (this.onlineServers) {
+ existingServer = findServerWithSameHostnamePortWithLock(serverName);
+ if (existingServer != null && (existingServer.getStartcode() > serverName.getStartcode())) {
+ LOG.info("ReplicationServer serverName={} rejected; we already have {} registered with "
+ + "same hostname and port", serverName, existingServer);
+ return false;
+ }
+ recordNewServerWithLock(serverName, sl);
+ // Note that we assume that same ts means same server, and don't expire in that case.
+ if (existingServer != null && (existingServer.getStartcode() < serverName.getStartcode())) {
+ LOG.info("Triggering server recovery; existingServer {} looks stale, new server: {}",
+ existingServer, serverName);
+ expireServerWithLock(existingServer);
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Assumes onlineServers is locked.
+ * @return ServerName with matching hostname and port.
+ */
+ private ServerName findServerWithSameHostnamePortWithLock(final ServerName serverName) {
+ ServerName end = ServerName.valueOf(serverName.getHostname(), serverName.getPort(),
+ Long.MAX_VALUE);
+
+ ServerName r = onlineServers.lowerKey(end);
+ if (r != null && ServerName.isSameAddress(r, serverName)) {
+ return r;
+ }
+ return null;
+ }
+
+ private void recordNewServerWithLock(final ServerName serverName, final ServerMetrics sl) {
+ LOG.info("Registering ReplicationServer={}", serverName);
+ this.onlineServers.put(serverName, sl);
+ }
+
+ /**
+ * Expire the passed server. Remove it from list of online servers
+ */
+ public void expireServerWithLock(final ServerName serverName) {
+ onlineServers.remove(serverName);
Review comment:
Nit: should we log server is getting expired?
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/ReplicationServerManager.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.ServerMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The ServerManager class manages info about replication servers.
+ * <p>
+ * Maintains lists of online and dead servers.
+ * <p>
+ * Servers are distinguished in two different ways. A given server has a
+ * location, specified by hostname and port, and of which there can only be one
+ * online at any given time. A server instance is specified by the location
+ * (hostname and port) as well as the startcode (timestamp from when the server
+ * was started). This is used to differentiate a restarted instance of a given
+ * server from the original instance.
+ */
+@InterfaceAudience.Private
+public class ReplicationServerManager {
Review comment:
Would it make sense to extend `ServerManager`, or maybe both this one and `ServerManager` could implement a common interface? It seems there is a common workflow for server managers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511740716
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##########
@@ -445,7 +418,7 @@ public PeerRegionServerListener(HBaseReplicationEndpoint endpoint) {
@Override
public synchronized void nodeChildrenChanged(String path) {
- if (path.equals(regionServerListNode)) {
+ if (replicationEndpoint.fetchServersUseZk && path.equals(regionServerListNode)) {
Review comment:
So in the new impl, it will always register zk listener but ignored the zk event if not use zk?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-714553561
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 59s | HBASE-24666 passed |
| +1 :green_heart: | compile | 2m 17s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 9m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 0s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 19s | the patch passed |
| +1 :green_heart: | compile | 2m 7s | the patch passed |
| +1 :green_heart: | javac | 2m 7s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 35s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 3s | hbase-protocol-shaded in the patch passed. |
| -1 :x: | unit | 156m 14s | hbase-server in the patch failed. |
| | | 194m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 20c2fdb2c851 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 1.8.0_232 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/testReport/ |
| Max. process+thread count | 3819 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-717049117
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 29s | HBASE-24666 passed |
| +1 :green_heart: | compile | 2m 12s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 7m 21s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 27s | the patch passed |
| +1 :green_heart: | compile | 2m 13s | the patch passed |
| +1 :green_heart: | javac | 2m 13s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 21s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 42s | hbase-server generated 1 new + 83 unchanged - 0 fixed = 84 total (was 83) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 1s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 196m 18s | hbase-server in the patch passed. |
| | | 230m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 6b3effa1ea31 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/testReport/ |
| Max. process+thread count | 3521 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511674844
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/ReplicationServerManager.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.ServerMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The ServerManager class manages info about replication servers.
+ * <p>
+ * Maintains lists of online and dead servers.
+ * <p>
+ * Servers are distinguished in two different ways. A given server has a
+ * location, specified by hostname and port, and of which there can only be one
+ * online at any given time. A server instance is specified by the location
+ * (hostname and port) as well as the startcode (timestamp from when the server
+ * was started). This is used to differentiate a restarted instance of a given
+ * server from the original instance.
+ */
+@InterfaceAudience.Private
+public class ReplicationServerManager {
Review comment:
Yes. But we want do this refator after we finished this, because ReplicationServerManager may have new featrues but ServerManager not need. It is not easy to decide that extend ServerManager or implement a common interface. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511741505
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
##########
@@ -388,4 +446,152 @@ protected ReplicationServerRpcServices createRpcServices() throws IOException {
protected boolean setAbortRequested() {
return abortRequested.compareAndSet(false, true);
}
+
+ protected void tryReplicationServerReport(long reportStartTime, long reportEndTime)
+ throws IOException {
+ ReplicationServerStatusService.BlockingInterface rss = rssStub;
+ if (rss == null) {
+ createReplicationServerStatusStub(true);
+ rss = rssStub;
+ if (rss == null) {
+ return;
+ }
+ }
+ ClusterStatusProtos.ServerLoad sl = buildServerLoad(reportStartTime, reportEndTime);
+ try {
+ RegionServerReportRequest.Builder request = RegionServerReportRequest
+ .newBuilder();
+ request.setServer(ProtobufUtil.toServerName(this.serverName));
+ request.setLoad(sl);
+ rss.replicationServerReport(null, request.build());
+ } catch (ServiceException se) {
+ IOException ioe = ProtobufUtil.getRemoteException(se);
+ if (ioe instanceof YouAreDeadException) {
+ // This will be caught and handled as a fatal error in run()
+ throw ioe;
+ }
+ if (rssStub == rss) {
+ rssStub = null;
+ }
+ // Couldn't connect to the master, get location from zk and reconnect
+ // Method blocks until new master is found or we are stopped
+ createReplicationServerStatusStub(true);
+ }
+ }
+
+ private ClusterStatusProtos.ServerLoad buildServerLoad(long reportStartTime, long reportEndTime) {
+ long usedMemory = -1L;
+ long maxMemory = -1L;
+ final MemoryUsage usage = MemorySizeUtil.safeGetHeapMemoryUsage();
+ if (usage != null) {
+ usedMemory = usage.getUsed();
+ maxMemory = usage.getMax();
+ }
+
+ ClusterStatusProtos.ServerLoad.Builder serverLoad = ClusterStatusProtos.ServerLoad.newBuilder();
+ serverLoad.setTotalNumberOfRequests(rpcServices.requestCount.sum());
+ serverLoad.setUsedHeapMB((int) (usedMemory / 1024 / 1024));
+ serverLoad.setMaxHeapMB((int) (maxMemory / 1024 / 1024));
+
+ serverLoad.setReportStartTime(reportStartTime);
+ serverLoad.setReportEndTime(reportEndTime);
+
+ // for the replicationLoad purpose. Only need to get from one executorService
+ // either source or sink will get the same info
+ ReplicationSinkService sinks = getReplicationSinkService();
+
+ if (sinks != null) {
+ // always refresh first to get the latest value
+ ReplicationLoad rLoad = sinks.refreshAndGetReplicationLoad();
+ if (rLoad != null) {
+ serverLoad.setReplLoadSink(rLoad.getReplicationLoadSink());
+ }
+ }
+ return serverLoad.build();
+ }
+
+ /**
+ * Get the current master from ZooKeeper and open the RPC connection to it. To get a fresh
+ * connection, the current rssStub must be null. Method will block until a master is available.
+ * You can break from this block by requesting the server stop.
+ * @param refresh If true then master address will be read from ZK, otherwise use cached data
+ * @return master + port, or null if server has been stopped
+ */
+ protected synchronized ServerName createReplicationServerStatusStub(boolean refresh) {
Review comment:
Ditto.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511738528
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/ReplicationServerManager.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.ServerMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The ServerManager class manages info about replication servers.
+ * <p>
+ * Maintains lists of online and dead servers.
+ * <p>
+ * Servers are distinguished in two different ways. A given server has a
+ * location, specified by hostname and port, and of which there can only be one
+ * online at any given time. A server instance is specified by the location
+ * (hostname and port) as well as the startcode (timestamp from when the server
+ * was started). This is used to differentiate a restarted instance of a given
+ * server from the original instance.
+ */
+@InterfaceAudience.Private
+public class ReplicationServerManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReplicationServerManager.class);
+
+ public static final String ONLINE_SERVER_REFRESH_INTERVAL =
+ "hbase.master.replication.server.refresh.interval";
+ public static final int ONLINE_SERVER_REFRESH_INTERVAL_DEFAULT = 60 * 1000; // 1 mins
+
+ private final MasterServices master;
+
+ /** Map of registered servers to their current load */
+ private final ConcurrentNavigableMap<ServerName, ServerMetrics> onlineServers =
+ new ConcurrentSkipListMap<>();
+
+ private OnlineServerRefresher onlineServerRefresher;
+ private int refreshPeriod;
+
+ /**
+ * Constructor.
+ */
+ public ReplicationServerManager(final MasterServices master) {
+ this.master = master;
+ }
+
+ /**
+ * start chore in ServerManager
+ */
+ public void startChore() {
+ Configuration conf = master.getConfiguration();
+ refreshPeriod = conf.getInt(ONLINE_SERVER_REFRESH_INTERVAL,
+ ONLINE_SERVER_REFRESH_INTERVAL_DEFAULT);
+ onlineServerRefresher = new OnlineServerRefresher("ReplicationServerRefresher", refreshPeriod);
+ master.getChoreService().scheduleChore(onlineServerRefresher);
+ }
+
+ /**
+ * Stop the ServerManager.
+ */
+ public void stop() {
+ if (onlineServerRefresher != null) {
+ onlineServerRefresher.cancel();
+ }
+ }
+
+ public void serverReport(ServerName sn, ServerMetrics sl) {
+ if (null == this.onlineServers.replace(sn, sl)) {
+ if (!checkAndRecordNewServer(sn, sl)) {
+ LOG.info("ReplicationServerReport ignored, could not record the server: {}", sn);
+ }
+ }
+ }
+
+ /**
+ * Check is a server of same host and port already exists,
+ * if not, or the existed one got a smaller start code, record it.
+ *
+ * @param serverName the server to check and record
+ * @param sl the server load on the server
+ * @return true if the server is recorded, otherwise, false
+ */
+ boolean checkAndRecordNewServer(final ServerName serverName, final ServerMetrics sl) {
Review comment:
private method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-717025458
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 41s | HBASE-24666 passed |
| +1 :green_heart: | compile | 1m 46s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 6m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 29s | the patch passed |
| +1 :green_heart: | compile | 1m 47s | the patch passed |
| +1 :green_heart: | javac | 1m 47s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 45s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 153m 25s | hbase-server in the patch passed. |
| | | 183m 22s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 8cc692c9a3a8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/testReport/ |
| Max. process+thread count | 4113 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ddupg commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
ddupg commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511707449
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/ReplicationServerManager.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.ServerMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The ServerManager class manages info about replication servers.
+ * <p>
+ * Maintains lists of online and dead servers.
+ * <p>
+ * Servers are distinguished in two different ways. A given server has a
+ * location, specified by hostname and port, and of which there can only be one
+ * online at any given time. A server instance is specified by the location
+ * (hostname and port) as well as the startcode (timestamp from when the server
+ * was started). This is used to differentiate a restarted instance of a given
+ * server from the original instance.
+ */
+@InterfaceAudience.Private
+public class ReplicationServerManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReplicationServerManager.class);
+
+ public static final String ONLINE_SERVER_REFRESH_INTERVAL =
+ "hbase.master.replication.server.refresh.interval";
+ public static final int ONLINE_SERVER_REFRESH_INTERVAL_DEFAULT = 60 * 1000; // 1 mins
+
+ private final MasterServices master;
+
+ /** Map of registered servers to their current load */
+ private final ConcurrentNavigableMap<ServerName, ServerMetrics> onlineServers =
+ new ConcurrentSkipListMap<>();
+
+ private OnlineServerRefresher onlineServerRefresher;
+ private int refreshPeriod;
+
+ /**
+ * Constructor.
+ */
+ public ReplicationServerManager(final MasterServices master) {
+ this.master = master;
+ }
+
+ /**
+ * start chore in ServerManager
+ */
+ public void startChore() {
+ Configuration conf = master.getConfiguration();
+ refreshPeriod = conf.getInt(ONLINE_SERVER_REFRESH_INTERVAL,
+ ONLINE_SERVER_REFRESH_INTERVAL_DEFAULT);
+ onlineServerRefresher = new OnlineServerRefresher("ReplicationServerRefresher", refreshPeriod);
+ master.getChoreService().scheduleChore(onlineServerRefresher);
+ }
+
+ /**
+ * Stop the ServerManager.
+ */
+ public void stop() {
+ if (onlineServerRefresher != null) {
+ onlineServerRefresher.cancel();
+ }
+ }
+
+ public void serverReport(ServerName sn, ServerMetrics sl) {
+ if (null == this.onlineServers.replace(sn, sl)) {
+ if (!checkAndRecordNewServer(sn, sl)) {
+ LOG.info("ReplicationServerReport ignored, could not record the server: {}", sn);
+ }
+ }
+ }
+
+ /**
+ * Check is a server of same host and port already exists,
+ * if not, or the existed one got a smaller start code, record it.
+ *
+ * @param serverName the server to check and record
+ * @param sl the server load on the server
+ * @return true if the server is recorded, otherwise, false
+ */
+ boolean checkAndRecordNewServer(final ServerName serverName, final ServerMetrics sl) {
+ ServerName existingServer = null;
+ synchronized (this.onlineServers) {
+ existingServer = findServerWithSameHostnamePortWithLock(serverName);
+ if (existingServer != null && (existingServer.getStartcode() > serverName.getStartcode())) {
+ LOG.info("ReplicationServer serverName={} rejected; we already have {} registered with "
+ + "same hostname and port", serverName, existingServer);
+ return false;
+ }
+ recordNewServerWithLock(serverName, sl);
+ // Note that we assume that same ts means same server, and don't expire in that case.
+ if (existingServer != null && (existingServer.getStartcode() < serverName.getStartcode())) {
+ LOG.info("Triggering server recovery; existingServer {} looks stale, new server: {}",
+ existingServer, serverName);
+ expireServerWithLock(existingServer);
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Assumes onlineServers is locked.
+ * @return ServerName with matching hostname and port.
+ */
+ private ServerName findServerWithSameHostnamePortWithLock(final ServerName serverName) {
+ ServerName end = ServerName.valueOf(serverName.getHostname(), serverName.getPort(),
+ Long.MAX_VALUE);
+
+ ServerName r = onlineServers.lowerKey(end);
+ if (r != null && ServerName.isSameAddress(r, serverName)) {
+ return r;
+ }
+ return null;
+ }
+
+ private void recordNewServerWithLock(final ServerName serverName, final ServerMetrics sl) {
+ LOG.info("Registering ReplicationServer={}", serverName);
+ this.onlineServers.put(serverName, sl);
+ }
+
+ /**
+ * Expire the passed server. Remove it from list of online servers
+ */
+ public void expireServerWithLock(final ServerName serverName) {
+ onlineServers.remove(serverName);
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511738198
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/ReplicationServerManager.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.ServerMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The ServerManager class manages info about replication servers.
Review comment:
ServerManager　=> ReplicationServerManager?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716337940
Add master UI to show ReplicationServer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-714317578
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 37s | HBASE-24666 passed |
| +1 :green_heart: | checkstyle | 1m 20s | HBASE-24666 passed |
| -1 :x: | spotbugs | 1m 58s | hbase-server in HBASE-24666 has 3 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded |
| +1 :green_heart: | checkstyle | 1m 6s | hbase-server: The patch generated 0 new + 147 unchanged - 3 fixed = 147 total (was 150) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | hbaseprotoc | 1m 38s | the patch passed |
| -1 :x: | spotbugs | 2m 9s | hbase-server generated 1 new + 0 unchanged - 3 fixed = 1 total (was 3) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 49m 3s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of org.apache.hadoop.hbase.replication.HReplicationServer.rpcClient; locked 50% of time Unsynchronized access at HReplicationServer.java:50% of time Unsynchronized access at HReplicationServer.java:[line 258] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 50929a93c1ec 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511741036
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##########
@@ -445,7 +418,7 @@ public PeerRegionServerListener(HBaseReplicationEndpoint endpoint) {
@Override
public synchronized void nodeChildrenChanged(String path) {
- if (path.equals(regionServerListNode)) {
+ if (replicationEndpoint.fetchServersUseZk && path.equals(regionServerListNode)) {
Review comment:
Add comment in the class javadoc for this new impl?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ddupg commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
ddupg commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716453008
Thanks @infraio for reviewing. I have fixed as comments.
> Add master UI to show ReplicationServer?
Do it in this PR? I was going to do it in a new PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716341713
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 4m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 24s | HBASE-24666 passed |
| +1 :green_heart: | compile | 1m 42s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 28s | the patch passed |
| +1 :green_heart: | compile | 1m 42s | the patch passed |
| +1 :green_heart: | javac | 1m 42s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 46s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 141m 45s | hbase-server in the patch passed. |
| | | 174m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 64a86fff670e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/testReport/ |
| Max. process+thread count | 4541 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r512361785
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
##########
@@ -388,4 +446,152 @@ protected ReplicationServerRpcServices createRpcServices() throws IOException {
protected boolean setAbortRequested() {
return abortRequested.compareAndSet(false, true);
}
+
+ private void tryReplicationServerReport(long reportStartTime, long reportEndTime)
+ throws IOException {
+ ReplicationServerStatusService.BlockingInterface rss = rssStub;
+ if (rss == null) {
+ createReplicationServerStatusStub(true);
+ rss = rssStub;
+ if (rss == null) {
+ return;
+ }
+ }
+ ClusterStatusProtos.ServerLoad sl = buildServerLoad(reportStartTime, reportEndTime);
+ try {
+ RegionServerReportRequest.Builder request = RegionServerReportRequest
+ .newBuilder();
+ request.setServer(ProtobufUtil.toServerName(this.serverName));
+ request.setLoad(sl);
+ rss.replicationServerReport(null, request.build());
+ } catch (ServiceException se) {
+ IOException ioe = ProtobufUtil.getRemoteException(se);
+ if (ioe instanceof YouAreDeadException) {
+ // This will be caught and handled as a fatal error in run()
+ throw ioe;
+ }
+ if (rssStub == rss) {
+ rssStub = null;
+ }
+ // Couldn't connect to the master, get location from zk and reconnect
+ // Method blocks until new master is found or we are stopped
+ createReplicationServerStatusStub(true);
+ }
+ }
+
+ private ClusterStatusProtos.ServerLoad buildServerLoad(long reportStartTime, long reportEndTime) {
+ long usedMemory = -1L;
+ long maxMemory = -1L;
+ final MemoryUsage usage = MemorySizeUtil.safeGetHeapMemoryUsage();
+ if (usage != null) {
+ usedMemory = usage.getUsed();
+ maxMemory = usage.getMax();
+ }
+
+ ClusterStatusProtos.ServerLoad.Builder serverLoad = ClusterStatusProtos.ServerLoad.newBuilder();
+ serverLoad.setTotalNumberOfRequests(rpcServices.requestCount.sum());
+ serverLoad.setUsedHeapMB((int) (usedMemory / 1024 / 1024));
+ serverLoad.setMaxHeapMB((int) (maxMemory / 1024 / 1024));
+
+ serverLoad.setReportStartTime(reportStartTime);
+ serverLoad.setReportEndTime(reportEndTime);
+
+ // for the replicationLoad purpose. Only need to get from one executorService
+ // either source or sink will get the same info
+ ReplicationSinkService sinks = getReplicationSinkService();
+
+ if (sinks != null) {
+ // always refresh first to get the latest value
+ ReplicationLoad rLoad = sinks.refreshAndGetReplicationLoad();
+ if (rLoad != null) {
+ serverLoad.setReplLoadSink(rLoad.getReplicationLoadSink());
+ }
+ }
+ return serverLoad.build();
+ }
+
+ /**
+ * Get the current master from ZooKeeper and open the RPC connection to it. To get a fresh
+ * connection, the current rssStub must be null. Method will block until a master is available.
+ * You can break from this block by requesting the server stop.
+ * @param refresh If true then master address will be read from ZK, otherwise use cached data
+ * @return master + port, or null if server has been stopped
+ */
+ private synchronized ServerName createReplicationServerStatusStub(boolean refresh) {
Review comment:
The return ServerName never used?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716382359
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 13m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 52s | HBASE-24666 passed |
| +1 :green_heart: | compile | 2m 41s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 8m 52s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 11s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 24s | the patch passed |
| +1 :green_heart: | compile | 2m 20s | the patch passed |
| +1 :green_heart: | javac | 2m 20s | the patch passed |
| +1 :green_heart: | shadedjars | 9m 48s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 198m 59s | hbase-server in the patch passed. |
| | | 253m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux a7955df059b1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 2020-01-14 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/testReport/ |
| Max. process+thread count | 3707 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511741469
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
##########
@@ -388,4 +446,152 @@ protected ReplicationServerRpcServices createRpcServices() throws IOException {
protected boolean setAbortRequested() {
return abortRequested.compareAndSet(false, true);
}
+
+ protected void tryReplicationServerReport(long reportStartTime, long reportEndTime)
Review comment:
private method is enough?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ddupg commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
ddupg commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511707355
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##########
@@ -288,39 +258,26 @@ public boolean isAborted() {
return false;
}
- /**
- * Get the connection to peer cluster
- * @return connection to peer cluster
- * @throws IOException If anything goes wrong connecting
- */
- private synchronized AsyncClusterConnection getPeerConnection() throws IOException {
- if (peerConnection == null) {
- Configuration conf = ctx.getConfiguration();
- peerConnection = ClusterConnectionFactory.createAsyncClusterConnection(conf, null,
- UserProvider.instantiate(conf).getCurrent());
- }
- return peerConnection;
- }
-
/**
* Get the list of all the servers that are responsible for replication sink
* from the specified peer master
* @return list of server addresses or an empty list if the slave is unavailable
Review comment:
Yes, I ignored changing the comment.
Fixed it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-714461718
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 40s | HBASE-24666 passed |
| +1 :green_heart: | checkstyle | 1m 18s | HBASE-24666 passed |
| -1 :x: | spotbugs | 1m 59s | hbase-server in HBASE-24666 has 3 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded |
| +1 :green_heart: | checkstyle | 1m 8s | hbase-server: The patch generated 0 new + 147 unchanged - 3 fixed = 147 total (was 150) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 21s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | hbaseprotoc | 1m 40s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 44s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | spotbugs | 2m 12s | hbase-server generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 49m 30s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 4dd3aa2fcc77 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-714465944
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 33s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 6m 1s | HBASE-24666 passed |
| +1 :green_heart: | compile | 2m 47s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 8m 42s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 43s | the patch passed |
| +1 :green_heart: | compile | 2m 49s | the patch passed |
| +1 :green_heart: | javac | 2m 49s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 41s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 5s | hbase-protocol-shaded in the patch passed. |
| -1 :x: | unit | 14m 13s | hbase-server in the patch failed. |
| | | 57m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux ef043f86cbad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 2020-01-14 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/testReport/ |
| Max. process+thread count | 788 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r512355359
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##########
@@ -3402,4 +3406,33 @@ public ListReplicationSinkServersResponse listReplicationSinkServers(
}
return builder.build();
}
+
+ @Override
+ public RegionServerReportResponse replicationServerReport(RpcController controller,
+ RegionServerReportRequest request) throws ServiceException {
+ try {
+ master.checkServiceStarted();
+ int versionNumber = 0;
+ String version = "0.0.0";
+ VersionInfo versionInfo = VersionInfoUtil.getCurrentClientVersionInfo();
+ if (versionInfo != null) {
+ version = versionInfo.getVersion();
+ versionNumber = VersionInfoUtil.getVersionNumber(versionInfo);
+ }
+ ClusterStatusProtos.ServerLoad sl = request.getLoad();
+ ServerName serverName = ProtobufUtil.toServerName(request.getServer());
+ ServerMetrics oldMetrics = master.getReplicationServerManager().getServerMetrics(serverName);
+ ServerMetrics newMetrics =
+ ServerMetricsBuilder.toServerMetrics(serverName, versionNumber, version, sl);
+ master.getReplicationServerManager().serverReport(serverName, newMetrics);
+ if (sl != null && master.metricsMaster != null) {
+ // Up our metrics.
+ master.metricsMaster.incrementRequests(sl.getTotalNumberOfRequests()
Review comment:
One new question: ReplicationServer should not have request count metric? Because this metric should mean that read metric or write metric?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716533757
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 26s | HBASE-24666 passed |
| +1 :green_heart: | compile | 1m 42s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | the patch passed |
| +1 :green_heart: | compile | 1m 45s | the patch passed |
| +1 :green_heart: | javac | 1m 45s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 45s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 141m 42s | hbase-server in the patch passed. |
| | | 171m 1s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 857effd163cc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/testReport/ |
| Max. process+thread count | 4564 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-714395727
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 41s | HBASE-24666 passed |
| +1 :green_heart: | compile | 1m 44s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 29s | the patch passed |
| +1 :green_heart: | compile | 1m 43s | the patch passed |
| +1 :green_heart: | javac | 1m 43s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 47s | hbase-protocol-shaded in the patch passed. |
| -1 :x: | unit | 149m 16s | hbase-server in the patch failed. |
| | | 179m 16s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux dd4be29af52d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 1.8.0_232 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/testReport/ |
| Max. process+thread count | 4483 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716476519
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 40s | HBASE-24666 passed |
| +1 :green_heart: | checkstyle | 1m 18s | HBASE-24666 passed |
| -1 :x: | spotbugs | 1m 55s | hbase-server in HBASE-24666 has 3 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 22s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded |
| +1 :green_heart: | checkstyle | 1m 7s | hbase-server: The patch generated 0 new + 147 unchanged - 3 fixed = 147 total (was 150) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 24s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | hbaseprotoc | 1m 40s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 40s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | spotbugs | 2m 10s | hbase-server generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 49m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux d554b84d17b2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716902255
> Thanks @infraio for reviewing. I have fixed as comments.
>
> > Add master UI to show ReplicationServer?
>
> Do it in this PR? I was going to do it in a new PR?
Ok. Can be done in follow-on PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716572121
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 9s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 44s | HBASE-24666 passed |
| +1 :green_heart: | compile | 2m 14s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 8m 18s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 38s | the patch passed |
| +1 :green_heart: | compile | 2m 37s | the patch passed |
| +1 :green_heart: | javac | 2m 37s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 0s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 41s | hbase-server generated 1 new + 83 unchanged - 0 fixed = 84 total (was 83) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 2s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | unit | 196m 58s | hbase-server in the patch passed. |
| | | 235m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 3e0525c8a849 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/testReport/ |
| Max. process+thread count | 3270 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-717610489
@ddupg Need to fix the spotbugs warnings: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716974126
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 0s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 24s | HBASE-24666 passed |
| +1 :green_heart: | checkstyle | 1m 18s | HBASE-24666 passed |
| -1 :x: | spotbugs | 1m 58s | hbase-server in HBASE-24666 has 3 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 23s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded |
| +1 :green_heart: | checkstyle | 1m 8s | hbase-server: The patch generated 0 new + 147 unchanged - 3 fixed = 147 total (was 150) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 16m 55s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | hbaseprotoc | 1m 38s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 51s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | spotbugs | 2m 7s | hbase-server generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
| | | 48m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 1ce4f90fc5a4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-716298879
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | prototool | 0m 1s | prototool was not available. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 26s | HBASE-24666 passed |
| +1 :green_heart: | checkstyle | 1m 17s | HBASE-24666 passed |
| -1 :x: | spotbugs | 1m 57s | hbase-server in HBASE-24666 has 3 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 22s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded |
| +1 :green_heart: | checkstyle | 1m 7s | hbase-server: The patch generated 0 new + 147 unchanged - 3 fixed = 147 total (was 150) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 16m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | hbaseprotoc | 1m 41s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 53s | hbase-protocol-shaded in the patch passed. |
| +1 :green_heart: | spotbugs | 2m 10s | hbase-server generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
| | | 48m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 7b97d81a9810 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio merged pull request #2579:
URL: https://github.com/apache/hbase/pull/2579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ddupg commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
ddupg commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r511852023
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##########
@@ -445,7 +418,7 @@ public PeerRegionServerListener(HBaseReplicationEndpoint endpoint) {
@Override
public synchronized void nodeChildrenChanged(String path) {
- if (path.equals(regionServerListNode)) {
+ if (replicationEndpoint.fetchServersUseZk && path.equals(regionServerListNode)) {
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-717856156
> @ddupg Need to fix the spotbugs warnings: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/5/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html
The spotbugs warnings is about HBASE-24666, not about this PR. And this PR fixed thems. "hbase-server generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3)"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#discussion_r512413846
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##########
@@ -3402,4 +3406,33 @@ public ListReplicationSinkServersResponse listReplicationSinkServers(
}
return builder.build();
}
+
+ @Override
+ public RegionServerReportResponse replicationServerReport(RpcController controller,
+ RegionServerReportRequest request) throws ServiceException {
+ try {
+ master.checkServiceStarted();
+ int versionNumber = 0;
+ String version = "0.0.0";
+ VersionInfo versionInfo = VersionInfoUtil.getCurrentClientVersionInfo();
+ if (versionInfo != null) {
+ version = versionInfo.getVersion();
+ versionNumber = VersionInfoUtil.getVersionNumber(versionInfo);
+ }
+ ClusterStatusProtos.ServerLoad sl = request.getLoad();
+ ServerName serverName = ProtobufUtil.toServerName(request.getServer());
+ ServerMetrics oldMetrics = master.getReplicationServerManager().getServerMetrics(serverName);
+ ServerMetrics newMetrics =
+ ServerMetricsBuilder.toServerMetrics(serverName, versionNumber, version, sl);
+ master.getReplicationServerManager().serverReport(serverName, newMetrics);
+ if (sl != null && master.metricsMaster != null) {
+ // Up our metrics.
+ master.metricsMaster.incrementRequests(sl.getTotalNumberOfRequests()
Review comment:
Got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2579: HBASE-24999 Master manages ReplicationServers
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2579:
URL: https://github.com/apache/hbase/pull/2579#issuecomment-714435188
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 55s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ HBASE-24666 Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 51s | HBASE-24666 passed |
| +1 :green_heart: | compile | 2m 20s | HBASE-24666 passed |
| +1 :green_heart: | shadedjars | 7m 53s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | HBASE-24666 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 17s | the patch passed |
| +1 :green_heart: | compile | 2m 34s | the patch passed |
| +1 :green_heart: | javac | 2m 34s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 58s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | hbase-protocol-shaded in the patch passed. |
| -1 :x: | unit | 213m 43s | hbase-server in the patch failed. |
| | | 255m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2579 |
| Optional Tests | unit javac javadoc shadedjars compile |
| uname | Linux 03a38497d885 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-24666 / 1a64f1a0db |
| Default Java | 2020-01-14 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/testReport/ |
| Max. process+thread count | 3419 (vs. ulimit of 30000) |
| modules | C: hbase-protocol-shaded hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2579/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org