You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2020/11/12 18:12:13 UTC
[hbase] branch HBASE-18070.branch-2 updated (e4e7386 -> 77749cf)
This is an automated email from the ASF dual-hosted git repository.
stack pushed a change to branch HBASE-18070.branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git.
omit e4e7386 HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
omit 557be8a HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
omit fcceb3d HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
add 38512e6 HBASE-25252 Move HMaster inner classes out (#2628)
add 7a07dfe HBASE-25254 Rewrite TestMultiLogThreshold to remove the LogDelegate in RSRpcServices (#2631)
add 21b32d9 HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
add 9fe62c3 HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
add ec3f843 HBASE-20598 Upgrade to JRuby 9.2
new 5c4d9dc HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
new eb9bbf4 HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
new 77749cf HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e4e7386)
\
N -- N -- N refs/heads/HBASE-18070.branch-2 (77749cf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop/hbase/security/EncryptionUtil.java | 34 +++--
.../hadoop/hbase/security/TestEncryptionUtil.java | 135 +++++++++++++++--
.../org/apache/hadoop/hbase/io/crypto/Context.java | 4 +-
.../apache/hadoop/hbase/io/crypto/Encryption.java | 159 ++++++++++++++-------
.../java/org/apache/hadoop/hbase/util/DNS.java | 14 +-
hbase-common/src/main/resources/hbase-default.xml | 6 +-
.../src/main/protobuf/Encryption.proto | 1 +
.../hadoop/hbase/rest/TestSecureRESTServer.java | 2 +-
.../org/apache/hadoop/hbase/master/HMaster.java | 123 ++--------------
.../hbase/master/MasterInitializationMonitor.java | 80 +++++++++++
.../hadoop/hbase/master/MasterRedirectServlet.java | 81 +++++++++++
.../hadoop/hbase/regionserver/HRegionServer.java | 35 +++--
.../hadoop/hbase/regionserver/RSRpcServices.java | 38 ++---
.../apache/hadoop/hbase/util/EncryptionTest.java | 15 +-
.../hbase/regionserver/TestEncryptionDisabled.java | 110 ++++++++++++++
.../hbase/regionserver/TestMultiLogThreshold.java | 121 +++++++++-------
.../regionserver/TestRegionServerHostname.java | 40 ++++--
.../hadoop/hbase/util/TestEncryptionTest.java | 30 ++++
hbase-shell/src/main/ruby/jar-bootstrap.rb | 8 ++
pom.xml | 6 +-
20 files changed, 747 insertions(+), 295 deletions(-)
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterInitializationMonitor.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRedirectServlet.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionDisabled.java
[hbase] 02/03: HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stack pushed a commit to branch HBASE-18070.branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit eb9bbf4174da24eb9da921dc8f08dae7ae814d66
Author: Michael Stack <sa...@users.noreply.github.com>
AuthorDate: Fri Oct 2 12:29:18 2020 -0700
HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
* HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS
Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'.
HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta
Table; hbase:meta Table could only do Phase 1 Region Replicas reading
the primary Regions' hfiles. Here we add 'async WAL Replication' to
hbase:meta so Replicas can be more current with the primary's changes.
Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates
all edits to the configured in-cluster endpoint (Defaults to the
RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to
target a different endpoint implementation).
Set hbase.region.replica.replication.catalog.enabled to enable async WAL
Replication for hbase:meta region replicas. Its off by default.
The CatalogReplicationSource for async WAL Replication of hbase:meta does
NOT need to keep up WAL offset or a queue of WALs-to-replicate in the
replication queue store as is done in other ReplicationSource implementations;
the CatalogReplicationSource is for Region Replicas only. General
Replication does not replicate hbase:meta. hbase:meta Region Replicas reset
on crash of the primary replica so there is no need to 'recover'
replication that was running on the crashed server.
Because it so different in operation, the CatalogReplicationSource is bolted
on to the side of the ReplicationSourceManager. It is lazily
instantiated to match the lazy instantiation of the hbase:meta
WALProvider, created and started on the open of the first Region of an
hbase:meta table. Thereafter it stays up till the process dies, even if
all hbase:meta Regions have moved off the server, in case a hbase:meta
Region is moved back (Doing this latter simplifies the implementation)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Read configuration to see if we need to wait on setting a Region read-enabled
(if so, replicas will only flip to enable read after confirming a
flush of the primary so they for sure are a replica of a known point)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
If configured, on open of hbase:meta, ask the ReplicationSourceManager
to add a ReplicationSource (if it hasn't already).
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
Edit log message.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
If configured, on close of hbase:meta, update ReplicationSourceManager
that a source Region has closed.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
javadoc and make constructor private.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
Add logPositionAndCleanOldLogs w/ default of the old behavior so
CatalogReplicationSource can bypass updating store with WAL position,
etc.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
Add creation and start of an CatalogReplicationSource.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
Javadoc.
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
Add utility for reading configurations for hbase:meta region replicas.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
Javadoc.
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
Use define.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
Specical version of ReplicationSource for Region Replicas on hbase:meta.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
Needs a special peer too (peers are baked into replication though we don't use 'peers' here)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java
Tests.
Signed-off-by: Duo Zhang <zh...@apache.org>
Signed-off-by: Huaxiang Sun <hu...@apache.com>
---
.../hadoop/hbase/regionserver/HRegionServer.java | 6 +-
.../regionserver/handler/AssignRegionHandler.java | 15 +-
.../handler/RegionReplicaFlushHandler.java | 6 +-
.../handler/UnassignRegionHandler.java | 12 +-
...Provider.java => CatalogReplicationSource.java} | 27 +-
.../regionserver/CatalogReplicationSourcePeer.java | 49 ++++
.../regionserver/NoopReplicationQueueStorage.java | 140 +++++++++
.../regionserver/RecoveredReplicationSource.java | 2 +-
.../replication/regionserver/Replication.java | 9 +-
.../regionserver/ReplicationSource.java | 32 +--
.../regionserver/ReplicationSourceFactory.java | 9 +-
.../regionserver/ReplicationSourceInterface.java | 22 +-
.../regionserver/ReplicationSourceManager.java | 117 ++++++--
.../regionserver/ReplicationSourceShipper.java | 4 +-
.../regionserver/SerialReplicationChecker.java | 3 +-
.../regionserver/WALFileLengthProvider.java | 3 +
.../hadoop/hbase/util/ServerRegionReplicaUtil.java | 57 +++-
.../org/apache/hadoop/hbase/wal/WALFactory.java | 17 +-
.../hbase/replication/ReplicationSourceDummy.java | 10 +-
.../TestMetaRegionReplicaReplicationEndpoint.java | 319 +++++++++++++++++++++
.../TestRegionReplicaReplicationEndpoint.java | 182 +++++++-----
.../regionserver/TestReplicationSourceManager.java | 2 +-
.../regionserver/TestSerialReplicationChecker.java | 2 +-
23 files changed, 872 insertions(+), 173 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index ca6b9f3..833e49d 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -2473,9 +2473,9 @@ public class HRegionServer extends Thread implements
if (ServerRegionReplicaUtil.isDefaultReplica(region.getRegionInfo())) {
return;
}
- if (!ServerRegionReplicaUtil.isRegionReplicaReplicationEnabled(region.conf) ||
- !ServerRegionReplicaUtil.isRegionReplicaWaitForPrimaryFlushEnabled(
- region.conf)) {
+ TableName tn = region.getTableDescriptor().getTableName();
+ if (!ServerRegionReplicaUtil.isRegionReplicaReplicationEnabled(region.conf, tn) ||
+ !ServerRegionReplicaUtil.isRegionReplicaWaitForPrimaryFlushEnabled(region.conf)) {
region.setReadsEnabled(true);
return;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
index 98d09b2..4ee6efc 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
@@ -20,8 +20,11 @@ package org.apache.hadoop.hbase.regionserver.handler;
import edu.umd.cs.findbugs.annotations.Nullable;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.executor.EventType;
@@ -31,6 +34,7 @@ import org.apache.hadoop.hbase.regionserver.Region;
import org.apache.hadoop.hbase.regionserver.RegionServerServices.PostOpenDeployContext;
import org.apache.hadoop.hbase.regionserver.RegionServerServices.RegionStateTransitionContext;
import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -129,8 +133,15 @@ public class AssignRegionHandler extends EventHandler {
}
// pass null for the last parameter, which used to be a CancelableProgressable, as now the
// opening can not be interrupted by a close request any more.
- region = HRegion.openHRegion(regionInfo, htd, rs.getWAL(regionInfo), rs.getConfiguration(),
- rs, null);
+ Configuration conf = rs.getConfiguration();
+ TableName tn = htd.getTableName();
+ if (ServerRegionReplicaUtil.isMetaRegionReplicaReplicationEnabled(conf, tn)) {
+ if (RegionReplicaUtil.isDefaultReplica(this.regionInfo.getReplicaId())) {
+ // Add the hbase:meta replication source on replica zero/default.
+ rs.getReplicationSourceService().getReplicationManager().addCatalogReplicationSource();
+ }
+ }
+ region = HRegion.openHRegion(regionInfo, htd, rs.getWAL(regionInfo), conf, rs, null);
} catch (IOException e) {
cleanUpAndReportFailure(e);
return;
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
index a20443c..b5204af 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
@@ -109,9 +109,9 @@ public class RegionReplicaFlushHandler extends EventHandler {
RetryCounter counter = new RetryCounterFactory(maxAttempts, (int)pause).create();
if (LOG.isDebugEnabled()) {
- LOG.debug("RPC'ing to primary region replica " +
- ServerRegionReplicaUtil.getRegionInfoForDefaultReplica(region.getRegionInfo()) + " from " +
- region.getRegionInfo() + " to trigger FLUSH");
+ LOG.debug("RPC'ing to primary " + ServerRegionReplicaUtil.
+ getRegionInfoForDefaultReplica(region.getRegionInfo()).getRegionNameAsString() +
+ " from " + region.getRegionInfo().getRegionNameAsString() + " to trigger FLUSH");
}
while (!region.isClosing() && !region.isClosed()
&& !server.isAborted() && !server.isStopped()) {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
index 0bf2543a..1ed74bb 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
import org.apache.hadoop.hbase.executor.EventHandler;
import org.apache.hadoop.hbase.executor.EventType;
import org.apache.hadoop.hbase.regionserver.HRegion;
@@ -30,6 +31,7 @@ import org.apache.hadoop.hbase.regionserver.Region;
import org.apache.hadoop.hbase.regionserver.RegionServerServices.RegionStateTransitionContext;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -113,12 +115,20 @@ public class UnassignRegionHandler extends EventHandler {
if (region.close(abort) == null) {
// XXX: Is this still possible? The old comment says about split, but now split is done at
// master side, so...
- LOG.warn("Can't close {} already closed during close()", regionName);
+ LOG.warn("Can't close {}, already closed during close()", regionName);
rs.getRegionsInTransitionInRS().remove(encodedNameBytes, Boolean.FALSE);
return;
}
rs.removeRegion(region, destination);
+ if (ServerRegionReplicaUtil.isMetaRegionReplicaReplicationEnabled(rs.getConfiguration(),
+ region.getTableDescriptor().getTableName())) {
+ if (RegionReplicaUtil.isDefaultReplica(region.getRegionInfo().getReplicaId())) {
+ // If hbase:meta read replicas enabled, remove replication source for hbase:meta Regions.
+ // See assign region handler where we add the replication source on open.
+ rs.getReplicationSourceService().getReplicationManager().removeCatalogReplicationSource();
+ }
+ }
if (!rs.reportRegionStateTransition(
new RegionStateTransitionContext(TransitionCode.CLOSED, HConstants.NO_SEQNUM, closeProcId,
-1, region.getRegionInfo()))) {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
similarity index 50%
copy from hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
copy to hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
index 010fa69..f36514d 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -17,18 +17,27 @@
*/
package org.apache.hadoop.hbase.replication.regionserver;
-import java.util.OptionalLong;
-
-import org.apache.hadoop.fs.Path;
+import java.util.Collections;
+import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
import org.apache.yetus.audience.InterfaceAudience;
/**
- * Used by replication to prevent replicating unacked log entries. See
- * https://issues.apache.org/jira/browse/HBASE-14004 for more details.
+ * ReplicationSource that reads catalog WAL files -- e.g. hbase:meta WAL files -- and lets through
+ * all WALEdits from these WALs. This ReplicationSource is NOT created via
+ * {@link ReplicationSourceFactory}.
*/
@InterfaceAudience.Private
-@FunctionalInterface
-public interface WALFileLengthProvider {
+class CatalogReplicationSource extends ReplicationSource {
+ CatalogReplicationSource() {
+ // Filters in hbase:meta WAL files and allows all edits, including 'meta' edits (these are
+ // filtered out in the 'super' class default implementation).
+ super(p -> AbstractFSWALProvider.isMetaFile(p), Collections.emptyList());
+ }
- OptionalLong getLogFileSizeIfBeingWritten(Path path);
+ @Override
+ public void logPositionAndCleanOldLogs(WALEntryBatch entryBatch) {
+ // Noop. This implementation does not persist state to backing storage nor does it keep its
+ // WALs in a general map up in ReplicationSourceManager so just skip calling through to the
+ // default implemenentation.
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
new file mode 100644
index 0000000..b853217
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.ReplicationPeerImpl;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * The 'peer' used internally by Catalog Region Replicas Replication Source.
+ * The Replication system has 'peer' baked into its core so though we do not need 'peering', we
+ * need a 'peer' and its configuration else the replication system breaks at a few locales.
+ * Set "hbase.region.replica.catalog.replication" if you want to change the configured endpoint.
+ */
+@InterfaceAudience.Private
+class CatalogReplicationSourcePeer extends ReplicationPeerImpl {
+ /**
+ * @param clusterKey Usually the UUID from zk passed in by caller as a String.
+ */
+ CatalogReplicationSourcePeer(Configuration configuration, String clusterKey, String peerId) {
+ super(configuration, ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER + "_catalog",
+ true,
+ ReplicationPeerConfig.newBuilder().
+ setClusterKey(clusterKey).
+ setReplicationEndpointImpl(
+ configuration.get("hbase.region.replica.catalog.replication",
+ RegionReplicaReplicationEndpoint.class.getName())).
+ setBandwidth(0). // '0' means no bandwidth.
+ setSerial(false).
+ build());
+ }
+}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/NoopReplicationQueueStorage.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/NoopReplicationQueueStorage.java
new file mode 100644
index 0000000..4ad41fc
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/NoopReplicationQueueStorage.java
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedSet;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.replication.ReplicationException;
+import org.apache.hadoop.hbase.replication.ReplicationQueueStorage;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Noop queue storage -- does nothing.
+ */
+@InterfaceAudience.Private
+class NoopReplicationQueueStorage implements ReplicationQueueStorage {
+ NoopReplicationQueueStorage() {}
+
+ @Override
+ public void removeQueue(ServerName serverName, String queueId) throws ReplicationException {}
+
+ @Override
+ public void addWAL(ServerName serverName, String queueId, String fileName)
+ throws ReplicationException {}
+
+ @Override
+ public void removeWAL(ServerName serverName, String queueId, String fileName)
+ throws ReplicationException { }
+
+ @Override
+ public void setWALPosition(ServerName serverName, String queueId, String fileName, long position,
+ Map<String, Long> lastSeqIds) throws ReplicationException {}
+
+ @Override
+ public long getLastSequenceId(String encodedRegionName, String peerId)
+ throws ReplicationException {
+ return 0;
+ }
+
+ @Override
+ public void setLastSequenceIds(String peerId, Map<String, Long> lastSeqIds)
+ throws ReplicationException {}
+
+ @Override
+ public void removeLastSequenceIds(String peerId) throws ReplicationException {}
+
+ @Override
+ public void removeLastSequenceIds(String peerId, List<String> encodedRegionNames)
+ throws ReplicationException {}
+
+ @Override
+ public long getWALPosition(ServerName serverName, String queueId, String fileName)
+ throws ReplicationException {
+ return 0;
+ }
+
+ @Override
+ public List<String> getWALsInQueue(ServerName serverName, String queueId)
+ throws ReplicationException {
+ return Collections.EMPTY_LIST;
+ }
+
+ @Override
+ public List<String> getAllQueues(ServerName serverName) throws ReplicationException {
+ return Collections.EMPTY_LIST;
+ }
+
+ @Override
+ public Pair<String, SortedSet<String>> claimQueue(ServerName sourceServerName, String queueId,
+ ServerName destServerName) throws ReplicationException {
+ return null;
+ }
+
+ @Override
+ public void removeReplicatorIfQueueIsEmpty(ServerName serverName)
+ throws ReplicationException {}
+
+ @Override
+ public List<ServerName> getListOfReplicators() throws ReplicationException {
+ return Collections.EMPTY_LIST;
+ }
+
+ @Override
+ public Set<String> getAllWALs() throws ReplicationException {
+ return Collections.EMPTY_SET;
+ }
+
+ @Override
+ public void addPeerToHFileRefs(String peerId) throws ReplicationException {}
+
+ @Override
+ public void removePeerFromHFileRefs(String peerId) throws ReplicationException {}
+
+ @Override
+ public void addHFileRefs(String peerId, List<Pair<Path, Path>> pairs)
+ throws ReplicationException {}
+
+ @Override
+ public void removeHFileRefs(String peerId, List<String> files) throws ReplicationException {}
+
+ @Override
+ public List<String> getAllPeersFromHFileRefsQueue() throws ReplicationException {
+ return Collections.EMPTY_LIST;
+ }
+
+ @Override
+ public List<String> getReplicableHFiles(String peerId) throws ReplicationException {
+ return Collections.EMPTY_LIST;
+ }
+
+ @Override
+ public Set<String> getAllHFileRefs() throws ReplicationException {
+ return Collections.EMPTY_SET;
+ }
+
+ @Override
+ public String getRsNode(ServerName serverName) {
+ return null;
+ }
+}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
index f9645dc..39c4beb 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
index 136dcd4..4ca2c56 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
@@ -20,7 +20,6 @@ package org.apache.hadoop.hbase.replication.regionserver;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
-import java.util.OptionalLong;
import java.util.UUID;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
@@ -124,11 +123,11 @@ public class Replication implements ReplicationSourceService, ReplicationSinkSer
}
this.globalMetricsSource = CompatibilitySingletonFactory
.getInstance(MetricsReplicationSourceFactory.class).getGlobalSource();
- WALProvider walProvider = walFactory.getWALProvider();
this.replicationManager = new ReplicationSourceManager(queueStorage, replicationPeers,
- replicationTracker, conf, this.server, fs, logDir, oldLogDir, clusterId,
- walProvider != null ? walProvider.getWALFileLengthProvider() : p -> OptionalLong.empty(),
- globalMetricsSource);
+ replicationTracker, conf, this.server, fs, logDir, oldLogDir, clusterId, walFactory,
+ globalMetricsSource);
+ // Get the user-space WAL provider
+ WALProvider walProvider = walFactory != null? walFactory.getWALProvider(): null;
if (walProvider != null) {
walProvider
.addWALActionsListener(new ReplicationSourceWALActionListener(conf, replicationManager));
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index 645c14d..f30ab29 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
@@ -65,10 +65,10 @@ import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
import org.apache.hadoop.hbase.wal.WAL.Entry;
-import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
/**
@@ -227,6 +227,7 @@ public class ReplicationSource implements ReplicationSourceInterface {
this.peerId = this.replicationQueueInfo.getPeerId();
this.logQueueWarnThreshold = this.conf.getInt("replication.source.log.queue.warn", 2);
+ // A defaultBandwidth of '0' means no bandwidth; i.e. no throttling.
defaultBandwidth = this.conf.getLong("replication.source.per.peer.node.bandwidth", 0);
currentBandwidth = getCurrentBandwidth();
this.throttler = new ReplicationThrottler((double) currentBandwidth / 10.0);
@@ -378,16 +379,10 @@ public class ReplicationSource implements ReplicationSourceInterface {
private void tryStartNewShipper(String walGroupId, PriorityBlockingQueue<Path> queue) {
workerThreads.compute(walGroupId, (key, value) -> {
if (value != null) {
- if (LOG.isDebugEnabled()) {
- LOG.debug(
- "{} Someone has beat us to start a worker thread for wal group {}",
- logPeerId(), key);
- }
+ LOG.debug("{} preempted start of shipping worker walGroupId={}", logPeerId(), walGroupId);
return value;
} else {
- if (LOG.isDebugEnabled()) {
- LOG.debug("{} Starting up worker for wal group {}", logPeerId(), key);
- }
+ LOG.debug("{} starting shipping worker for walGroupId={}", logPeerId(), walGroupId);
ReplicationSourceShipper worker = createNewShipper(walGroupId, queue);
ReplicationSourceWALReader walReader =
createNewWALReader(walGroupId, queue, worker.getStartPosition());
@@ -532,7 +527,7 @@ public class ReplicationSource implements ReplicationSourceInterface {
private long getCurrentBandwidth() {
long peerBandwidth = replicationPeer.getPeerBandwidth();
- // user can set peer bandwidth to 0 to use default bandwidth
+ // User can set peer bandwidth to 0 to use default bandwidth.
return peerBandwidth != 0 ? peerBandwidth : defaultBandwidth;
}
@@ -627,11 +622,11 @@ public class ReplicationSource implements ReplicationSourceInterface {
this.startupOngoing.set(false);
throw new IllegalStateException("Source should be active.");
}
- LOG.info("{} queueId={} is replicating from cluster={} to cluster={}",
- logPeerId(), this.replicationQueueInfo.getQueueId(), clusterId, peerClusterId);
-
+ LOG.info("{} queueId={} (queues={}) is replicating from cluster={} to cluster={}",
+ logPeerId(), this.replicationQueueInfo.getQueueId(), this.queues.size(), clusterId,
+ peerClusterId);
initializeWALEntryFilter(peerClusterId);
- // start workers
+ // Start workers
for (Map.Entry<String, PriorityBlockingQueue<Path>> entry : queues.entrySet()) {
String walGroupId = entry.getKey();
PriorityBlockingQueue<Path> queue = entry.getValue();
@@ -641,11 +636,10 @@ public class ReplicationSource implements ReplicationSourceInterface {
}
@Override
- public void startup() {
+ public ReplicationSourceInterface startup() {
if (this.sourceRunning) {
- return;
+ return this;
}
- // Mark we are running now
this.sourceRunning = true;
startupOngoing.set(true);
initThread = new Thread(this::initialize);
@@ -673,6 +667,7 @@ public class ReplicationSource implements ReplicationSourceInterface {
}
} while ((this.startupOngoing.get() || this.retryStartup.get()) && !this.abortOnError);
});
+ return this;
}
@Override
@@ -851,7 +846,8 @@ public class ReplicationSource implements ReplicationSourceInterface {
return server;
}
- ReplicationQueueStorage getQueueStorage() {
+ @Override
+ public ReplicationQueueStorage getReplicationQueueStorage() {
return queueStorage;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
index d613049..8863f14 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
@@ -19,19 +19,22 @@
package org.apache.hadoop.hbase.replication.regionserver;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.replication.ReplicationQueueInfo;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.hadoop.hbase.replication.ReplicationQueueInfo;
/**
* Constructs a {@link ReplicationSourceInterface}
+ * Note, not used to create specialized ReplicationSources
+ * @see CatalogReplicationSource
*/
@InterfaceAudience.Private
-public class ReplicationSourceFactory {
-
+public final class ReplicationSourceFactory {
private static final Logger LOG = LoggerFactory.getLogger(ReplicationSourceFactory.class);
+ private ReplicationSourceFactory() {}
+
static ReplicationSourceInterface create(Configuration conf, String queueId) {
ReplicationQueueInfo replicationQueueInfo = new ReplicationQueueInfo(queueId);
boolean isQueueRecovered = replicationQueueInfo.isQueueRecovered();
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
index d287acb..ca68f9a 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
@@ -23,7 +23,6 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@ -43,7 +42,6 @@ import org.apache.yetus.audience.InterfaceAudience;
*/
@InterfaceAudience.Private
public interface ReplicationSourceInterface {
-
/**
* Initializer for the source
* @param conf the configuration to use
@@ -76,7 +74,7 @@ public interface ReplicationSourceInterface {
/**
* Start the replication
*/
- void startup();
+ ReplicationSourceInterface startup();
/**
* End the replication
@@ -159,7 +157,6 @@ public interface ReplicationSourceInterface {
/**
* Try to throttle when the peer config with a bandwidth
* @param batchSize entries size will be pushed
- * @throws InterruptedException
*/
void tryThrottle(int batchSize) throws InterruptedException;
@@ -191,4 +188,21 @@ public interface ReplicationSourceInterface {
default boolean isRecovered() {
return false;
}
+
+ /**
+ * @return The instance of queueStorage used by this ReplicationSource.
+ */
+ ReplicationQueueStorage getReplicationQueueStorage();
+
+ /**
+ * Log the current position to storage. Also clean old logs from the replication queue.
+ * Use to bypass the default call to
+ * {@link ReplicationSourceManager#logPositionAndCleanOldLogs(ReplicationSourceInterface,
+ * WALEntryBatch)} whem implementation does not need to persist state to backing storage.
+ * @param entryBatch the wal entry batch we just shipped
+ * @return The instance of queueStorage used by this ReplicationSource.
+ */
+ default void logPositionAndCleanOldLogs(WALEntryBatch entryBatch) {
+ getSourceManager().logPositionAndCleanOldLogs(this, entryBatch);
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
index 4c4b0de..93730d4 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -26,6 +26,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.NavigableSet;
+import java.util.OptionalLong;
import java.util.Set;
import java.util.SortedSet;
import java.util.TreeSet;
@@ -39,6 +40,7 @@ import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
@@ -48,6 +50,7 @@ import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.regionserver.wal.WALActionsListener;
import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationListener;
import org.apache.hadoop.hbase.replication.ReplicationPeer;
@@ -58,12 +61,14 @@ import org.apache.hadoop.hbase.replication.ReplicationQueueInfo;
import org.apache.hadoop.hbase.replication.ReplicationQueueStorage;
import org.apache.hadoop.hbase.replication.ReplicationTracker;
import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
+import org.apache.hadoop.hbase.wal.WALFactory;
+import org.apache.hadoop.hbase.wal.WALProvider;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
@@ -120,7 +125,15 @@ public class ReplicationSourceManager implements ReplicationListener {
private final ConcurrentMap<String, ReplicationSourceInterface> sources;
// List of all the sources we got from died RSs
private final List<ReplicationSourceInterface> oldsources;
+
+ /**
+ * Storage for queues that need persistance; e.g. Replication state so can be recovered
+ * after a crash. queueStorage upkeep is spread about this class and passed
+ * to ReplicationSource instances for these to do updates themselves. Not all ReplicationSource
+ * instances keep state.
+ */
private final ReplicationQueueStorage queueStorage;
+
private final ReplicationTracker replicationTracker;
private final ReplicationPeers replicationPeers;
// UUID for this cluster
@@ -145,7 +158,7 @@ public class ReplicationSourceManager implements ReplicationListener {
private final Path logDir;
// Path to the wal archive
private final Path oldLogDir;
- private final WALFileLengthProvider walFileLengthProvider;
+ private final WALFactory walFactory;
// The number of ms that we wait before moving znodes, HBASE-3596
private final long sleepBeforeFailover;
// Homemade executer service for replication
@@ -160,21 +173,29 @@ public class ReplicationSourceManager implements ReplicationListener {
private final MetricsReplicationGlobalSourceSource globalMetrics;
/**
+ * A special ReplicationSource for hbase:meta Region Read Replicas.
+ * Usually this reference remains empty. If an hbase:meta Region is opened on this server, we
+ * will create an instance of a hbase:meta CatalogReplicationSource and it will live the life of
+ * the Server thereafter; i.e. we will not shut it down even if the hbase:meta moves away from
+ * this server (in case it later gets moved back). We synchronize on this instance testing for
+ * presence and if absent, while creating so only created and started once.
+ */
+ @VisibleForTesting
+ AtomicReference<ReplicationSourceInterface> catalogReplicationSource = new AtomicReference<>();
+
+ /**
* Creates a replication manager and sets the watch on all the other registered region servers
* @param queueStorage the interface for manipulating replication queues
- * @param replicationPeers
- * @param replicationTracker
* @param conf the configuration to use
* @param server the server for this region server
* @param fs the file system to use
* @param logDir the directory that contains all wal directories of live RSs
* @param oldLogDir the directory where old logs are archived
- * @param clusterId
*/
public ReplicationSourceManager(ReplicationQueueStorage queueStorage,
ReplicationPeers replicationPeers, ReplicationTracker replicationTracker, Configuration conf,
Server server, FileSystem fs, Path logDir, Path oldLogDir, UUID clusterId,
- WALFileLengthProvider walFileLengthProvider,
+ WALFactory walFactory,
MetricsReplicationGlobalSourceSource globalMetrics) throws IOException {
// CopyOnWriteArrayList is thread-safe.
// Generally, reading is more than modifying.
@@ -193,7 +214,7 @@ public class ReplicationSourceManager implements ReplicationListener {
this.sleepBeforeFailover = conf.getLong("replication.sleep.before.failover", 30000); // 30
// seconds
this.clusterId = clusterId;
- this.walFileLengthProvider = walFileLengthProvider;
+ this.walFactory = walFactory;
this.replicationTracker.registerListener(this);
// It's preferable to failover 1 RS at a time, but with good zk servers
// more could be processed at the same time.
@@ -320,18 +341,21 @@ public class ReplicationSourceManager implements ReplicationListener {
}
/**
- * Factory method to create a replication source
- * @param queueId the id of the replication queue
- * @return the created source
+ * @return a new 'classic' user-space replication source.
+ * @param queueId the id of the replication queue to associate the ReplicationSource with.
+ * @see #createCatalogReplicationSource() for creating a ReplicationSource for hbase:meta.
*/
private ReplicationSourceInterface createSource(String queueId, ReplicationPeer replicationPeer)
throws IOException {
ReplicationSourceInterface src = ReplicationSourceFactory.create(conf, queueId);
-
- MetricsSource metrics = new MetricsSource(queueId);
- // init replication source
+ // Init the just created replication source. Pass the default walProvider's wal file length
+ // provider. Presumption is we replicate user-space Tables only. For hbase:meta region replica
+ // replication, see #createCatalogReplicationSource().
+ WALFileLengthProvider walFileLengthProvider =
+ this.walFactory.getWALProvider() != null?
+ this.walFactory.getWALProvider().getWALFileLengthProvider() : p -> OptionalLong.empty();
src.init(conf, fs, this, queueStorage, replicationPeer, server, queueId, clusterId,
- walFileLengthProvider, metrics);
+ walFileLengthProvider, new MetricsSource(queueId));
return src;
}
@@ -518,17 +542,16 @@ public class ReplicationSourceManager implements ReplicationListener {
/**
* This method will log the current position to storage. And also clean old logs from the
* replication queue.
- * @param queueId id of the replication queue
- * @param queueRecovered indicates if this queue comes from another region server
* @param entryBatch the wal entry batch we just shipped
*/
- public void logPositionAndCleanOldLogs(String queueId, boolean queueRecovered,
+ public void logPositionAndCleanOldLogs(ReplicationSourceInterface source,
WALEntryBatch entryBatch) {
String fileName = entryBatch.getLastWalPath().getName();
+ String queueId = source.getQueueId();
interruptOrAbortWhenFail(() -> this.queueStorage
.setWALPosition(server.getServerName(), queueId, fileName, entryBatch.getLastWalPosition(),
entryBatch.getLastSeqIds()));
- cleanOldLogs(fileName, entryBatch.isEndOfFile(), queueId, queueRecovered);
+ cleanOldLogs(fileName, entryBatch.isEndOfFile(), queueId, source.isRecovered());
}
/**
@@ -959,4 +982,60 @@ public class ReplicationSourceManager implements ReplicationListener {
MetricsReplicationGlobalSourceSource getGlobalMetrics() {
return this.globalMetrics;
}
+
+ /**
+ * Add an hbase:meta Catalog replication source. Called on open of an hbase:meta Region.
+ * @see #removeCatalogReplicationSource()
+ */
+ public ReplicationSourceInterface addCatalogReplicationSource() throws IOException {
+ // Open/Create the hbase:meta ReplicationSource once only.
+ synchronized (this.catalogReplicationSource) {
+ ReplicationSourceInterface rs = this.catalogReplicationSource.get();
+ return rs != null ? rs :
+ this.catalogReplicationSource.getAndSet(createCatalogReplicationSource());
+ }
+ }
+
+ /**
+ * Remove the hbase:meta Catalog replication source.
+ * Called when we close hbase:meta.
+ * @see #addCatalogReplicationSource()
+ */
+ public void removeCatalogReplicationSource() {
+ // Nothing to do. Leave any CatalogReplicationSource in place in case an hbase:meta Region
+ // comes back to this server.
+ }
+
+ /**
+ * Create, initialize, and start the Catalog ReplicationSource.
+ */
+ private ReplicationSourceInterface createCatalogReplicationSource() throws IOException {
+ // Has the hbase:meta WALProvider been instantiated?
+ WALProvider walProvider = this.walFactory.getMetaWALProvider();
+ boolean addListener = false;
+ if (walProvider == null) {
+ // The meta walProvider has not been instantiated. Create it.
+ walProvider = this.walFactory.getMetaProvider();
+ addListener = true;
+ }
+ CatalogReplicationSourcePeer peer = new CatalogReplicationSourcePeer(this.conf,
+ this.clusterId.toString(), "meta_" + ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER);
+ final ReplicationSourceInterface crs = new CatalogReplicationSource();
+ crs.init(conf, fs, this, new NoopReplicationQueueStorage(), peer, server, peer.getId(),
+ clusterId, walProvider.getWALFileLengthProvider(), new MetricsSource(peer.getId()));
+ if (addListener) {
+ walProvider.addWALActionsListener(new WALActionsListener() {
+ @Override
+ public void postLogRoll(Path oldPath, Path newPath) throws IOException {
+ crs.enqueueLog(newPath);
+ }
+ });
+ } else {
+ // This is a problem. We'll have a ReplicationSource but no listener on hbase:meta WALs
+ // so nothing will be replicated.
+ LOG.error("Did not install WALActionsListener creating CatalogReplicationSource!");
+ }
+ // Start this ReplicationSource.
+ return crs.startup();
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
index b171eb4..f0202ef 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
@@ -33,7 +33,6 @@ import org.apache.hadoop.hbase.wal.WALEdit;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-
import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.BulkLoadDescriptor;
import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.StoreDescriptor;
@@ -260,8 +259,7 @@ public class ReplicationSourceShipper extends Thread {
// position and the file will be removed soon in cleanOldLogs.
if (batch.isEndOfFile() || !batch.getLastWalPath().equals(currentPath) ||
batch.getLastWalPosition() != currentPosition) {
- source.getSourceManager().logPositionAndCleanOldLogs(source.getQueueId(),
- source.isRecovered(), batch);
+ source.logPositionAndCleanOldLogs(batch);
updated = true;
}
// if end of file is true, then we can just skip to the next file in queue.
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SerialReplicationChecker.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SerialReplicationChecker.java
index 321bbb4..4dd6611 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SerialReplicationChecker.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SerialReplicationChecker.java
@@ -37,7 +37,6 @@ import org.apache.hadoop.hbase.wal.WAL.Entry;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-
import org.apache.hbase.thirdparty.com.google.common.cache.Cache;
import org.apache.hbase.thirdparty.com.google.common.cache.CacheBuilder;
import org.apache.hbase.thirdparty.com.google.common.cache.CacheLoader;
@@ -140,7 +139,7 @@ class SerialReplicationChecker {
public SerialReplicationChecker(Configuration conf, ReplicationSource source) {
this.peerId = source.getPeerId();
- this.storage = source.getQueueStorage();
+ this.storage = source.getReplicationQueueStorage();
this.conn = source.getServer().getConnection();
this.waitTimeMs =
conf.getLong(REPLICATION_SERIALLY_WAITING_KEY, REPLICATION_SERIALLY_WAITING_DEFAULT);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
index 010fa69..c60faa9 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
@@ -25,6 +25,9 @@ import org.apache.yetus.audience.InterfaceAudience;
/**
* Used by replication to prevent replicating unacked log entries. See
* https://issues.apache.org/jira/browse/HBASE-14004 for more details.
+ * WALFileLengthProvider exists because we do not want to reference WALFactory and WALProvider
+ * directly in the replication code so in the future it will be easier to decouple them.
+ * Each walProvider will have its own implementation.
*/
@InterfaceAudience.Private
@FunctionalInterface
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
index fbd8d30..5583a47 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -22,6 +22,8 @@ import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.ReplicationPeerNotFoundException;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.client.RegionReplicaUtil;
@@ -57,7 +59,15 @@ public class ServerRegionReplicaUtil extends RegionReplicaUtil {
public static final String REGION_REPLICA_REPLICATION_CONF_KEY
= "hbase.region.replica.replication.enabled";
private static final boolean DEFAULT_REGION_REPLICA_REPLICATION = false;
- private static final String REGION_REPLICA_REPLICATION_PEER = "region_replica_replication";
+ public static final String REGION_REPLICA_REPLICATION_PEER = "region_replica_replication";
+
+ /**
+ * Same as for {@link #REGION_REPLICA_REPLICATION_CONF_KEY} but for catalog replication.
+ */
+ public static final String REGION_REPLICA_REPLICATION_CATALOG_CONF_KEY
+ = "hbase.region.replica.replication.catalog.enabled";
+ private static final boolean DEFAULT_REGION_REPLICA_REPLICATION_CATALOG = false;
+
/**
* Enables or disables refreshing store files of secondary region replicas when the memory is
@@ -116,7 +126,6 @@ public class ServerRegionReplicaUtil extends RegionReplicaUtil {
* files of the primary region, so an HFileLink is used to construct the StoreFileInfo. This
* way ensures that the secondary will be able to continue reading the store files even if
* they are moved to archive after compaction
- * @throws IOException
*/
public static StoreFileInfo getStoreFileInfo(Configuration conf, FileSystem fs,
RegionInfo regionInfo, RegionInfo regionInfoForFs, String familyName, Path path)
@@ -153,8 +162,7 @@ public class ServerRegionReplicaUtil extends RegionReplicaUtil {
}
/**
- * Create replication peer for replicating to region replicas if needed.
- * <p/>
+ * Create replication peer for replicating user-space Region Read Replicas.
* This methods should only be called at master side.
*/
public static void setupRegionReplicaReplication(MasterServices services)
@@ -174,16 +182,42 @@ public class ServerRegionReplicaUtil extends RegionReplicaUtil {
services.addReplicationPeer(REGION_REPLICA_REPLICATION_PEER, peerConfig, true);
}
- public static boolean isRegionReplicaReplicationEnabled(Configuration conf) {
- return conf.getBoolean(REGION_REPLICA_REPLICATION_CONF_KEY,
- DEFAULT_REGION_REPLICA_REPLICATION);
+ /**
+ * @return True if Region Read Replica is enabled for <code>tn</code> (whether hbase:meta or
+ * user-space tables).
+ */
+ public static boolean isRegionReplicaReplicationEnabled(Configuration conf, TableName tn) {
+ return isMetaRegionReplicaReplicationEnabled(conf, tn) ||
+ isRegionReplicaReplicationEnabled(conf);
+ }
+
+ /**
+ * @return True if Region Read Replica is enabled for user-space tables.
+ */
+ private static boolean isRegionReplicaReplicationEnabled(Configuration conf) {
+ return conf.getBoolean(REGION_REPLICA_REPLICATION_CONF_KEY, DEFAULT_REGION_REPLICA_REPLICATION);
}
+ /**
+ * @return True if hbase:meta Region Read Replica is enabled.
+ */
+ public static boolean isMetaRegionReplicaReplicationEnabled(Configuration conf, TableName tn) {
+ return TableName.isMetaTableName(tn) &&
+ conf.getBoolean(REGION_REPLICA_REPLICATION_CATALOG_CONF_KEY,
+ DEFAULT_REGION_REPLICA_REPLICATION_CATALOG);
+ }
+
+ /**
+ * @return True if wait for primary to flush is enabled for user-space tables.
+ */
public static boolean isRegionReplicaWaitForPrimaryFlushEnabled(Configuration conf) {
return conf.getBoolean(REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH_CONF_KEY,
DEFAULT_REGION_REPLICA_WAIT_FOR_PRIMARY_FLUSH);
}
+ /**
+ * @return True if we are to refresh user-space hfiles in Region Read Replicas.
+ */
public static boolean isRegionReplicaStoreFileRefreshEnabled(Configuration conf) {
return conf.getBoolean(REGION_REPLICA_STORE_FILE_REFRESH,
DEFAULT_REGION_REPLICA_STORE_FILE_REFRESH);
@@ -194,11 +228,4 @@ public class ServerRegionReplicaUtil extends RegionReplicaUtil {
DEFAULT_REGION_REPLICA_STORE_FILE_REFRESH_MEMSTORE_MULTIPLIER);
}
- /**
- * Return the peer id used for replicating to secondary region replicas
- */
- public static String getReplicationPeerId() {
- return REGION_REPLICA_REPLICATION_PEER;
- }
-
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
index 3b7f311..7b8b3a0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -67,7 +67,7 @@ public class WALFactory {
/**
* Maps between configuration names for providers and implementation classes.
*/
- static enum Providers {
+ enum Providers {
defaultProvider(AsyncFSWALProvider.class),
filesystem(FSHLogProvider.class),
multiwal(RegionGroupingProvider.class),
@@ -256,8 +256,12 @@ public class WALFactory {
return provider.getWALs();
}
- @VisibleForTesting
- WALProvider getMetaProvider() throws IOException {
+ /**
+ * Called when we lazily create a hbase:meta WAL OR from ReplicationSourceManager ahead of
+ * creating the first hbase:meta WAL so we can register a listener.
+ * @see #getMetaWALProvider()
+ */
+ public WALProvider getMetaProvider() throws IOException {
for (;;) {
WALProvider provider = this.metaProvider.get();
if (provider != null) {
@@ -306,7 +310,6 @@ public class WALFactory {
* to reopen it multiple times, use {@link WAL.Reader#reset()} instead of this method
* then just seek back to the last known good position.
* @return A WAL reader. Close when done with it.
- * @throws IOException
*/
public Reader createReader(final FileSystem fs, final Path path,
CancelableProgressable reporter) throws IOException {
@@ -490,6 +493,10 @@ public class WALFactory {
return this.provider;
}
+ /**
+ * @return Current metaProvider... may be null if not yet initialized.
+ * @see #getMetaProvider()
+ */
public final WALProvider getMetaWALProvider() {
return this.metaProvider.get();
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
index 305a818..884dad9 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -70,8 +70,9 @@ public class ReplicationSourceDummy implements ReplicationSourceInterface {
}
@Override
- public void startup() {
+ public ReplicationSourceInterface startup() {
startup.set(true);
+ return this;
}
public boolean isStartup() {
@@ -160,4 +161,9 @@ public class ReplicationSourceDummy implements ReplicationSourceInterface {
public ServerName getServerWALsBelongTo() {
return null;
}
+
+ @Override
+ public ReplicationQueueStorage getReplicationQueueStorage() {
+ return null;
+ }
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
new file mode 100644
index 0000000..34432e4
--- /dev/null
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellScanner;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.MiniHBaseCluster;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.Waiter;
+import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.regionserver.HRegion;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tests RegionReplicaReplicationEndpoint class for hbase:meta by setting up region replicas and
+ * verifying async wal replication replays the edits to the secondary region in various scenarios.
+ * @see TestRegionReplicaReplicationEndpoint
+ */
+@Category({LargeTests.class})
+public class TestMetaRegionReplicaReplicationEndpoint {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestMetaRegionReplicaReplicationEndpoint.class);
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestMetaRegionReplicaReplicationEndpoint.class);
+ private static final int NB_SERVERS = 3;
+ private static final HBaseTestingUtility HTU = new HBaseTestingUtility();
+
+ @Rule
+ public TestName name = new TestName();
+
+ @Before
+ public void before() throws Exception {
+ Configuration conf = HTU.getConfiguration();
+ conf.setFloat("hbase.regionserver.logroll.multiplier", 0.0003f);
+ conf.setInt("replication.source.size.capacity", 10240);
+ conf.setLong("replication.source.sleepforretries", 100);
+ conf.setInt("hbase.regionserver.maxlogs", 10);
+ conf.setLong("hbase.master.logcleaner.ttl", 10);
+ conf.setInt("zookeeper.recovery.retry", 1);
+ conf.setInt("zookeeper.recovery.retry.intervalmill", 10);
+ conf.setLong(HConstants.THREAD_WAKE_FREQUENCY, 100);
+ conf.setInt("replication.stats.thread.period.seconds", 5);
+ conf.setBoolean("hbase.tests.use.shortcircuit.reads", false);
+ conf.setInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER, 1);
+ // Enable hbase:meta replication.
+ conf.setBoolean(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_CATALOG_CONF_KEY, true);
+ // Set hbase:meta replicas to be 3.
+ conf.setInt(HConstants.META_REPLICAS_NUM, NB_SERVERS);
+ HTU.startMiniCluster(NB_SERVERS);
+ HTU.waitFor(30000,
+ () -> HTU.getMiniHBaseCluster().getRegions(TableName.META_TABLE_NAME).size() >= NB_SERVERS);
+ }
+
+ @After
+ public void after() throws Exception {
+ HTU.shutdownMiniCluster();
+ }
+
+ /**
+ * Assert that the ReplicationSource for hbase:meta gets created when hbase:meta is opened.
+ */
+ @Test
+ public void testHBaseMetaReplicationSourceCreatedOnOpen()
+ throws IOException, InterruptedException {
+ MiniHBaseCluster cluster = HTU.getMiniHBaseCluster();
+ HRegionServer hrs = cluster.getRegionServer(cluster.getServerHoldingMeta());
+ assertTrue(isMetaRegionReplicaReplicationSource(hrs));
+ // Now move the hbase:meta and make sure the ReplicationSoruce is in both places.
+ HRegionServer hrsOther = null;
+ for (int i = 0; i < cluster.getNumLiveRegionServers(); i++) {
+ hrsOther = cluster.getRegionServer(i);
+ if (hrsOther.getServerName().equals(hrs.getServerName())) {
+ hrsOther = null;
+ continue;
+ }
+ break;
+ }
+ assertNotNull(hrsOther);
+ assertFalse(isMetaRegionReplicaReplicationSource(hrsOther));
+ Region meta = null;
+ for (Region region: hrs.getOnlineRegionsLocalContext()) {
+ if (region.getRegionInfo().isMetaRegion()) {
+ meta = region;
+ break;
+ }
+ }
+ assertNotNull(meta);
+ HTU.moveRegionAndWait(meta.getRegionInfo(), hrsOther.getServerName());
+ // Assert that there is a ReplicationSource in both places now.
+ assertTrue(isMetaRegionReplicaReplicationSource(hrs));
+ assertTrue(isMetaRegionReplicaReplicationSource(hrsOther));
+ }
+
+ /**
+ * @return Whether the special meta region replica peer is enabled on <code>hrs</code>
+ */
+ private boolean isMetaRegionReplicaReplicationSource(HRegionServer hrs) {
+ return hrs.getReplicationSourceService().getReplicationManager().
+ catalogReplicationSource.get() != null;
+ }
+
+ /**
+ * Test meta region replica replication. Create some tables and see if replicas pick up the
+ * additions.
+ */
+ @Test
+ public void testHBaseMetaReplicates() throws Exception {
+ try (Table table = HTU.createTable(TableName.valueOf(this.name.getMethodName() + "_0"),
+ HConstants.CATALOG_FAMILY,
+ Arrays.copyOfRange(HBaseTestingUtility.KEYS, 1, HBaseTestingUtility.KEYS.length))) {
+ verifyReplication(TableName.META_TABLE_NAME, NB_SERVERS, getMetaCells(table.getName()));
+ }
+ try (Table table = HTU.createTable(TableName.valueOf(this.name.getMethodName() + "_1"),
+ HConstants.CATALOG_FAMILY,
+ Arrays.copyOfRange(HBaseTestingUtility.KEYS, 1, HBaseTestingUtility.KEYS.length))) {
+ verifyReplication(TableName.META_TABLE_NAME, NB_SERVERS, getMetaCells(table.getName()));
+ // Try delete.
+ HTU.deleteTableIfAny(table.getName());
+ verifyDeletedReplication(TableName.META_TABLE_NAME, NB_SERVERS, table.getName());
+ }
+ }
+
+ /**
+ * Replicas come online after primary.
+ */
+ private void waitForMetaReplicasToOnline() throws IOException {
+ final RegionLocator regionLocator =
+ HTU.getConnection().getRegionLocator(TableName.META_TABLE_NAME);
+ HTU.waitFor(10000,
+ // getRegionLocations returns an entry for each replica but if unassigned, entry is null.
+ // Pass reload to force us to skip cache else it just keeps returning default.
+ () -> regionLocator.getRegionLocations(HConstants.EMPTY_START_ROW, true).stream().
+ filter(Objects::nonNull).count() >= NB_SERVERS);
+ List<HRegionLocation> locations = regionLocator.getRegionLocations(HConstants.EMPTY_START_ROW);
+ LOG.info("Found locations {}", locations);
+ assertEquals(NB_SERVERS, locations.size());
+ }
+
+ /**
+ * Scan hbase:meta for <code>tableName</code> content.
+ */
+ private List<Result> getMetaCells(TableName tableName) throws IOException {
+ final List<Result> results = new ArrayList<>();
+ MetaTableAccessor.Visitor visitor = new MetaTableAccessor.Visitor() {
+ @Override public boolean visit(Result r) throws IOException {
+ results.add(r);
+ return true;
+ }
+ };
+ MetaTableAccessor.scanMetaForTableRegions(HTU.getConnection(), visitor, tableName);
+ return results;
+ }
+
+ /**
+ * @return All Regions for tableName including Replicas.
+ */
+ private Region [] getAllRegions(TableName tableName, int replication) {
+ final Region[] regions = new Region[replication];
+ for (int i = 0; i < NB_SERVERS; i++) {
+ HRegionServer rs = HTU.getMiniHBaseCluster().getRegionServer(i);
+ List<HRegion> onlineRegions = rs.getRegions(tableName);
+ for (HRegion region : onlineRegions) {
+ regions[region.getRegionInfo().getReplicaId()] = region;
+ }
+ }
+ for (Region region : regions) {
+ assertNotNull(region);
+ }
+ return regions;
+ }
+
+ /**
+ * Verify when a Table is deleted from primary, then there are no references in replicas
+ * (because they get the delete of the table rows too).
+ */
+ private void verifyDeletedReplication(TableName tableName, int regionReplication,
+ final TableName deletedTableName) {
+ final Region[] regions = getAllRegions(tableName, regionReplication);
+
+ // Start count at '1' so we skip default, primary replica and only look at secondaries.
+ for (int i = 1; i < regionReplication; i++) {
+ final Region region = regions[i];
+ // wait until all the data is replicated to all secondary regions
+ Waiter.waitFor(HTU.getConfiguration(), 30000, 1000, new Waiter.Predicate<Exception>() {
+ @Override
+ public boolean evaluate() throws Exception {
+ LOG.info("Verifying replication for region replica {}", region.getRegionInfo());
+ try (RegionScanner rs = region.getScanner(new Scan())) {
+ List<Cell> cells = new ArrayList<>();
+ while (rs.next(cells)) {
+ continue;
+ }
+ return doesNotContain(cells, deletedTableName);
+ } catch(Throwable ex) {
+ LOG.warn("Verification from secondary region is not complete yet", ex);
+ // still wait
+ return false;
+ }
+ }
+ });
+ }
+ }
+
+ /**
+ * Cells are from hbase:meta replica so will start w/ 'tableName,'; i.e. the tablename followed
+ * by HConstants.DELIMITER. Make sure the deleted table is no longer present in passed
+ * <code>cells</code>.
+ */
+ private boolean doesNotContain(List<Cell> cells, TableName tableName) {
+ for (Cell cell: cells) {
+ String row = Bytes.toString(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength());
+ if (row.startsWith(tableName.toString() + HConstants.DELIMITER)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Verify Replicas have results (exactly).
+ */
+ private void verifyReplication(TableName tableName, int regionReplication,
+ List<Result> contains) {
+ final Region[] regions = getAllRegions(tableName, regionReplication);
+
+ // Start count at '1' so we skip default, primary replica and only look at secondaries.
+ for (int i = 1; i < regionReplication; i++) {
+ final Region region = regions[i];
+ // wait until all the data is replicated to all secondary regions
+ Waiter.waitFor(HTU.getConfiguration(), 30000, 1000, new Waiter.Predicate<Exception>() {
+ @Override
+ public boolean evaluate() throws Exception {
+ LOG.info("Verifying replication for region replica {}", region.getRegionInfo());
+ try (RegionScanner rs = region.getScanner(new Scan())) {
+ List<Cell> cells = new ArrayList<>();
+ while (rs.next(cells)) {
+ continue;
+ }
+ return contains(contains, cells);
+ } catch(Throwable ex) {
+ LOG.warn("Verification from secondary region is not complete yet", ex);
+ // still wait
+ return false;
+ }
+ }
+ });
+ }
+ }
+
+ /**
+ * Presumes sorted Cells. Verify that <code>cells</code> has <code>contains</code> at least.
+ */
+ static boolean contains(List<Result> contains, List<Cell> cells) throws IOException {
+ CellScanner containsScanner = CellUtil.createCellScanner(contains);
+ CellScanner cellsScanner = CellUtil.createCellScanner(cells);
+ int matches = 0;
+ int count = 0;
+ while (containsScanner.advance()) {
+ while (cellsScanner.advance()) {
+ count++;
+ LOG.info("{} {}", containsScanner.current(), cellsScanner.current());
+ if (containsScanner.current().equals(cellsScanner.current())) {
+ matches++;
+ break;
+ }
+ }
+ }
+ return !containsScanner.advance() && matches >= 1 && count >= matches && count == cells.size();
+ }
+}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
index c86f2f4..54a8dcc 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -20,12 +20,14 @@ package org.apache.hadoop.hbase.replication.regionserver;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.io.IOException;
import java.util.List;
+import java.util.UUID;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.hadoop.conf.Configuration;
@@ -42,6 +44,7 @@ import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.ReplicationPeerNotFoundException;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.Waiter;
+import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.ClusterConnection;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
@@ -51,6 +54,7 @@ import org.apache.hadoop.hbase.client.replication.ReplicationAdmin;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.HRegionServer;
import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.ReplicationEndpoint;
import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.testclassification.FlakeyTests;
@@ -69,6 +73,7 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.rules.TestName;
+import org.mockito.Mockito;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -124,91 +129,97 @@ public class TestRegionReplicaReplicationEndpoint {
public void testRegionReplicaReplicationPeerIsCreated() throws IOException, ReplicationException {
// create a table with region replicas. Check whether the replication peer is created
// and replication started.
- ReplicationAdmin admin = new ReplicationAdmin(HTU.getConfiguration());
- String peerId = "region_replica_replication";
-
- ReplicationPeerConfig peerConfig = null;
- try {
- peerConfig = admin.getPeerConfig(peerId);
- } catch (ReplicationPeerNotFoundException e) {
- LOG.warn("Region replica replication peer id=" + peerId + " not exist", e);
- }
-
- if (peerConfig != null) {
- admin.removePeer(peerId);
- peerConfig = null;
- }
+ try (Connection connection = ConnectionFactory.createConnection(HTU.getConfiguration());
+ Admin admin = connection.getAdmin()) {
+ String peerId = ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER;
+
+ ReplicationPeerConfig peerConfig = null;
+ try {
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ } catch (ReplicationPeerNotFoundException e) {
+ LOG.warn("Region replica replication peer id=" + peerId + " not exist", e);
+ }
- HTableDescriptor htd = HTU.createTableDescriptor(
- "testReplicationPeerIsCreated_no_region_replicas");
- HTU.getAdmin().createTable(htd);
- try {
- peerConfig = admin.getPeerConfig(peerId);
- fail("Should throw ReplicationException, because replication peer id=" + peerId
- + " not exist");
- } catch (ReplicationPeerNotFoundException e) {
- }
- assertNull(peerConfig);
+ try {
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ } catch (ReplicationPeerNotFoundException e) {
+ LOG.warn("Region replica replication peer id=" + peerId + " not exist", e);
+ }
- htd = HTU.createTableDescriptor("testReplicationPeerIsCreated");
- htd.setRegionReplication(2);
- HTU.getAdmin().createTable(htd);
+ if (peerConfig != null) {
+ admin.removeReplicationPeer(peerId);
+ peerConfig = null;
+ }
- // assert peer configuration is correct
- peerConfig = admin.getPeerConfig(peerId);
- assertNotNull(peerConfig);
- assertEquals(peerConfig.getClusterKey(), ZKConfig.getZooKeeperClusterKey(
- HTU.getConfiguration()));
- assertEquals(RegionReplicaReplicationEndpoint.class.getName(),
- peerConfig.getReplicationEndpointImpl());
- admin.close();
+ HTableDescriptor htd = HTU.createTableDescriptor(
+ "testReplicationPeerIsCreated_no_region_replicas");
+ HTU.getAdmin().createTable(htd);
+ try {
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ fail("Should throw ReplicationException, because replication peer id=" + peerId
+ + " not exist");
+ } catch (ReplicationPeerNotFoundException e) {
+ }
+ assertNull(peerConfig);
+
+ htd = HTU.createTableDescriptor("testReplicationPeerIsCreated");
+ htd.setRegionReplication(2);
+ HTU.getAdmin().createTable(htd);
+
+ // assert peer configuration is correct
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ assertNotNull(peerConfig);
+ assertEquals(peerConfig.getClusterKey(), ZKConfig.getZooKeeperClusterKey(
+ HTU.getConfiguration()));
+ assertEquals(RegionReplicaReplicationEndpoint.class.getName(),
+ peerConfig.getReplicationEndpointImpl());
+ }
}
@Test
public void testRegionReplicaReplicationPeerIsCreatedForModifyTable() throws Exception {
// modify a table by adding region replicas. Check whether the replication peer is created
// and replication started.
- ReplicationAdmin admin = new ReplicationAdmin(HTU.getConfiguration());
- String peerId = "region_replica_replication";
-
- ReplicationPeerConfig peerConfig = null;
- try {
- peerConfig = admin.getPeerConfig(peerId);
- } catch (ReplicationPeerNotFoundException e) {
- LOG.warn("Region replica replication peer id=" + peerId + " not exist", e);
- }
+ try (Connection connection = ConnectionFactory.createConnection(HTU.getConfiguration());
+ Admin admin = connection.getAdmin()) {
+ String peerId = ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER;
+ ReplicationPeerConfig peerConfig = null;
+ try {
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ } catch (ReplicationPeerNotFoundException e) {
+ LOG.warn("Region replica replication peer id=" + peerId + " not exist", e);
+ }
- if (peerConfig != null) {
- admin.removePeer(peerId);
- peerConfig = null;
- }
+ if (peerConfig != null) {
+ admin.removeReplicationPeer(peerId);
+ peerConfig = null;
+ }
- HTableDescriptor htd
- = HTU.createTableDescriptor("testRegionReplicaReplicationPeerIsCreatedForModifyTable");
- HTU.getAdmin().createTable(htd);
+ HTableDescriptor htd = HTU.createTableDescriptor("testRegionReplicaReplicationPeerIsCreatedForModifyTable");
+ HTU.getAdmin().createTable(htd);
- // assert that replication peer is not created yet
- try {
- peerConfig = admin.getPeerConfig(peerId);
- fail("Should throw ReplicationException, because replication peer id=" + peerId
+ // assert that replication peer is not created yet
+ try {
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ fail("Should throw ReplicationException, because replication peer id=" + peerId
+ " not exist");
- } catch (ReplicationPeerNotFoundException e) {
- }
- assertNull(peerConfig);
-
- HTU.getAdmin().disableTable(htd.getTableName());
- htd.setRegionReplication(2);
- HTU.getAdmin().modifyTable(htd.getTableName(), htd);
- HTU.getAdmin().enableTable(htd.getTableName());
-
- // assert peer configuration is correct
- peerConfig = admin.getPeerConfig(peerId);
- assertNotNull(peerConfig);
- assertEquals(peerConfig.getClusterKey(), ZKConfig.getZooKeeperClusterKey(
- HTU.getConfiguration()));
- assertEquals(RegionReplicaReplicationEndpoint.class.getName(),
+ } catch (ReplicationPeerNotFoundException e) {
+ }
+ assertNull(peerConfig);
+
+ HTU.getAdmin().disableTable(htd.getTableName());
+ htd.setRegionReplication(2);
+ HTU.getAdmin().modifyTable(htd.getTableName(), htd);
+ HTU.getAdmin().enableTable(htd.getTableName());
+
+ // assert peer configuration is correct
+ peerConfig = admin.getReplicationPeerConfig(peerId);
+ assertNotNull(peerConfig);
+ assertEquals(peerConfig.getClusterKey(), ZKConfig.getZooKeeperClusterKey(HTU.getConfiguration()));
+ assertEquals(RegionReplicaReplicationEndpoint.class.getName(),
peerConfig.getReplicationEndpointImpl());
- admin.close();
+ admin.close();
+ }
}
public void testRegionReplicaReplication(int regionReplication) throws Exception {
@@ -405,8 +416,7 @@ public class TestRegionReplicaReplicationEndpoint {
HTU.getAdmin().createTable(htd);
// both tables are created, now pause replication
- ReplicationAdmin admin = new ReplicationAdmin(HTU.getConfiguration());
- admin.disablePeer(ServerRegionReplicaUtil.getReplicationPeerId());
+ HTU.getAdmin().disableReplicationPeer(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER);
// now that the replication is disabled, write to the table to be dropped, then drop the table.
@@ -450,6 +460,27 @@ public class TestRegionReplicaReplicationEndpoint {
assertEquals(2, skippedEdits.get());
+ HRegionServer rs = HTU.getMiniHBaseCluster().getRegionServer(0);
+ MetricsSource metrics = mock(MetricsSource.class);
+ ReplicationEndpoint.Context ctx =
+ new ReplicationEndpoint.Context(HTU.getConfiguration(), HTU.getConfiguration(),
+ HTU.getTestFileSystem(), ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER,
+ UUID.fromString(rs.getClusterId()), rs.getReplicationSourceService().
+ getReplicationManager().getReplicationPeers()
+ .getPeer(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER),
+ metrics, rs.getTableDescriptors(), rs);
+ RegionReplicaReplicationEndpoint rrpe = new RegionReplicaReplicationEndpoint();
+ rrpe.init(ctx);
+ rrpe.start();
+ ReplicationEndpoint.ReplicateContext repCtx = new ReplicationEndpoint.ReplicateContext();
+ repCtx.setEntries(Lists.newArrayList(entry, entry));
+ assertTrue(rrpe.replicate(repCtx));
+ /* Come back here. There is a difference on how counting is done here and in master branch.
+ St.Ack
+ Mockito.verify(metrics, Mockito.times(1)).
+ incrLogEditsFiltered(Mockito.eq(2L));
+ */
+ rrpe.stop();
if (disableReplication) {
// enable replication again so that we can verify replication
HTU.getAdmin().disableTable(toBeDisabledTable); // disable the table
@@ -465,12 +496,11 @@ public class TestRegionReplicaReplicationEndpoint {
HTU.loadNumericRows(table, HBaseTestingUtility.fam1, 0, 1000);
// now enable the replication
- admin.enablePeer(ServerRegionReplicaUtil.getReplicationPeerId());
+ HTU.getAdmin().enableReplicationPeer(ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER);
verifyReplication(tableName, regionReplication, 0, 1000);
} finally {
- admin.close();
table.close();
rl.close();
tableToBeDisabled.close();
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
index 5f7a1c2..2745ad6 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
@@ -320,7 +320,7 @@ public abstract class TestReplicationSourceManager {
wal.rollWriter();
- manager.logPositionAndCleanOldLogs("1", false,
+ manager.logPositionAndCleanOldLogs(manager.getSources().get(0),
new WALEntryBatch(0, manager.getSources().get(0).getCurrentPath()));
wal.appendData(hri,
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSerialReplicationChecker.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSerialReplicationChecker.java
index 29749bd..9e513cd 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSerialReplicationChecker.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSerialReplicationChecker.java
@@ -109,7 +109,7 @@ public class TestSerialReplicationChecker {
public void setUp() throws IOException {
ReplicationSource source = mock(ReplicationSource.class);
when(source.getPeerId()).thenReturn(PEER_ID);
- when(source.getQueueStorage()).thenReturn(QUEUE_STORAGE);
+ when(source.getReplicationQueueStorage()).thenReturn(QUEUE_STORAGE);
conn = mock(Connection.class);
when(conn.isClosed()).thenReturn(false);
doAnswer(new Answer<Table>() {
[hbase] 01/03: HBASE-25068 Pass WALFactory to Replication so it
knows of all WALProviders, not just default/user-space
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stack pushed a commit to branch HBASE-18070.branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 5c4d9dcd5364f984c05b106bc0ffe315ba2ed902
Author: stack <st...@apache.org>
AuthorDate: Fri Sep 18 17:29:23 2020 -0700
HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
---
.../apache/hadoop/hbase/regionserver/HRegionServer.java | 17 +++++++----------
.../hadoop/hbase/regionserver/ReplicationService.java | 11 ++++-------
.../hbase/replication/regionserver/Replication.java | 11 ++++++-----
.../replication/regionserver/ReplicationSyncUp.java | 6 ++++--
.../hadoop/hbase/replication/TestReplicationBase.java | 1 -
.../regionserver/TestReplicationSourceManager.java | 3 ++-
6 files changed, 23 insertions(+), 26 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 31363b5..ca6b9f3 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1918,7 +1918,6 @@ public class HRegionServer extends Thread implements
*/
private void setupWALAndReplication() throws IOException {
WALFactory factory = new WALFactory(conf, serverName.toString(), (Server)this);
-
// TODO Replication make assumptions here based on the default filesystem impl
Path oldLogDir = new Path(walRootDir, HConstants.HREGION_OLDLOGDIR_NAME);
String logName = AbstractFSWALProvider.getWALDirectoryName(this.serverName.toString());
@@ -1935,8 +1934,7 @@ public class HRegionServer extends Thread implements
throw new IOException("Can not create wal directory " + logDir);
}
// Instantiate replication if replication enabled. Pass it the log directories.
- createNewReplicationInstance(conf, this, this.walFs, logDir, oldLogDir,
- factory.getWALProvider());
+ createNewReplicationInstance(conf, this, this.walFs, logDir, oldLogDir, factory);
this.walFactory = factory;
}
@@ -3076,12 +3074,11 @@ public class HRegionServer extends Thread implements
* Load the replication executorService objects, if any
*/
private static void createNewReplicationInstance(Configuration conf, HRegionServer server,
- FileSystem walFs, Path walDir, Path oldWALDir, WALProvider walProvider) throws IOException {
+ FileSystem walFs, Path walDir, Path oldWALDir, WALFactory walFactory) throws IOException {
if ((server instanceof HMaster) &&
(!LoadBalancer.isTablesOnMaster(conf) || LoadBalancer.isSystemTablesOnlyOnMaster(conf))) {
return;
}
-
// read in the name of the source replication class from the config file.
String sourceClassname = conf.get(HConstants.REPLICATION_SOURCE_SERVICE_CLASSNAME,
HConstants.REPLICATION_SERVICE_CLASSNAME_DEFAULT);
@@ -3094,19 +3091,19 @@ public class HRegionServer extends Thread implements
// only one object.
if (sourceClassname.equals(sinkClassname)) {
server.replicationSourceHandler = newReplicationInstance(sourceClassname,
- ReplicationSourceService.class, conf, server, walFs, walDir, oldWALDir, walProvider);
+ ReplicationSourceService.class, conf, server, walFs, walDir, oldWALDir, walFactory);
server.replicationSinkHandler = (ReplicationSinkService) server.replicationSourceHandler;
} else {
server.replicationSourceHandler = newReplicationInstance(sourceClassname,
- ReplicationSourceService.class, conf, server, walFs, walDir, oldWALDir, walProvider);
+ ReplicationSourceService.class, conf, server, walFs, walDir, oldWALDir, walFactory);
server.replicationSinkHandler = newReplicationInstance(sinkClassname,
- ReplicationSinkService.class, conf, server, walFs, walDir, oldWALDir, walProvider);
+ ReplicationSinkService.class, conf, server, walFs, walDir, oldWALDir, walFactory);
}
}
private static <T extends ReplicationService> T newReplicationInstance(String classname,
Class<T> xface, Configuration conf, HRegionServer server, FileSystem walFs, Path logDir,
- Path oldLogDir, WALProvider walProvider) throws IOException {
+ Path oldLogDir, WALFactory walFactory) throws IOException {
final Class<? extends T> clazz;
try {
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
@@ -3115,7 +3112,7 @@ public class HRegionServer extends Thread implements
throw new IOException("Could not find class for " + classname);
}
T service = ReflectionUtils.newInstance(clazz, conf);
- service.initialize(server, walFs, logDir, oldLogDir, walProvider);
+ service.initialize(server, walFs, logDir, oldLogDir, walFactory);
return service;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
index e9bbaea..33b3321 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -22,7 +22,7 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad;
-import org.apache.hadoop.hbase.wal.WALProvider;
+import org.apache.hadoop.hbase.wal.WALFactory;
import org.apache.yetus.audience.InterfaceAudience;
/**
@@ -32,14 +32,11 @@ import org.apache.yetus.audience.InterfaceAudience;
*/
@InterfaceAudience.Private
public interface ReplicationService {
-
/**
* Initializes the replication service object.
- * @param walProvider can be null if not initialized inside a live region server environment, for
- * example, {@code ReplicationSyncUp}.
*/
- void initialize(Server rs, FileSystem fs, Path logdir, Path oldLogDir, WALProvider walProvider)
- throws IOException;
+ void initialize(Server rs, FileSystem fs, Path logdir, Path oldLogDir, WALFactory walFactory)
+ throws IOException;
/**
* Start replication services.
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
index 10ddd0c..136dcd4 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -42,6 +42,7 @@ import org.apache.hadoop.hbase.replication.ReplicationStorageFactory;
import org.apache.hadoop.hbase.replication.ReplicationTracker;
import org.apache.hadoop.hbase.replication.ReplicationUtils;
import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.wal.WALFactory;
import org.apache.hadoop.hbase.wal.WALProvider;
import org.apache.hadoop.hbase.zookeeper.ZKClusterId;
import org.apache.yetus.audience.InterfaceAudience;
@@ -85,7 +86,7 @@ public class Replication implements ReplicationSourceService, ReplicationSinkSer
@Override
public void initialize(Server server, FileSystem fs, Path logDir, Path oldLogDir,
- WALProvider walProvider) throws IOException {
+ WALFactory walFactory) throws IOException {
this.server = server;
this.conf = this.server.getConfiguration();
this.isReplicationForBulkLoadDataEnabled =
@@ -123,8 +124,9 @@ public class Replication implements ReplicationSourceService, ReplicationSinkSer
}
this.globalMetricsSource = CompatibilitySingletonFactory
.getInstance(MetricsReplicationSourceFactory.class).getGlobalSource();
- this.replicationManager = new ReplicationSourceManager(queueStorage, replicationPeers, replicationTracker, conf,
- this.server, fs, logDir, oldLogDir, clusterId,
+ WALProvider walProvider = walFactory.getWALProvider();
+ this.replicationManager = new ReplicationSourceManager(queueStorage, replicationPeers,
+ replicationTracker, conf, this.server, fs, logDir, oldLogDir, clusterId,
walProvider != null ? walProvider.getWALFileLengthProvider() : p -> OptionalLong.empty(),
globalMetricsSource);
if (walProvider != null) {
@@ -187,7 +189,6 @@ public class Replication implements ReplicationSourceService, ReplicationSinkSer
/**
* If replication is enabled and this cluster is a master,
* it starts
- * @throws IOException
*/
@Override
public void startReplicationService() throws IOException {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
index 5e3a09f..a8fd640 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSyncUp.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -32,6 +32,7 @@ import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.client.ClusterConnection;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.hbase.wal.WALFactory;
import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
@@ -82,7 +83,8 @@ public class ReplicationSyncUp extends Configured implements Tool {
System.out.println("Start Replication Server start");
Replication replication = new Replication();
- replication.initialize(new DummyServer(zkw), fs, logDir, oldLogDir, null);
+ replication.initialize(new DummyServer(zkw), fs, logDir, oldLogDir,
+ new WALFactory(conf, "test", null));
ReplicationSourceManager manager = replication.getReplicationManager();
manager.init().get();
while (manager.activeFailoverTaskCount() > 0) {
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
index e76c222..b017a89 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
@@ -1,5 +1,4 @@
/*
- *
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
index 1e40457..5f7a1c2 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
@@ -182,7 +182,8 @@ public abstract class TestReplicationSourceManager {
logDir = new Path(utility.getDataTestDir(),
HConstants.HREGION_LOGDIR_NAME);
replication = new Replication();
- replication.initialize(new DummyServer(), fs, logDir, oldLogDir, null);
+ replication.initialize(new DummyServer(), fs, logDir, oldLogDir,
+ new WALFactory(conf, "test", null));
managerOfCluster = getManagerFromCluster();
if (managerOfCluster != null) {
// After replication procedure, we need to add peer by hand (other than by receiving
[hbase] 03/03: HBASE-25151 warmupRegion frustrates registering WALs
on the catalog replicationsource
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stack pushed a commit to branch HBASE-18070.branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 77749cf6a6a9adfd287d488fb37df9b45cd13cf3
Author: stack <st...@apache.org>
AuthorDate: Sat Oct 3 01:37:45 2020 -0700
HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
warmupRegion called by Master on Region move will instatiate
the meta WALProvider as part of its action making it so
it is already created by the time we go to open the
hbsae:meta Region. Accommodate meta walProvider
being already up.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
Pass regionInfo. Needed internally.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
Add handling if meta wal provider already instantiated when
addCatalogReplicationSource runs.
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
Add exercising moving meta around between servers. Test replication
keeps working.
---
.../apache/hadoop/hbase/regionserver/HRegion.java | 14 +--
.../hadoop/hbase/regionserver/RSRpcServices.java | 28 ++---
.../regionserver/handler/AssignRegionHandler.java | 3 +-
.../handler/UnassignRegionHandler.java | 3 +-
.../replication/ReplicationSinkServiceImpl.java | 118 +++++++++++++++++++++
.../RegionReplicaReplicationEndpoint.java | 15 ++-
.../regionserver/ReplicationSourceManager.java | 61 ++++++-----
.../org/apache/hadoop/hbase/wal/WALFactory.java | 4 +-
.../TestMetaRegionReplicaReplicationEndpoint.java | 27 ++++-
.../regionserver/TestReplicationSource.java | 4 +
10 files changed, 212 insertions(+), 65 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 3a08239..a163aeb 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1193,11 +1193,11 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi
private void initializeWarmup(final CancelableProgressable reporter) throws IOException {
MonitoredTask status = TaskMonitor.get().createStatus("Initializing region " + this);
// Initialize all the HStores
- status.setStatus("Warming up all the Stores");
+ status.setStatus("Warmup all stores of " + this.getRegionInfo().getRegionNameAsString());
try {
initializeStores(reporter, status, true);
} finally {
- status.markComplete("Done warming up.");
+ status.markComplete("Warmed up " + this.getRegionInfo().getRegionNameAsString());
}
}
@@ -8125,15 +8125,10 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi
final CancelableProgressable reporter)
throws IOException {
- if (info == null) throw new NullPointerException("Passed region info is null");
-
- if (LOG.isDebugEnabled()) {
- LOG.debug("HRegion.Warming up region: " + info);
- }
-
+ Objects.requireNonNull(info, "RegionInfo cannot be null");
+ LOG.debug("Warmup {}", info);
Path rootDir = CommonFSUtils.getRootDir(conf);
Path tableDir = CommonFSUtils.getTableDir(rootDir, info.getTable());
-
FileSystem fs = null;
if (rsServices != null) {
fs = rsServices.getFileSystem();
@@ -8141,7 +8136,6 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi
if (fs == null) {
fs = rootDir.getFileSystem(conf);
}
-
HRegion r = HRegion.newHRegion(tableDir, wal, fs, conf, info, htd, null);
r.initializeWarmup(reporter);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
index 94279e4..8ab02c7 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
@@ -2040,48 +2040,34 @@ public class RSRpcServices implements HBaseRPCErrorHandler,
}
/**
- * Wamrmup a region on this server.
- *
- * This method should only be called by Master. It synchrnously opens the region and
+ * Warmup a region on this server.
+ * This method should only be called by Master. It synchronously opens the region and
* closes the region bringing the most important pages in cache.
- * <p>
- *
- * @param controller the RPC controller
- * @param request the request
- * @throws ServiceException
*/
@Override
public WarmupRegionResponse warmupRegion(final RpcController controller,
final WarmupRegionRequest request) throws ServiceException {
-
final RegionInfo region = ProtobufUtil.toRegionInfo(request.getRegionInfo());
- TableDescriptor htd;
WarmupRegionResponse response = WarmupRegionResponse.getDefaultInstance();
-
try {
checkOpen();
String encodedName = region.getEncodedName();
byte[] encodedNameBytes = region.getEncodedNameAsBytes();
final HRegion onlineRegion = regionServer.getRegion(encodedName);
-
if (onlineRegion != null) {
- LOG.info("Region already online. Skipping warming up " + region);
+ LOG.info("{} is online; skipping warmup", region);
return response;
}
-
- htd = regionServer.tableDescriptors.get(region.getTable());
-
+ TableDescriptor htd = regionServer.tableDescriptors.get(region.getTable());
if (regionServer.getRegionsInTransitionInRS().containsKey(encodedNameBytes)) {
- LOG.info("Region is in transition. Skipping warmup " + region);
+ LOG.info("{} is in transition; skipping warmup", region);
return response;
}
-
- LOG.info("Warming up region " + region.getRegionNameAsString());
+ LOG.info("Warmup {}", region.getRegionNameAsString());
HRegion.warmupHRegion(region, htd, regionServer.getWAL(region),
regionServer.getConfiguration(), regionServer, null);
-
} catch (IOException ie) {
- LOG.error("Failed warming up region " + region.getRegionNameAsString(), ie);
+ LOG.error("Failed warmup of {}", region.getRegionNameAsString(), ie);
throw new ServiceException(ie);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
index 4ee6efc..5d9819c 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
@@ -138,7 +138,8 @@ public class AssignRegionHandler extends EventHandler {
if (ServerRegionReplicaUtil.isMetaRegionReplicaReplicationEnabled(conf, tn)) {
if (RegionReplicaUtil.isDefaultReplica(this.regionInfo.getReplicaId())) {
// Add the hbase:meta replication source on replica zero/default.
- rs.getReplicationSourceService().getReplicationManager().addCatalogReplicationSource();
+ rs.getReplicationSourceService().getReplicationManager().
+ addCatalogReplicationSource(this.regionInfo);
}
}
region = HRegion.openHRegion(regionInfo, htd, rs.getWAL(regionInfo), conf, rs, null);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
index 1ed74bb..0d02f30 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
@@ -126,7 +126,8 @@ public class UnassignRegionHandler extends EventHandler {
if (RegionReplicaUtil.isDefaultReplica(region.getRegionInfo().getReplicaId())) {
// If hbase:meta read replicas enabled, remove replication source for hbase:meta Regions.
// See assign region handler where we add the replication source on open.
- rs.getReplicationSourceService().getReplicationManager().removeCatalogReplicationSource();
+ rs.getReplicationSourceService().getReplicationManager().
+ removeCatalogReplicationSource(region.getRegionInfo());
}
}
if (!rs.reportRegionStateTransition(
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSinkServiceImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSinkServiceImpl.java
new file mode 100644
index 0000000..edd5679
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSinkServiceImpl.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellScanner;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.regionserver.ReplicationSinkService;
+import org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad;
+import org.apache.hadoop.hbase.replication.regionserver.ReplicationSink;
+import org.apache.hadoop.hbase.wal.WALFactory;
+import org.apache.hadoop.hbase.wal.WALProvider;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos;
+
+@InterfaceAudience.Private
+public class ReplicationSinkServiceImpl implements ReplicationSinkService {
+ private static final Logger LOG = LoggerFactory.getLogger(ReplicationSinkServiceImpl.class);
+
+ private Configuration conf;
+
+ private Server server;
+
+ private ReplicationSink replicationSink;
+
+ // ReplicationLoad to access replication metrics
+ private ReplicationLoad replicationLoad;
+
+ private int statsPeriodInSecond;
+
+ @Override
+ public void replicateLogEntries(List<AdminProtos.WALEntry> entries, CellScanner cells,
+ String replicationClusterId, String sourceBaseNamespaceDirPath,
+ String sourceHFileArchiveDirPath) throws IOException {
+ this.replicationSink.replicateEntries(entries, cells, replicationClusterId,
+ sourceBaseNamespaceDirPath, sourceHFileArchiveDirPath);
+ }
+
+ @Override
+ public void initialize(Server server, FileSystem fs, Path logdir, Path oldLogDir,
+ WALFactory walFactory) throws IOException {
+ this.server = server;
+ this.conf = server.getConfiguration();
+ this.statsPeriodInSecond =
+ this.conf.getInt("replication.stats.thread.period.seconds", 5 * 60);
+ this.replicationLoad = new ReplicationLoad();
+ }
+
+ @Override
+ public void startReplicationService() throws IOException {
+ this.replicationSink = new ReplicationSink(this.conf);
+ this.server.getChoreService().scheduleChore(
+ new ReplicationStatisticsChore("ReplicationSinkStatistics", server,
+ (int) TimeUnit.SECONDS.toMillis(statsPeriodInSecond)));
+ }
+
+ @Override
+ public void stopReplicationService() {
+ if (this.replicationSink != null) {
+ this.replicationSink.stopReplicationSinkServices();
+ }
+ }
+
+ @Override
+ public ReplicationLoad refreshAndGetReplicationLoad() {
+ if (replicationLoad == null) {
+ return null;
+ }
+ // always build for latest data
+ replicationLoad.buildReplicationLoad(Collections.emptyList(), replicationSink.getSinkMetrics());
+ return replicationLoad;
+ }
+
+ private final class ReplicationStatisticsChore extends ScheduledChore {
+
+ ReplicationStatisticsChore(String name, Stoppable stopper, int period) {
+ super(name, stopper, period);
+ }
+
+ @Override
+ protected void chore() {
+ printStats(replicationSink.getStats());
+ }
+
+ private void printStats(String stats) {
+ if (!stats.isEmpty()) {
+ LOG.info(stats);
+ }
+ }
+ }
+}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
index 7eca732..d5ebbe9 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
@@ -444,11 +444,22 @@ public class RegionReplicaReplicationEndpoint extends HBaseReplicationEndpoint {
try {
locations = RegionReplicaReplayCallable
.getRegionLocations(connection, tableName, row, useCache, 0);
-
if (locations == null) {
throw new HBaseIOException("Cannot locate locations for "
+ tableName + ", row:" + Bytes.toStringBinary(row));
}
+ // Replicas can take a while to come online. The cache may have only the primary. If we
+ // keep going to the cache, we will not learn of the replicas and their locations after
+ // they come online.
+ if (useCache && locations.size() == 1 && TableName.isMetaTableName(tableName)) {
+ if (tableDescriptors.get(tableName).getRegionReplication() > 1) {
+ // Make an obnoxious log here. See how bad this issue is. Add a timer if happening
+ // too much.
+ LOG.info("Skipping location cache; only one location found for {}", tableName);
+ useCache = false;
+ continue;
+ }
+ }
} catch (TableNotFoundException e) {
if (LOG.isTraceEnabled()) {
LOG.trace("Skipping " + entries.size() + " entries because table " + tableName
@@ -467,7 +478,7 @@ public class RegionReplicaReplicationEndpoint extends HBaseReplicationEndpoint {
// entry is not coming from the primary region, filter it out.
HRegionLocation primaryLocation = locations.getDefaultRegionLocation();
if (!Bytes.equals(primaryLocation.getRegionInfo().getEncodedNameAsBytes(),
- encodedRegionName)) {
+ encodedRegionName)) {
if (useCache) {
useCache = false;
continue; // this will retry location lookup
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
index 93730d4..860cbd3 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
@@ -50,6 +50,8 @@ import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL;
import org.apache.hadoop.hbase.regionserver.wal.WALActionsListener;
import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationListener;
@@ -63,6 +65,7 @@ import org.apache.hadoop.hbase.replication.ReplicationTracker;
import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
+import org.apache.hadoop.hbase.wal.WAL;
import org.apache.hadoop.hbase.wal.WALFactory;
import org.apache.hadoop.hbase.wal.WALProvider;
import org.apache.yetus.audience.InterfaceAudience;
@@ -343,7 +346,7 @@ public class ReplicationSourceManager implements ReplicationListener {
/**
* @return a new 'classic' user-space replication source.
* @param queueId the id of the replication queue to associate the ReplicationSource with.
- * @see #createCatalogReplicationSource() for creating a ReplicationSource for hbase:meta.
+ * @see #createCatalogReplicationSource(RegionInfo) for creating a ReplicationSource for meta.
*/
private ReplicationSourceInterface createSource(String queueId, ReplicationPeer replicationPeer)
throws IOException {
@@ -985,57 +988,65 @@ public class ReplicationSourceManager implements ReplicationListener {
/**
* Add an hbase:meta Catalog replication source. Called on open of an hbase:meta Region.
- * @see #removeCatalogReplicationSource()
+ * Create it once only. If exists already, use the existing one.
+ * @see #removeCatalogReplicationSource(RegionInfo)
+ * @see #addSource(String) This is specialization on the addSource method.
*/
- public ReplicationSourceInterface addCatalogReplicationSource() throws IOException {
- // Open/Create the hbase:meta ReplicationSource once only.
+ public ReplicationSourceInterface addCatalogReplicationSource(RegionInfo regionInfo)
+ throws IOException {
+ // Poor-man's putIfAbsent
synchronized (this.catalogReplicationSource) {
ReplicationSourceInterface rs = this.catalogReplicationSource.get();
return rs != null ? rs :
- this.catalogReplicationSource.getAndSet(createCatalogReplicationSource());
+ this.catalogReplicationSource.getAndSet(createCatalogReplicationSource(regionInfo));
}
}
/**
* Remove the hbase:meta Catalog replication source.
* Called when we close hbase:meta.
- * @see #addCatalogReplicationSource()
+ * @see #addCatalogReplicationSource(RegionInfo regionInfo)
*/
- public void removeCatalogReplicationSource() {
+ public void removeCatalogReplicationSource(RegionInfo regionInfo) {
// Nothing to do. Leave any CatalogReplicationSource in place in case an hbase:meta Region
// comes back to this server.
}
/**
* Create, initialize, and start the Catalog ReplicationSource.
+ * Presumes called one-time only (caller must ensure one-time only call).
+ * @see #addSource(String) This is a specialization of the addSource call.
*/
- private ReplicationSourceInterface createCatalogReplicationSource() throws IOException {
- // Has the hbase:meta WALProvider been instantiated?
+ private ReplicationSourceInterface createCatalogReplicationSource(RegionInfo regionInfo)
+ throws IOException {
+ // Instantiate meta walProvider. Instantiated here or over in the #warmupRegion call made by the
+ // Master on a 'move' operation. Need to do extra work if we did NOT instantiate the provider.
WALProvider walProvider = this.walFactory.getMetaWALProvider();
- boolean addListener = false;
- if (walProvider == null) {
- // The meta walProvider has not been instantiated. Create it.
+ boolean instantiate = walProvider == null;
+ if (instantiate) {
walProvider = this.walFactory.getMetaProvider();
- addListener = true;
}
CatalogReplicationSourcePeer peer = new CatalogReplicationSourcePeer(this.conf,
this.clusterId.toString(), "meta_" + ServerRegionReplicaUtil.REGION_REPLICA_REPLICATION_PEER);
final ReplicationSourceInterface crs = new CatalogReplicationSource();
crs.init(conf, fs, this, new NoopReplicationQueueStorage(), peer, server, peer.getId(),
clusterId, walProvider.getWALFileLengthProvider(), new MetricsSource(peer.getId()));
- if (addListener) {
- walProvider.addWALActionsListener(new WALActionsListener() {
- @Override
- public void postLogRoll(Path oldPath, Path newPath) throws IOException {
- crs.enqueueLog(newPath);
- }
- });
- } else {
- // This is a problem. We'll have a ReplicationSource but no listener on hbase:meta WALs
- // so nothing will be replicated.
- LOG.error("Did not install WALActionsListener creating CatalogReplicationSource!");
+ // Add listener on the provider so we can pick up the WAL to replicate on roll.
+ WALActionsListener listener = new WALActionsListener() {
+ @Override public void postLogRoll(Path oldPath, Path newPath) throws IOException {
+ crs.enqueueLog(newPath);
+ }
+ };
+ walProvider.addWALActionsListener(listener);
+ if (!instantiate) {
+ // If we did not instantiate provider, need to add our listener on already-created WAL
+ // instance too (listeners are passed by provider to WAL instance on creation but if provider
+ // created already, our listener add above is missed). And add the current WAL file to the
+ // Replication Source so it can start replicating it.
+ WAL wal = walProvider.getWAL(regionInfo);
+ wal.registerWALActionsListener(listener);
+ crs.enqueueLog(((AbstractFSWAL)wal).getCurrentFileName());
}
- // Start this ReplicationSource.
return crs.startup();
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
index 7b8b3a0..9747673 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
@@ -289,10 +289,10 @@ public class WALFactory {
}
/**
- * @param region the region which we want to get a WAL for it. Could be null.
+ * @param region the region which we want to get a WAL for. Could be null.
*/
public WAL getWAL(RegionInfo region) throws IOException {
- // use different WAL for hbase:meta
+ // Use different WAL for hbase:meta. Instantiates the meta WALProvider if not already up.
if (region != null && region.isMetaRegion() &&
region.getReplicaId() == RegionInfo.DEFAULT_REPLICA_ID) {
return getMetaProvider().getWAL(region);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
index 34432e4..3cefce5 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
@@ -109,12 +109,13 @@ public class TestMetaRegionReplicaReplicationEndpoint {
* Assert that the ReplicationSource for hbase:meta gets created when hbase:meta is opened.
*/
@Test
- public void testHBaseMetaReplicationSourceCreatedOnOpen()
- throws IOException, InterruptedException {
+ public void testHBaseMetaReplicationSourceCreatedOnOpen() throws Exception {
MiniHBaseCluster cluster = HTU.getMiniHBaseCluster();
HRegionServer hrs = cluster.getRegionServer(cluster.getServerHoldingMeta());
+ // Replicate a row to prove all working.
+ testHBaseMetaReplicatesOneRow(0);
assertTrue(isMetaRegionReplicaReplicationSource(hrs));
- // Now move the hbase:meta and make sure the ReplicationSoruce is in both places.
+ // Now move the hbase:meta and make sure the ReplicationSource is in both places.
HRegionServer hrsOther = null;
for (int i = 0; i < cluster.getNumLiveRegionServers(); i++) {
hrsOther = cluster.getRegionServer(i);
@@ -138,6 +139,26 @@ public class TestMetaRegionReplicaReplicationEndpoint {
// Assert that there is a ReplicationSource in both places now.
assertTrue(isMetaRegionReplicaReplicationSource(hrs));
assertTrue(isMetaRegionReplicaReplicationSource(hrsOther));
+ // Replicate to show stuff still works.
+ testHBaseMetaReplicatesOneRow(1);
+ // Now pretend a few hours have gone by... roll the meta WAL in original location... Move the
+ // meta back and retry replication. See if it works.
+ hrs.getWAL(meta.getRegionInfo()).rollWriter(true);
+ testHBaseMetaReplicatesOneRow(2);
+ hrs.getWAL(meta.getRegionInfo()).rollWriter(true);
+ testHBaseMetaReplicatesOneRow(3);
+ }
+
+ /**
+ * Test meta region replica replication. Create some tables and see if replicas pick up the
+ * additions.
+ */
+ private void testHBaseMetaReplicatesOneRow(int i) throws Exception {
+ waitForMetaReplicasToOnline();
+ try (Table table = HTU.createTable(TableName.valueOf(this.name.getMethodName() + "_" + i),
+ HConstants.CATALOG_FAMILY)) {
+ verifyReplication(TableName.META_TABLE_NAME, NB_SERVERS, getMetaCells(table.getName()));
+ }
}
/**
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSource.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSource.java
index 275ade2..c050b86 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSource.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSource.java
@@ -535,7 +535,11 @@ public class TestReplicationSource {
FlakyReplicationEndpoint.class.getName());
try {
rs.startup();
+ assertTrue(rs.isSourceActive());
+ Waiter.waitFor(conf, 1000, () -> FaultyReplicationEndpoint.count > 0);
Waiter.waitFor(conf, 1000, () -> rss.isAborted());
+ assertTrue(rss.isAborted());
+ Waiter.waitFor(conf, 1000, () -> !rs.isSourceActive());
assertFalse(rs.isSourceActive());
} finally {
rs.terminate("Done");