[GitHub] [hbase] huaxiangsun opened a new pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun opened a new pull request #2522:
URL: https://github.com/apache/hbase/pull/2522


   …ation.
   
   It creates a replication endpoint for meta/root:
   1. Supports multiple primary meta regions per region server.
   2. For each replica region, there is a shipper, internally, wal edits are queued to shipper's own queue.
      each shipper ships edits at their own paces so the slow region server wont block the fast ones.
      The queue is cleanedup when flush event is seen.
   3. Each primary region registers/deregisters based on REGION_OPEN/REGION_CLOSE events from wal.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706464344


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m 13s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 46s |  HBASE-18070 passed  |
   | +1 :green_heart: |  compile  |   1m  0s |  HBASE-18070 passed  |
   | +1 :green_heart: |  shadedjars  |   7m  9s |  branch has no errors when building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 36s |  HBASE-18070 passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 50s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  0s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m  0s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m  8s |  patch has no errors when building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 36s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 222m 56s |  hbase-server in the patch failed.  |
   |  |   | 251m 14s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 27c6a3e8b716 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | Default Java | 1.8.0_232 |
   | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/testReport/ |
   | Max. process+thread count | 3134 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] saintstack commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

saintstack commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502586911



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java
##########
@@ -63,6 +64,13 @@
   CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, byte[] row,
       List<Entry> entries, int replicaId, int numRetries, long operationTimeoutNs);
 
+  /**
+   * Replicate wal edits for replica regions, the location for the replica is already included.
+   * The return value is the edits we skipped, as the original return value is useless.
+   */
+  CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, List<Entry> entries,
+    int replicaId, int numRetries, long operationTimeoutNs, HRegionLocation loc);

Review comment:
       Why pass the replicaid? Is this not already inside the 'loc'?
   
   The numRetries is passed as a param? It is not just a config that we read internally?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionReplicaReplayRetryingCaller.java
##########
@@ -75,6 +77,22 @@ public AsyncRegionReplicaReplayRetryingCaller(HashedWheelTimer retryTimer,
     this.row = row;
     this.entries = entries.toArray(new Entry[0]);
     this.replicaId = replicaId;
+    this.loc = null;

Review comment:
       Seems a bit odd taking this on construction but then the call takes an HRL that it figures out just before 'call' is made.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java
##########
@@ -40,6 +39,10 @@
   private static final Logger LOG = LoggerFactory.getLogger(BaseReplicationEndpoint.class);
   public static final String REPLICATION_WALENTRYFILTER_CONFIG_KEY
       = "hbase.replication.source.custom.walentryfilters";
+  // Can be configured differently than hbase.client.retries.number
+  public static final String CLIENT_RETRIES_NUMBER =
+    "hbase.region.replica.replication.client.retries.number";

Review comment:
       This should make it out to release notes.
   
   We should make sure that the prefix used here aligns w/ prefix used elsewhere in this feature (maybe for this config it doesn't matter).

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,495 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.

Review comment:
       So if a server is carrying 100 meta Regions, there'd be 100 threads?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,495 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.

Review comment:
       +1 as follow-on.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",

Review comment:
       No need of word 'region' in log message

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));

Review comment:
       isFlush method?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java
##########
@@ -63,6 +64,13 @@
   CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, byte[] row,
       List<Entry> entries, int replicaId, int numRetries, long operationTimeoutNs);
 
+  /**
+   * Replicate wal edits for replica regions, the location for the replica is already included.
+   * The return value is the edits we skipped, as the original return value is useless.
+   */
+  CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, List<Entry> entries,
+    int replicaId, int numRetries, long operationTimeoutNs, HRegionLocation loc);

Review comment:
       If the read replica location changes while we are trying to play these edits, does this fail or do we adjust to the new location and replay edits to the new place?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {

Review comment:
       Hmm... The isFlush was something I added but then removed as not relevant.
   
   I see there isRegionCloseMarker and you add isRegionOpenMarker... you can't use these here?
   
   I suppose suggestion is to add something like these? 
   
   nit!

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionReplicaReplayRetryingCaller.java
##########
@@ -63,6 +63,8 @@
 
   private final int replicaId;
 
+  private final HRegionLocation loc;

Review comment:
       We pass this in rather than figure out the location internally as we used to do before this change?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionReplicaReplayRetryingCaller.java
##########
@@ -133,15 +150,19 @@ protected void doCall() {
     } else {
       locateTimeoutNs = -1L;
     }
-    addListener(conn.getLocator().getRegionLocation(tableName, row, replicaId,
-      RegionLocateType.CURRENT, locateTimeoutNs), (loc, error) -> {
+    if (loc == null) {
+      addListener(conn.getLocator()
+        .getRegionLocation(tableName, row, replicaId, RegionLocateType.CURRENT, locateTimeoutNs), (loc, error) -> {
         if (error != null) {
-          onError(error,
-            () -> "Locate '" + Bytes.toStringBinary(row) + "' in " + tableName + " failed", err -> {
+          onError(error, () -> "Locate '" + Bytes.toStringBinary(row) + "' in " + tableName + " failed",
+            err -> {
             });
           return;
         }
         call(loc);
       });
+    } else {
+      call(this.loc);

Review comment:
       Perhaps make a new subclass if you have to do this rather than this if/else? New subclass seems to be the way we usually do it.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionReplicaReplayRetryingCaller.java
##########
@@ -133,15 +150,19 @@ protected void doCall() {
     } else {
       locateTimeoutNs = -1L;
     }
-    addListener(conn.getLocator().getRegionLocation(tableName, row, replicaId,
-      RegionLocateType.CURRENT, locateTimeoutNs), (loc, error) -> {
+    if (loc == null) {

Review comment:
       Yeah, wondering why old mechanism doesn't apply here.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();

Review comment:
       If you made your own connection, you wouldn't have to pass timeout and retries as params?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)

Review comment:
       regionName has encodedRegionName in it?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {

Review comment:
       s/unregisterRegion/deregisterRegion/g

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);

Review comment:
       Yeah, if you did this up top then you wouldn't need encoded name as param?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {

Review comment:
       Can the edit have families that shouldn't be replicated too? i.e. do you have to make a new WALEdit that strips families that shouldn't be replicated?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already

Review comment:
       This is a little flakey.
   
   Default is 35? Or 350?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);

Review comment:
       s/regoin/region/

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =

Review comment:
       Duo is moving this to table descriptor. You'll be able to get it from there when it goes in?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);

Review comment:
       s/setRunning/stop/?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }

Review comment:
       Big method. Looks like you could make it into a few smaller ones. nit.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {

Review comment:
       We have isFlush.... Add isClose and isOpen... rather than doing this type compare?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALEdit.java
##########
@@ -320,6 +323,15 @@ public static FlushDescriptor getFlushDescriptor(Cell cell) throws IOException {
         FlushDescriptor.parseFrom(CellUtil.cloneValue(cell)): null;
   }
 
+  public static FlushDescriptor getCommitFlushDescriptor(Cell cell) throws IOException {

Review comment:
       Maybe add an isFlush convenience method?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {

Review comment:
       Yeah, I have to do same thing... set my editor to not inline the @Overrides.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);

Review comment:
       Add stop to CatalogReplicaShiipper?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;

Review comment:
       Why take an HRL if we have one of these? Can't we ask it to do look up?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);

Review comment:
       ditto
   

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;

Review comment:
       isRunning is name of method to access a boolean. Change this data member name to be 'running'.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {

Review comment:
       s/addEditEntries/addEntries/?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));

Review comment:
       Maybe add this message higher up too where you are doing the filter by famlies?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {

Review comment:
       New line at 'public'.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;

Review comment:
       Is this regionInfo different to what is inside HRL?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);

Review comment:
       No need of the word 'region' in this log message.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {

Review comment:
       Move out? Make own standalone class. Add some class doc.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();
+            entries = null;
+          } catch (InterruptedException e) {
+            // restore the interrupted state
+            Thread.currentThread().interrupt();
+            continue;
+          } catch (ExecutionException e) {
+            Throwable cause = e.getCause();
+            LOG.warn("Failed to replicate {} entries for region  of meta table", entries.size());
+            LOG.warn("Cause: " + cause);
+            location = null;
+            reload = true;
+          }
+
+        } catch (InterruptedException | ReplicationRuntimeException e) {
+          // It is interrupted and needs to quit.
+          LOG.warn("Interrupted while waiting for next replication entry batch", e);
+          Thread.currentThread().interrupt();
+        }
+      }
+    }
+
+    /**
+     * @return whether the thread is running
+     */
+    public boolean isRunning() {
+      return isRunning && !isInterrupted();
+    }
+
+    /**
+     * @param running the running to set
+     */
+    public void setRunning(boolean running) {

Review comment:
       s/setRunning/ ... start/stop?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,

Review comment:
       Why not pass regionInfo instead of table, then encoded name and replica id?

##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestCatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,42 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.junit.ClassRule;
+import org.junit.experimental.categories.Category;
+
+/**
+ * Tests CatalogReplicaReplicationEndpoint class by setting up region replicas and verifying
+ * async wal replication replays the edits to the secondary region in various scenarios.
+ */
+@Category({ LargeTests.class })
+public class TestCatalogReplicaReplicationEndpoint extends
+  TestMetaRegionReplicaReplicationEndpoint {

Review comment:
       cool

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {

Review comment:
       We'll never get a meta edit in a bunch of entries?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();
+            entries = null;
+          } catch (InterruptedException e) {
+            // restore the interrupted state
+            Thread.currentThread().interrupt();
+            continue;
+          } catch (ExecutionException e) {
+            Throwable cause = e.getCause();
+            LOG.warn("Failed to replicate {} entries for region  of meta table", entries.size());
+            LOG.warn("Cause: " + cause);

Review comment:
       Make one log line, no need of two.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java
##########
@@ -63,6 +64,13 @@
   CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, byte[] row,
       List<Entry> entries, int replicaId, int numRetries, long operationTimeoutNs);
 
+  /**
+   * Replicate wal edits for replica regions, the location for the replica is already included.
+   * The return value is the edits we skipped, as the original return value is useless.
+   */
+  CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, List<Entry> entries,
+    int replicaId, int numRetries, long operationTimeoutNs, HRegionLocation loc);

Review comment:
       Yeah, operation timeout as a param and not as config read internally?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already

Review comment:
       Put this difficult stuff into a method so contained?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);

Review comment:
       s/Try/Tried/

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();

Review comment:
       No timeout?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,495 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.

Review comment:
       Oh, you talk about this below.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);

Review comment:
       Oh, no need of the work 'region' in the LOG... what else would we be registering? So 'region' is redundant.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);

Review comment:
       s/Try/Tried/

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALEdit.java
##########
@@ -320,6 +323,15 @@ public static FlushDescriptor getFlushDescriptor(Cell cell) throws IOException {
         FlushDescriptor.parseFrom(CellUtil.cloneValue(cell)): null;
   }
 
+  public static FlushDescriptor getCommitFlushDescriptor(Cell cell) throws IOException {

Review comment:
       Would it help?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();
+            entries = null;
+          } catch (InterruptedException e) {
+            // restore the interrupted state
+            Thread.currentThread().interrupt();
+            continue;
+          } catch (ExecutionException e) {
+            Throwable cause = e.getCause();
+            LOG.warn("Failed to replicate {} entries for region  of meta table", entries.size());
+            LOG.warn("Cause: " + cause);
+            location = null;
+            reload = true;
+          }
+
+        } catch (InterruptedException | ReplicationRuntimeException e) {
+          // It is interrupted and needs to quit.
+          LOG.warn("Interrupted while waiting for next replication entry batch", e);
+          Thread.currentThread().interrupt();
+        }
+      }
+    }
+
+    /**
+     * @return whether the thread is running
+     */
+    public boolean isRunning() {
+      return isRunning && !isInterrupted();
+    }
+
+    /**
+     * @param running the running to set
+     */
+    public void setRunning(boolean running) {

Review comment:
       Check if already running before setting flag?  Might have to synchronize the check/set or use atomicboolean

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));

Review comment:
       nit




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502668082



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();
+            entries = null;
+          } catch (InterruptedException e) {
+            // restore the interrupted state
+            Thread.currentThread().interrupt();
+            continue;
+          } catch (ExecutionException e) {
+            Throwable cause = e.getCause();
+            LOG.warn("Failed to replicate {} entries for region  of meta table", entries.size());
+            LOG.warn("Cause: " + cause);
+            location = null;
+            reload = true;
+          }
+
+        } catch (InterruptedException | ReplicationRuntimeException e) {
+          // It is interrupted and needs to quit.
+          LOG.warn("Interrupted while waiting for next replication entry batch", e);
+          Thread.currentThread().interrupt();
+        }
+      }
+    }
+
+    /**
+     * @return whether the thread is running
+     */
+    public boolean isRunning() {
+      return isRunning && !isInterrupted();
+    }
+
+    /**
+     * @param running the running to set
+     */
+    public void setRunning(boolean running) {

Review comment:
       Changed to stopShipper() as stop() cannot be overridden. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-707209866


   Will update a new addressing review comments today.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706460208


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 22s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 40s |  HBASE-18070 passed  |
   | +1 :green_heart: |  compile  |   1m 12s |  HBASE-18070 passed  |
   | +1 :green_heart: |  shadedjars  |   7m 21s |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 42s |  hbase-server in HBASE-18070 failed.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 34s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 10s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 10s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m 22s |  patch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 41s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 191m 28s |  hbase-server in the patch passed.  |
   |  |   | 221m 23s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux a4d4485dc8e0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | Default Java | 2020-01-14 |
   | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
   | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/testReport/ |
   | Max. process+thread count | 3042 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706408254


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  9s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 47s |  HBASE-18070 passed  |
   | +1 :green_heart: |  compile  |   1m 12s |  HBASE-18070 passed  |
   | +1 :green_heart: |  shadedjars  |   7m 22s |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 43s |  hbase-server in HBASE-18070 failed.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 33s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 19s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 19s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   8m  0s |  patch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 44s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 203m 49s |  hbase-server in the patch passed.  |
   |  |   | 235m 32s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux e96fc7ea3ca2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | Default Java | 2020-01-14 |
   | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
   | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/testReport/ |
   | Max. process+thread count | 3026 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502665389



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();

Review comment:
       timeout is not needed, as inside replicate(), it already has timeout.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();
+            entries = null;
+          } catch (InterruptedException e) {
+            // restore the interrupted state
+            Thread.currentThread().interrupt();
+            continue;
+          } catch (ExecutionException e) {
+            Throwable cause = e.getCause();
+            LOG.warn("Failed to replicate {} entries for region  of meta table", entries.size());
+            LOG.warn("Cause: " + cause);

Review comment:
       will do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706407940


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m 18s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  7s |  HBASE-18070 passed  |
   | +1 :green_heart: |  compile  |   0m 58s |  HBASE-18070 passed  |
   | +1 :green_heart: |  shadedjars  |   7m 10s |  branch has no errors when building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  HBASE-18070 passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 50s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 59s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 59s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m 11s |  patch has no errors when building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 205m 49s |  hbase-server in the patch failed.  |
   |  |   | 234m 33s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux d4a1154539c8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | Default Java | 1.8.0_232 |
   | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/testReport/ |
   | Max. process+thread count | 3712 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502701555



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already

Review comment:
       Hmm, this is from region replica. Will try to do something.
   The default is 15, multiply is 3, so it will try 5 times.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] saintstack commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

saintstack commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502685087



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionReplicaReplayRetryingCaller.java
##########
@@ -63,6 +63,8 @@
 
   private final int replicaId;
 
+  private final HRegionLocation loc;

Review comment:
       The RegionLocator would have cached the location too, right? Just get it from there?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502694878



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {

Review comment:
       Could happen in the future, but do not want to go along this route for now as it is unnecessary makes coding complicated.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706469217


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 29s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 40s |  HBASE-18070 passed  |
   | +1 :green_heart: |  checkstyle  |   1m  8s |  HBASE-18070 passed  |
   | +1 :green_heart: |  spotbugs  |   2m  1s |  HBASE-18070 passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 22s |  the patch passed  |
   | -0 :warning: |  checkstyle  |   1m  3s |  hbase-server: The patch generated 18 new + 27 unchanged - 0 fixed = 45 total (was 27)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  hadoopcheck  |  11m 22s |  Patch does not cause any errors with Hadoop 3.1.2 3.2.1.  |
   | -1 :x: |  spotbugs  |   2m 11s |  hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 15s |  The patch does not generate ASF License warnings.  |
   |  |   |  32m 48s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hbase-server |
   |  |  Unread field:CatalogReplicaReplicationEndpoint.java:[line 349] |
   |  |  Unread field:CatalogReplicaReplicationEndpoint.java:[line 364] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
   | uname | Linux 828d6fa5666b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
   | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html |
   | Max. process+thread count | 94 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706326780


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   2m 32s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  5s |  HBASE-18070 passed  |
   | +1 :green_heart: |  checkstyle  |   1m 14s |  HBASE-18070 passed  |
   | +1 :green_heart: |  spotbugs  |   2m  9s |  HBASE-18070 passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 44s |  the patch passed  |
   | -0 :warning: |  checkstyle  |   1m 10s |  hbase-server: The patch generated 28 new + 27 unchanged - 0 fixed = 55 total (was 27)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  hadoopcheck  |  12m 21s |  Patch does not cause any errors with Hadoop 3.1.2 3.2.1.  |
   | -1 :x: |  spotbugs  |   2m 18s |  hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 12s |  The patch does not generate ASF License warnings.  |
   |  |   |  37m 29s |   |
   
   
   | Reason | Tests |
   |-------:|:------|
   | FindBugs | module:hbase-server |
   |  |  Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.hbase.replication.regionserver.CatalogReplicaReplicationEndpoint$CatalogReplicaShipper.lambda$replicate$1(CompletableFuture, Long, Throwable)  At CatalogReplicaReplicationEndpoint.java:then immediately reboxed in org.apache.hadoop.hbase.replication.regionserver.CatalogReplicaReplicationEndpoint$CatalogReplicaShipper.lambda$replicate$1(CompletableFuture, Long, Throwable)  At CatalogReplicaReplicationEndpoint.java:[line 411] |
   |  |  Unread field:CatalogReplicaReplicationEndpoint.java:[line 365] |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
   | uname | Linux 6b45a97d1407 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
   | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html |
   | Max. process+thread count | 84 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502644247



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java
##########
@@ -63,6 +64,13 @@
   CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, byte[] row,
       List<Entry> entries, int replicaId, int numRetries, long operationTimeoutNs);
 
+  /**
+   * Replicate wal edits for replica regions, the location for the replica is already included.
+   * The return value is the edits we skipped, as the original return value is useless.
+   */
+  CompletableFuture<Long> replay(TableName tableName, byte[] encodedRegionName, List<Entry> entries,
+    int replicaId, int numRetries, long operationTimeoutNs, HRegionLocation loc);

Review comment:
       Yeah, thought about it. It is messy with my current implementation. The whole idea is that since shipper only ships to one replica region sever and passing in location will save extra cost to look up location. 
   For the case of region move case, the retry does not  work in this implementation as location is fixed. I think it will be better just let vanilla implementation to take care of it. Let me clean this up and update the patch.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706417044


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 28s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 46s |  HBASE-18070 passed  |
   | +1 :green_heart: |  checkstyle  |   1m  6s |  HBASE-18070 passed  |
   | +1 :green_heart: |  spotbugs  |   2m  0s |  HBASE-18070 passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 26s |  the patch passed  |
   | -0 :warning: |  checkstyle  |   1m  4s |  hbase-server: The patch generated 28 new + 27 unchanged - 0 fixed = 55 total (was 27)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  hadoopcheck  |  11m 17s |  Patch does not cause any errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   2m  8s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 14s |  The patch does not generate ASF License warnings.  |
   |  |   |  32m 48s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
   | uname | Linux 6d4d12c7e3d9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
   | Max. process+thread count | 94 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502695123



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);

Review comment:
       Will update in the next update.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-710482867


   > First, the issue number is incorrect, it should be HBASE-25125, not HBASE-25215.
   
   Thanks Duo, will correct.
   > 
   > What I concern here is that, this is only an enhancement right? The RegionReplicaReplicationEndpoint could also be used to replicate meta edits? The only disadvantage is that, the speed will be slowed down if there is a slow replica, but I do not think it is a blocker for meta replica? Could we make the meta replicas feature work first so we can merge HBASE-18070 back and then let's do the enhancement?
   
   Yeah, agreed here. Client side work is in progress, that is the reason, review comments from Stack has not been addressed in this week.
   
   > And in general, I do not like that we have a specialized implementation for meta replica inside the region replica framework. They are almost the same so they should share the same code base. I think this enhancement could also be applied to normal RegionReplicaReplicationEndpoint, what we need is the config for the max distance between different secondary replicas. For normal region replicas, the value will be small, which means we will soon block the replication if any replicas are slow, to save memory. For meta replicas, we could have a large default value to let the good replicas catch up fast.
   > Anyway, my biggest concern here is that, this is not a blocker to prevent us merge HBASE-18070 back, as people said we want this feature in 2.4, we'd better focus on other blocker issues, such as how to let client go to secondary meta replicas as much as possible. Feel free to correct me if I'm wrong.
   
   I moved your comments in the design doc, we put some thoughts there already, happy to continue the discussion there.
   
   https://docs.google.com/document/d/1jJWVc-idHhhgL4KDRpjMsQJKCl_NRaCLGiH3Wqwd3O8/edit#heading=h.8n622lgkwlai
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache9 commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache9 commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r505536165



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java
##########
@@ -40,6 +39,10 @@
   private static final Logger LOG = LoggerFactory.getLogger(BaseReplicationEndpoint.class);
   public static final String REPLICATION_WALENTRYFILTER_CONFIG_KEY
       = "hbase.replication.source.custom.walentryfilters";
+  // Can be configured differently than hbase.client.retries.number
+  public static final String CLIENT_RETRIES_NUMBER =
+    "hbase.region.replica.replication.client.retries.number";

Review comment:
       Why move it here? It is base replication endpoint, not BaseRegionReplicaReplicationEndpoint, I do not think we should put region replica specific configs in this general class.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,459 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ * One sending thread which loops through shipper's queue and send out the walEdits.
+ * One receiving thread which will wait for futures, once it is ready, it processes that
+ * future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {

Review comment:
       Why not just call remove and check the return value?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {

Review comment:
       Second this. Let's keep the same style.

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,459 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ * One sending thread which loops through shipper's queue and send out the walEdits.
+ * One receiving thread which will wait for futures, once it is ready, it processes that
+ * future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.stopShipper();
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}", regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);

Review comment:
       Could we have better variable names here?

##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,459 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ * One sending thread which loops through shipper's queue and send out the walEdits.
+ * One receiving thread which will wait for futures, once it is ready, it processes that
+ * future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {

Review comment:
       Could this be done in a WALEntryFilter? IIRC you've done a prior work to be able to config different WALEntryFilters for replication endpoint or replication source?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502693001



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java
##########
@@ -40,6 +39,10 @@
   private static final Logger LOG = LoggerFactory.getLogger(BaseReplicationEndpoint.class);
   public static final String REPLICATION_WALENTRYFILTER_CONFIG_KEY
       = "hbase.replication.source.custom.walentryfilters";
+  // Can be configured differently than hbase.client.retries.number
+  public static final String CLIENT_RETRIES_NUMBER =
+    "hbase.region.replica.replication.client.retries.number";

Review comment:
       This is an existing config, I just moved from RegionReplicaReplicationEndpoint.java  to this place. Let me check if this  has been release noted.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502627479



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionReplicaReplayRetryingCaller.java
##########
@@ -63,6 +63,8 @@
 
   private final int replicaId;
 
+  private final HRegionLocation loc;

Review comment:
       Yeah, cause the shipper already caches the location for the replica region.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706490502


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 23s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 29s |  HBASE-18070 passed  |
   | +1 :green_heart: |  compile  |   1m 11s |  HBASE-18070 passed  |
   | +1 :green_heart: |  shadedjars  |   7m 22s |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 41s |  hbase-server in HBASE-18070 failed.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 30s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  9s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m  9s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m 13s |  patch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 39s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 191m 13s |  hbase-server in the patch passed.  |
   |  |   | 220m 41s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux f5a81c74a8f1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | Default Java | 2020-01-14 |
   | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
   | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/testReport/ |
   | Max. process+thread count | 2952 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502693297



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,495 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.

Review comment:
       Yeah, will work on the improvement later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502702094



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();

Review comment:
       This is the connection in the region server, timeout and retries will be passed in when replay() is called.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502694988



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)

Review comment:
       these are two separate fields.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

Apache-HBase commented on pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#issuecomment-706485479


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 28s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-18070 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 46s |  HBASE-18070 passed  |
   | +1 :green_heart: |  compile  |   0m 57s |  HBASE-18070 passed  |
   | +1 :green_heart: |  shadedjars  |   6m 34s |  branch has no errors when building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  HBASE-18070 passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 30s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 56s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 56s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m 34s |  patch has no errors when building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 143m 55s |  hbase-server in the patch failed.  |
   |  |   | 170m 12s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/2522 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 4e0092784190 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-18070 / 4e47554af8 |
   | Default Java | 1.8.0_232 |
   | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/testReport/ |
   | Max. process+thread count | 4422 (vs. ulimit of 30000) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2522/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502696137



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {

Review comment:
       Yeah, meta edit is always its own edit.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hbase] huaxiangsun commented on a change in pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…

[GitBox <gi...@apache.org>]

huaxiangsun commented on a change in pull request #2522:
URL: https://github.com/apache/hbase/pull/2522#discussion_r502667323



##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicaReplicationEndpoint.java
##########
@@ -0,0 +1,493 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.replication.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CatalogFamilyFormat;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.NotServingRegionException;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTableRegionLocator;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.exceptions.RegionMovedException;
+import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.replication.BaseReplicationEndpoint;
+import org.apache.hadoop.hbase.replication.WALEntryFilter;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
+import org.apache.hadoop.hbase.wal.WAL.Entry;
+import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link org.apache.hadoop.hbase.replication.ReplicationEndpoint} endpoint which receives the WAL
+ * edits from the WAL, and sends the edits to replicas of regions.
+ * This EndPoint is specially for system tables. It does a few things:
+ *
+ * 1. For each replica region, there is a shipper to ship edits for this region. Internally,
+ * the shipper maintains a in-memory queue, where edits from wal log are queued in this
+ * in-memory queue.
+ * 2. Each shipper will read edits from its queue and ship edits to the specific replica region.
+ * The pace of shippers are independent from each other, so they are not synced.
+ * 3. In case of flush event COMMIT_FLUSH, edits in the slow shipper will be discarded as
+ * those edits are already in flushed hfiles. This also prevents the queue grow out-of-bound
+ * when the replica region server is slow processing edits.
+ * 4. register/deregister region is based on region events from wal log.
+ * 5. CatalogReplicaReplicationEndpoint is based on BaseReplicationEndpoint as it does not need
+ * zookeeper support.
+ *
+ * TODO: the threading model for shippers needs to be improved. Right now, each shipper maps to
+ * one thread, which is not efficient. It needs to be improved so two threads are needed.
+ *  One sending thread which loops through shipper's queue and send out the walEdits.
+ *  One receiving thread which will wait for futures, once it is ready, it processes that
+ *  future accordingly.
+ */
+@InterfaceAudience.Private public class CatalogReplicaReplicationEndpoint
+  extends BaseReplicationEndpoint {
+
+  private static final Logger LOG =
+    LoggerFactory.getLogger(CatalogReplicaReplicationEndpoint.class);
+
+  private Configuration conf;
+  private AsyncClusterConnection connection;
+
+  private int numRetries;
+  private long operationTimeoutNs;
+  private long shippedEdits;
+
+  // Track primary regions under replication.
+  ConcurrentHashMap<String, CatalogReplicaShipper[]> primaryRegionsUnderReplication;
+
+  private byte[][] replicatedFamilies;
+
+  public CatalogReplicaReplicationEndpoint() {
+    // TODO: right now, it only allow family info to be replicated.
+    // This need to be changed in the future.
+    replicatedFamilies = new byte[][] { HConstants.CATALOG_FAMILY };
+    primaryRegionsUnderReplication = new ConcurrentHashMap<>();
+  }
+
+  public long getShippedEdits() {
+    return shippedEdits;
+  }
+
+  @Override public void init(Context context) throws IOException {
+    super.init(context);
+    this.conf = context.getConfiguration();
+    // HRS multiplies client retries by 10 globally for meta operations, but we do not want this.
+    // We are resetting it here because we want default number of retries (35) rather than 10 times
+    // that which makes very long retries for disabled tables etc.
+    int defaultNumRetries = conf.getInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER,
+      HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER);
+    if (defaultNumRetries > 10) {
+      int mult = conf.getInt(HConstants.HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER,
+        HConstants.DEFAULT_HBASE_CLIENT_SERVERSIDE_RETRIES_MULTIPLIER);
+      defaultNumRetries = defaultNumRetries / mult; // reset if HRS has multiplied this already
+    }
+    this.numRetries = conf.getInt(CLIENT_RETRIES_NUMBER, defaultNumRetries);
+    // use the regular RPC timeout for replica replication RPC's
+    this.operationTimeoutNs = TimeUnit.MILLISECONDS.toNanos(conf
+      .getInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT,
+        HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT));
+    this.connection = context.getServer().getAsyncClusterConnection();
+  }
+
+  /**
+   * returns true if the specified entry must be replicated. We should always replicate meta
+   * operations (e.g. flush) and use the configured family to filter out edits which do not
+   * need to be replicated.
+   */
+  private boolean requiresReplication(final WALEdit edit) {
+
+    // TODO: let meta edits go through. Need to filter out flush events which are not for
+    // targetting families.
+    if (edit.isMetaEdit()) {
+      return true;
+    }
+    // empty edit does not need to be replicated
+    if (edit.isEmpty()) {
+      return false;
+    }
+
+    for (byte[] fam : replicatedFamilies) {
+      if (edit.getFamilies().contains(fam)) {
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  // Register a new primary region to the endpoint when a region is opened, it will setup its
+  // shipper to replicate its wal edits to replica regions.
+  private void registerRegion(final String encodedName, final byte[] regionName)
+    throws IOException {
+    // Make sure it does not exist.
+    LOG.info("Register region {}", encodedName);
+
+    if (primaryRegionsUnderReplication.get(encodedName) != null) {
+      LOG.warn("Try to register an existing regoin {}", encodedName);
+      return;
+    }
+
+    RegionInfo regionInfo = CatalogFamilyFormat.parseRegionInfoFromRegionName(regionName);
+
+    // TODO: meta replicas is not at the table descriptor at this moment.
+    int numReplicas =
+      conf.getInt(HConstants.META_REPLICAS_NUM, HConstants.DEFAULT_META_REPLICA_NUM);
+
+    if (numReplicas <= 1) {
+      LOG.warn("Trying to register region {} without replica enabled", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = new CatalogReplicaShipper[numReplicas - 1];
+
+    for (int i = 1; i < numReplicas; i++) {
+      // It leaves each ReplicaShipper to resolve its replica region location.
+      replicaShippers[i - 1] = new CatalogReplicaShipper(this, this.connection, null,
+        RegionInfoBuilder.newBuilder(regionInfo).setReplicaId(i).build());
+
+      replicaShippers[i - 1].start();
+    }
+
+    primaryRegionsUnderReplication.put(encodedName, replicaShippers);
+  }
+
+  // Deregister a primary region from the endpoint when a region is closed, it will clean up all
+  // resources allocated for this region.
+  private void unregisterRegion(final String encodedName) {
+    LOG.info("Unregister region {}", encodedName);
+    // Make sure it does exist.
+    if (primaryRegionsUnderReplication.get(encodedName) == null) {
+      LOG.warn("Try to unregister a unregistered region {}", encodedName);
+      return;
+    }
+
+    CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.remove(encodedName);
+
+    for (CatalogReplicaShipper s : replicaShippers) {
+      s.setRunning(false);
+    }
+  }
+
+  @Override public boolean replicate(ReplicateContext replicateContext) {
+    Map<String, List<List<Entry>>> encodedRegionName2Entries = new TreeMap<>();
+    long skippedEdits = 0;
+
+    // First, separate edits based on primary region.
+    for (Entry entry : replicateContext.getEntries()) {
+      WALEdit edit = entry.getEdit();
+      if (!requiresReplication(edit)) {
+        skippedEdits++;
+        continue;
+      }
+      shippedEdits++;
+
+      try {
+        // If an edit is a meta regionevent edit, we need to check a few things.
+        //    1. If it is a region_open event, it needs to register this region.
+        //    2. If it is a region_close event, it needs to deregister this region.
+        if (edit.isMetaEdit()) {
+          WALProtos.RegionEventDescriptor regionEvent =
+            WALEdit.getRegionEventDescriptor(edit.getCells().get(0));
+
+          if (regionEvent != null) {
+            if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_CLOSE) {
+              unregisterRegion(Bytes.toString(entry.getKey().getEncodedRegionName()));
+            } else if (regionEvent.getEventType()
+              == WALProtos.RegionEventDescriptor.EventType.REGION_OPEN) {
+              try {
+                registerRegion(Bytes.toString(regionEvent.getEncodedRegionName().toByteArray()),
+                  regionEvent.getRegionName().toByteArray());
+              } catch (IOException e) {
+                LOG.warn("REGION_OPEN, invalid region name {}",
+                  regionEvent.getRegionName());
+              }
+            }
+          }
+        }
+        // add entry to the lists, please note one thing, if it is flush event, it needs to be
+        // in its own list, it makes the cleanup of shipper's queue much easier.
+        String encodeName = Bytes.toString(entry.getKey().getEncodedRegionName());
+        List<List<Entry>> ll = encodedRegionName2Entries.get(encodeName);
+        if (ll == null) {
+          ll = new ArrayList<>();
+          List<Entry> l = new ArrayList();
+          l.add(entry);
+          ll.add(l);
+          encodedRegionName2Entries.put(encodeName, ll);
+        } else {
+          // Check if it can be merged into the last set.
+          List<Entry> l = ll.get(ll.size() - 1);
+          WALProtos.FlushDescriptor flushDesc =
+            WALEdit.getFlushDescriptor(l.get(0).getEdit().getCells().get(0));
+          if (flushDesc != null) {
+            // Flush events need to be in their own list
+            List<Entry> newList = new ArrayList<>();
+            newList.add(entry);
+            ll.add(newList);
+          } else {
+            // Check if the entry itself is a flush event.
+            flushDesc = WALEdit.getFlushDescriptor(edit.getCells().get(0));
+            if (flushDesc != null) {
+              // Flush events need to be in their own list
+              List<Entry> newList = new ArrayList<>();
+              newList.add(entry);
+              ll.add(newList);
+            } else {
+              l.add(entry);
+            }
+          }
+        }
+      } catch (IOException ioe) {
+        LOG.warn("Failed to parse {}", entry.getEdit().getCells().get(0));
+        skippedEdits++;
+      }
+    }
+
+    // queue edits to shippers.
+
+    for (Map.Entry<String, List<List<Entry>>> entry : encodedRegionName2Entries.entrySet()) {
+      CatalogReplicaShipper[] replicaShippers = primaryRegionsUnderReplication.get(entry.getKey());
+      if (replicaShippers != null) {
+        for (List<Entry> l : entry.getValue()) {
+          for (CatalogReplicaShipper shipper : replicaShippers) {
+            shipper.addEditEntries(l);
+          }
+        }
+      } else {
+        skippedEdits ++;
+      }
+    }
+    ctx.getMetrics().incrLogEditsFiltered(skippedEdits);
+
+    return true;
+  }
+
+  @Override public UUID getPeerUUID() {
+    return this.ctx.getClusterId();
+  }
+
+  @Override public boolean canReplicateToSameCluster() {
+    return true;
+  }
+
+  @Override protected WALEntryFilter getScopeWALEntryFilter() {
+    // we do not care about scope. We replicate everything.
+    return null;
+  }
+
+  @Override public void start() {
+    startAsync();
+  }
+
+  @Override public void stop() {
+    // Stop shippers
+    for (Map.Entry<String, CatalogReplicaShipper[]> e : primaryRegionsUnderReplication.entrySet()) {
+      for (CatalogReplicaShipper s : e.getValue()) {
+        s.setRunning(false);
+      }
+    }
+    stopAsync();
+  }
+
+  @Override protected void doStart() {
+    notifyStarted();
+  }
+
+  @Override protected void doStop() {
+    notifyStopped();
+  }
+
+  public int getNumRetries() {
+    return numRetries;
+  }
+
+  public long getOperationTimeoutNs() {
+    return operationTimeoutNs;
+  }
+
+  private static class CatalogReplicaShipper extends Thread {
+    private final CatalogReplicaReplicationEndpoint endpoint;
+    private LinkedBlockingQueue<List<Entry>> queue;
+    private final AsyncClusterConnection conn;
+    //Indicates whether this particular worker is running
+    private volatile boolean isRunning = true;
+    private HRegionLocation location = null;
+    private RegionInfo regionInfo;
+    private long operationTimeoutNs;
+    private AsyncTableRegionLocator locator;
+
+    CatalogReplicaShipper(final CatalogReplicaReplicationEndpoint endpoint,
+      final AsyncClusterConnection connection, final HRegionLocation loc,
+      final RegionInfo regionInfo) {
+      this.endpoint = endpoint;
+      this.conn = connection;
+      this.location = loc;
+      this.regionInfo = regionInfo;
+      this.queue = new LinkedBlockingQueue<>();
+      this.operationTimeoutNs = endpoint.getOperationTimeoutNs();
+      locator = conn.getRegionLocator(regionInfo.getTable());
+    }
+
+    private void cleanQueue(final long startFlushSeqOpId) {
+      // remove entries with seq# less than startFlushSeqOpId
+
+      // TODO: It assumes one family in the queue.
+      // In the future, if more families are involved, it needs to have families into
+      // consideration.
+      queue.removeIf(n -> (n.get(0).getKey().getSequenceId() < startFlushSeqOpId));
+    }
+
+    public void addEditEntries(final List<Entry> entries) {
+      try {
+        if ((entries.size() == 1) && entries.get(0).getEdit().isMetaEdit()) {
+          try {
+            // When it is a COMMIT_FLUSH event, it can remove all edits in the queue whose
+            // seq# is less flushSeq#. Those events include flush events and
+            // OPEN_REGION/CLOSE_REGION events, as they work like flush events.
+            WALProtos.FlushDescriptor flushDesc =
+              WALEdit.getCommitFlushDescriptor(entries.get(0).getEdit().getCells().get(0));
+            if (flushDesc != null) {
+              cleanQueue(flushDesc.getFlushSequenceNumber());
+            }
+          } catch (IOException ioe) {
+            LOG.warn("Failed to parse {}", entries.get(0).getEdit().getCells().get(0));
+          }
+        }
+        queue.put(entries);
+      } catch (InterruptedException e) {
+
+      }
+    }
+
+    private CompletableFuture<Long> replicate(List<Entry> entries) {
+      CompletableFuture<Long> future = new CompletableFuture<>();
+      FutureUtils.addListener(conn
+        .replay(regionInfo.getTable(), regionInfo.getEncodedNameAsBytes(), entries,
+          regionInfo.getReplicaId(), 1, operationTimeoutNs, location), (r, e) -> {
+        if (e != null) {
+          LOG.warn("Failed to replicate to {}", regionInfo);
+          LOG.warn("Exception: " + e);
+          future.completeExceptionally(e);
+        } else {
+          future.complete(r);
+        }
+      });
+
+      return future;
+    }
+
+    @Override public final void run() {
+      boolean reload = false;
+      boolean sleep = false;
+      List<Entry> entries = null;
+      while (isRunning()) { // we only loop back here if something fatal happened to our stream
+        if (location == null) {
+          // If location is null, keep searching for it. This applies to replica region server
+          // crashes, and waiting to be assigned. reload flag will be set accordingly in case of
+          // error.
+          try {
+            if (sleep) {
+              Thread.sleep(100);
+            }
+            location =
+              locator.getRegionLocation(regionInfo.getStartKey(), regionInfo.getReplicaId(), reload)
+                .get();
+            sleep = false;
+          } catch (ExecutionException ee) {
+            LOG.warn("Error getting location for meta region " + regionInfo + ", " + ee);
+            reload = true;
+            sleep = true;
+            continue;
+          } catch (InterruptedException ie) {
+            LOG.warn("Interrupted while getting location for meta region " +
+              regionInfo + ", " + ie);
+            Thread.currentThread().interrupt();
+          }
+        }
+        try {
+          if (entries == null) {
+            entries = queue.poll(20000, TimeUnit.MILLISECONDS);
+          }
+          if (entries == null) {
+            continue;
+          }
+
+          CompletableFuture<Long> future = replicate(entries);
+
+          try {
+            future.get();
+            entries = null;
+          } catch (InterruptedException e) {
+            // restore the interrupted state
+            Thread.currentThread().interrupt();
+            continue;
+          } catch (ExecutionException e) {
+            Throwable cause = e.getCause();
+            LOG.warn("Failed to replicate {} entries for region  of meta table", entries.size());
+            LOG.warn("Cause: " + cause);
+            location = null;
+            reload = true;
+          }
+
+        } catch (InterruptedException | ReplicationRuntimeException e) {
+          // It is interrupted and needs to quit.
+          LOG.warn("Interrupted while waiting for next replication entry batch", e);
+          Thread.currentThread().interrupt();
+        }
+      }
+    }
+
+    /**
+     * @return whether the thread is running
+     */
+    public boolean isRunning() {
+      return isRunning && !isInterrupted();
+    }
+
+    /**
+     * @param running the running to set
+     */
+    public void setRunning(boolean running) {

Review comment:
       Once the shipper is stopped, it cannot be restarted. Will change the method to stop(). I think there is no need to check the flag before setting if it is changed to stop(). volatile is good enough.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org