[jira] [Commented] (HBASE-21406) "status 'replication'" should not show SINK if the cluster does not act as sink

["Wellington Chevreuil (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674255#comment-16674255 ] 

Wellington Chevreuil commented on HBASE-21406:
----------------------------------------------

Added initial patch proposal for *branch-1.* Idea here is to not show stats for SINK, until it has not received any edits. Added additional metrics showing the sink startup time, something as below:
{noformat}
SINK  : TimeStampStarted=1541292912227, Waiting for OPs...{noformat}
 
BTW, while testing, noticed additional issues with metrics for source on current branch-1 version:
1) Once started and while no OP eligible for replication occurs, TimeStampsOfLastShippedOp shows "Thu Jan 01 01:00:00 GMT 1970", and huge Replication Lag is accounted. This seems to be due HBASE-15995, which removed code on ReplicationSource class that initializes AgeOfLastShippedOp to the startup time:

{noformat}
-          // Reset the sleep multiplier if nothing has actually gone wrong
-          if (!gotIOE) {
-            sleepMultiplier = 1;
-            // if there was nothing to ship and it's not an error
-            // set "ageOfLastShippedOp" to <now> to indicate that we're current
-            metrics.setAgeOfLastShippedOp(EnvironmentEdgeManager.currentTime(), walGroupId);
+          WALEntryBatch entryBatch = entryReader.take();
+          for (Map.Entry<String, Long> entry : entryBatch.getLastSeqIds().entrySet()) {
+            waitingUntilCanPush(entry);
{noformat}

2) After source gets OPs to replicate and successfully ships it to target, source metrics then keep showing lags, even if there was no new edits to replicate. This is also wrong, and was apparently introduced by changes from HBASE-15093, which has modified the way log que size is accounted, and replication lag calculation logic seems to rely on the log queue size in ReplicationLoad:
{noformat}
      long ageOfLastShippedOp = sm.getAgeOfLastShippedOp();
      int sizeOfLogQueue = sm.getSizeOfLogQueue();
      long timeStampOfLastShippedOp = sm.getTimeStampOfLastShippedOp();
      long replicationLag;
      long timePassedAfterLastShippedOp =
          EnvironmentEdgeManager.currentTime() - timeStampOfLastShippedOp;
      if (sizeOfLogQueue != 0) {
        // err on the large side
        replicationLag = Math.max(ageOfLastShippedOp, timePassedAfterLastShippedOp);
      } else if (timePassedAfterLastShippedOp < 2 * ageOfLastShippedOp) {
        replicationLag = ageOfLastShippedOp; // last shipped happen recently
      } else {
        // last shipped may happen last night,
        // so NO real lag although ageOfLastShippedOp is non-zero
        replicationLag = 0;
      }
{noformat}

I'll be opening another jira to fix the source metrics issues mentioned above.

> "status 'replication'" should not show SINK if the cluster does not act as sink
> -------------------------------------------------------------------------------
>
>                 Key: HBASE-21406
>                 URL: https://issues.apache.org/jira/browse/HBASE-21406
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Daisuke Kobayashi
>            Assignee: Wellington Chevreuil
>            Priority: Minor
>         Attachments: HBASE-21406-branch-1.001.patch, Screen Shot 2018-10-31 at 18.12.54.png
>
>
> When replicating in 1 way, from source to target, {{status 'replication'}} on source always dumps SINK with meaningless metrics. It only makes sense when running the command on target cluster.
> {{status 'replication'}} on source, for example. {{AgeOfLastAppliedOp}} is always zero and {{TimeStampsOfLastAppliedOp}} does not get updated from the time the RS started since it's not acting as sink.
> {noformat}
>     source-1.com
>        SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=0, TimeStampsOfLastShippedOp=Mon Oct 29 23:44:14 PDT 2018, Replication Lag=0
>        SINK  : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Oct 25 23:56:53 PDT 2018
> {noformat}
> {{status 'replication'}} on target works as expected. SOURCE is empty as it's not acting as source:
> {noformat}
>     target-1.com
>        SOURCE:
>        SINK  : AgeOfLastAppliedOp=70, TimeStampsOfLastAppliedOp=Mon Oct 29 23:44:08 PDT 2018
> {noformat}
> This is because {{getReplicationLoadSink}}, called in {{admin.rb}}, always returns a value (not null).
> 1.X
> https://github.com/apache/hbase/blob/rel/1.4.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L194-L204
> 2.X
> https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L392-L399



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)