You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2018/12/05 15:35:00 UTC
[jira] [Commented] (HBASE-21505) Several inconsistencies on
information reported for Replication Sources by hbase shell status
'replication' command.
[ https://issues.apache.org/jira/browse/HBASE-21505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710239#comment-16710239 ]
Hadoop QA commented on HBASE-21505:
-----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 19s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue} hbase-hadoop2-compat in master has 18 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m 24s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 18s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 42s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 18s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 42s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 42s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} hbase-hadoop2-compat: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s{color} | {color:red} hbase-client: The patch generated 5 new + 185 unchanged - 0 fixed = 190 total (was 185) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 15s{color} | {color:red} hbase-server: The patch generated 19 new + 86 unchanged - 2 fixed = 105 total (was 88) {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 8s{color} | {color:red} The patch generated 55 new + 405 unchanged - 9 fixed = 460 total (was 414) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange} 0m 4s{color} | {color:orange} The patch generated 3 new + 748 unchanged - 1 fixed = 751 total (was 749) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 2m 5s{color} | {color:red} patch has 14 errors when building our shaded downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 21s{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 38s{color} | {color:red} The patch causes 14 errors with Hadoop v3.0.0. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 19s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 31s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 18s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 29s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 48s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 53s{color} | {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 56s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 58s{color} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestShell |
| | hadoop.hbase.client.TestAdminShell2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21505 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12950701/HBASE-21505-master.001.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc rubocop ruby_lint |
| uname | Linux 62d0de0a3c64 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
| git revision | master / 8bf966c8e9 |
| maven | version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| mvninstall | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-mvninstall-root.txt |
| compile | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-compile-hbase-client.txt |
| compile | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-compile-hbase-server.txt |
| cc | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-compile-hbase-client.txt |
| cc | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-compile-hbase-server.txt |
| javac | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-compile-hbase-client.txt |
| javac | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-compile-hbase-server.txt |
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/diff-checkstyle-hbase-hadoop2-compat.txt |
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/diff-checkstyle-hbase-client.txt |
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/diff-checkstyle-hbase-server.txt |
| rubocop | v0.60.0 |
| rubocop | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/diff-patch-rubocop.txt |
| ruby-lint | v2.3.1 |
| ruby-lint | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/diff-patch-ruby-lint.txt |
| shadedjars | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-shadedjars.txt |
| hadoopcheck | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-javac-2.7.4.txt |
| hadoopcheck | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-javac-3.0.0.txt |
| hbaseprotoc | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-hbaseprotoc-hbase-client.txt |
| hbaseprotoc | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-hbaseprotoc-hbase-server.txt |
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-findbugs-hbase-client.txt |
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-findbugs-hbase-server.txt |
| javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/diff-javadoc-javadoc-hbase-client.txt |
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-unit-hbase-client.txt |
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-unit-hbase-server.txt |
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/artifact/patchprocess/patch-unit-hbase-shell.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/testReport/ |
| Max. process+thread count | 2436 (vs. ulimit of 10000) |
| modules | C: hbase-protocol-shaded hbase-hadoop-compat hbase-hadoop2-compat hbase-protocol hbase-client hbase-server hbase-shell U: . |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/15198/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Several inconsistencies on information reported for Replication Sources by hbase shell status 'replication' command.
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-21505
> URL: https://issues.apache.org/jira/browse/HBASE-21505
> Project: HBase
> Issue Type: Bug
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
> Attachments: 0001-HBASE-21505-initial-version-for-more-detailed-report.patch, HBASE-21505-master.001.patch
>
>
> While reviewing hbase shell status 'replication' command, noticed the following issues related to replication source section:
> 1) TimeStampsOfLastShippedOp keeps getting updated and increasing even when no new edits were added to source, so nothing was really shipped. Test steps performed:
> 1.1) Source cluster with only one table targeted to replication;
> 1.2) Added a new row, confirmed the row appeared in Target cluster;
> 1.3) Issued status 'replication' command in source, TimeStampsOfLastShippedOp shows current timestamp T1.
> 1.4) Waited 30 seconds, no new data added to source. Issued status 'replication' command, now shows timestamp T2.
> 2) When replication is stuck due some connectivity issues or target unavailability, if new edits are added in source, reported AgeOfLastShippedOp is wrongly showing same value as "Replication Lag". This is incorrect, AgeOfLastShippedOp should not change until there's indeed another edit shipped to target. Test steps performed:
> 2.1) Source cluster with only one table targeted to replication;
> 2.2) Stopped target cluster RS;
> 2.3) Put a new row on source. Running status 'replication' command does show lag increasing. TimeStampsOfLastShippedOp seems correct also, no further updates as described on bullet #1 above.
> 2.4) AgeOfLastShippedOp keeps increasing together with Replication Lag, even though there's no new edit shipped to target:
> {noformat}
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=5581, SizeOfLogQueue=1, TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=5581
> ...
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=8586, SizeOfLogQueue=1, TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=8586
> ...
> {noformat}
> 3) AgeOfLastShippedOp gets set to 0 even when a given edit had taken some time before it got finally shipped to target. Test steps performed:
> 3.1) Source cluster with only one table targeted to replication;
> 3.2) Stopped target cluster RS;
> 3.3) Put a new row on source.
> 3.4) AgeOfLastShippedOp keeps increasing together with Replication Lag, even though there's no new edit shipped to target:
> {noformat}
> T1:
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=5581, SizeOfLogQueue=1, TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=5581
> ...
> T2:
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=8586, SizeOfLogQueue=1, TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=8586
> ...
> {noformat}
> 3.5) Restart target cluster RS and verified the new row appeared there. No new edit added, but status 'replication' command reports AgeOfLastShippedOp as 0, while it should be the diff between the time it concluded shipping at target and the time it was added in source:
> {noformat}
> SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=1, TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=0
> {noformat}
> 4) When replication is stuck due some connectivity issues or target unavailability, if RS is restarted, once recovered queue source is started, TimeStampsOfLastShippedOp is set to initial java date (Thu Jan 01 01:00:00 GMT 1970, for example), thus "Replication Lag" also gives a complete inaccurate value.
> Tests performed:
> 4.1) Source cluster with only one table targeted to replication;
> 4.2) Stopped target cluster RS;
> 4.3) Put a new row on source, restart RS on source, waited a few seconds for recovery queue source to startup, then it gives:
> {noformat}
> SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=1, TimeStampsOfLastShippedOp=Thu Jan 01 01:00:00 GMT 1970, Replication Lag=9223372036854775807
> {noformat}
> Also, we should report status to all sources running, current output format gives the impression there’s only one, even when there are recovery queues, for instance.
> Here is a list of ideas on how the command should report under different states of replication:
> a) Source started, target stopped, no edits arrived on source yet:
> Status replication should not show any lags, no edits shipped, no edits arrived;
> b) Source started, target stopped, add edit on source:
> Status replication should report following info -> lag, time of edit arrival on source, additional message saying no edits had been shipped to target;
> c) Source started, target stopped, edit added on source, restart source:
> Status replication should list two sources, one normal, other recovered. Normal source should show no lags, no edits shipped, no edits arrived. Recovered should show no edits shipped, but should have edits arrived in source and lag > 0;
> d) Source started, target stopped, add edit on source, restart source, add another edit on source:
> Status replication should list two sources, one normal, other recovered. Both sources should show no edits shipped, but should have edits arrived in source and lag > 0;
> e) Source started, target stopped, add edit on source, restart source, add another edit on source, start target:
> Status replication should list normal source only (after some short period), with proper times for last shipped, last arrived in source and no replication lag.
> f) Source started, target stopped, add edit on source, restart source, restart target:
> Status replication should list normal source only, with no shipped, nor arrived edits, and lag should be 0;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)