You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/05/28 18:47:17 UTC
[GitHub] [hbase] bharathv opened a new pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
bharathv opened a new pull request #3332:
URL: https://github.com/apache/hbase/pull/3332
…lementation.
This bug was exposed by the test from HBASE-25924 and is only applicable for
Async FSWAL implementation. Since this wal implementation closes the wal
asynchronously, replication can potentially miss the trailer bytes.
(see jira comment for detailed analysis).
While this is not a correctness problem (since trailer does not have any entry data),
it erroneously bumps a metric that is used to track skipped bytes in WAL resulting
in false alarms which is something we should avoid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850699156
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 58s | master passed |
| +1 :green_heart: | compile | 1m 1s | master passed |
| +1 :green_heart: | shadedjars | 10m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 22s | the patch passed |
| +1 :green_heart: | compile | 1m 18s | the patch passed |
| +1 :green_heart: | javac | 1m 18s | the patch passed |
| +1 :green_heart: | shadedjars | 10m 35s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 165m 51s | hbase-server in the patch failed. |
| | | 202m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux c6ff07f9347e 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3f7d2897a1 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/testReport/ |
| Max. process+thread count | 3611 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850834962
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 37s | master passed |
| +1 :green_heart: | compile | 1m 9s | master passed |
| +1 :green_heart: | shadedjars | 8m 56s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 5s | the patch passed |
| +1 :green_heart: | javac | 1m 5s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 39s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 147m 15s | hbase-server in the patch passed. |
| | | 179m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d7afb6a79356 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9a2027bf71 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/testReport/ |
| Max. process+thread count | 3801 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642781222
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
Thanks for clarifying.
> What we are in effect doing is, when the WAL file is not yet closed, we get the length from writer which tracks the synced length. Once the WAL file is closed,
Correct.
> Now with the async close, it is possible that when this method execution is at Line 1208, the close is not over but before we return from this method, the WAL file is really closed
Correct
> Still we will return a value and system thinks WAL file is still active one and undergo related logic. It may be very much fine in those places. But I just wanted to make sure we double check.
Yes this is transparently handled by WalEntryStream. If what you said happens (which is possible), WALEntryStream thinks that the file is not fully done yet (in that loop iteration) and in the next attempt when it does the same check again to read more bytes, it sees that the file is already done and closed and either reads or terminates accordingly.
Relevant code links
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L283 (the entry read is null, so it just loops again)
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L197 (this is for looping again, in the subsequent next() call, this fill will be seen as closed and it enters the if block in L180)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850817040
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 59s | master passed |
| +1 :green_heart: | compile | 3m 12s | master passed |
| +1 :green_heart: | checkstyle | 1m 3s | master passed |
| +1 :green_heart: | spotbugs | 2m 5s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 38s | the patch passed |
| +1 :green_heart: | compile | 3m 4s | the patch passed |
| +1 :green_heart: | javac | 3m 4s | the patch passed |
| -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 18m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 11s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 46m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 934f4cd080d3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9a2027bf71 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 95 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642682623
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
Not sure what you mean, we are returning a long from getSyncedLength() under the same lock (we are not returning the reference temp), actual method getSyncedLength() is thread-safe, if thats what you are asking..
Also, both shutdown() / rollWriter are under the scope of rollWriterLock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851167865
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 21s | master passed |
| +1 :green_heart: | compile | 3m 22s | master passed |
| +1 :green_heart: | checkstyle | 1m 12s | master passed |
| +1 :green_heart: | spotbugs | 2m 11s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 58s | the patch passed |
| +1 :green_heart: | compile | 3m 22s | the patch passed |
| +1 :green_heart: | javac | 3m 22s | the patch passed |
| -0 :warning: | checkstyle | 1m 10s | hbase-server: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 22s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 13s | The patch generated 1 ASF License warnings. |
| | | 51m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 29aba4c3593a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f2ff816532 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
bharathv merged pull request #3332:
URL: https://github.com/apache/hbase/pull/3332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850689600
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 31s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 30s | master passed |
| +1 :green_heart: | compile | 1m 12s | master passed |
| +1 :green_heart: | shadedjars | 8m 12s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 16s | the patch passed |
| +1 :green_heart: | compile | 1m 11s | the patch passed |
| +1 :green_heart: | javac | 1m 11s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 6s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 138m 58s | hbase-server in the patch passed. |
| | | 171m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux b8a2c5b4b019 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3f7d2897a1 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/testReport/ |
| Max. process+thread count | 4158 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851286534
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 0s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 25s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 2s | the patch passed |
| +1 :green_heart: | compile | 1m 2s | the patch passed |
| +1 :green_heart: | javac | 1m 2s | the patch passed |
| -1 :x: | shadedjars | 7m 10s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 209m 18s | hbase-server in the patch failed. |
| | | 240m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux eefbeedd5d9b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f2ff816532 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/testReport/ |
| Max. process+thread count | 3484 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
bharathv commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850759150
Test failures are not related, looped them over 100 times and they passed locally (failed-to-read.. is a known issue).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850635611
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 9s | master passed |
| +1 :green_heart: | compile | 4m 9s | master passed |
| +1 :green_heart: | checkstyle | 1m 28s | master passed |
| +1 :green_heart: | spotbugs | 2m 45s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 24s | the patch passed |
| +1 :green_heart: | compile | 3m 44s | the patch passed |
| +1 :green_heart: | javac | 3m 44s | the patch passed |
| -0 :warning: | checkstyle | 1m 23s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 22m 24s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 15s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 57m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux bab35c191369 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3f7d2897a1 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851618311
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 11s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 9m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 5s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| -1 :x: | shadedjars | 7m 10s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 212m 11s | hbase-server in the patch failed. |
| | | 243m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5b3a055d4b7c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1ccba10847 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/testReport/ |
| Max. process+thread count | 3373 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642333369
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
We wont be having the rollWriterLock based lock in WAL close area. It might be possible that the check pass here and by when it do temp.getSyncedLength(), the WAL might be closed. That will be ok? Good to double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851537971
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 9s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 21s | master passed |
| +1 :green_heart: | compile | 3m 26s | master passed |
| +1 :green_heart: | checkstyle | 1m 9s | master passed |
| +1 :green_heart: | spotbugs | 2m 12s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 1s | the patch passed |
| +1 :green_heart: | compile | 3m 20s | the patch passed |
| +1 :green_heart: | javac | 3m 20s | the patch passed |
| -0 :warning: | checkstyle | 1m 9s | hbase-server: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 19m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 21s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 13s | The patch generated 1 ASF License warnings. |
| | | 51m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux bf344310ac48 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1ccba10847 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850836634
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 42s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 58s | master passed |
| +1 :green_heart: | compile | 1m 39s | master passed |
| +1 :green_heart: | shadedjars | 11m 9s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 2s | the patch passed |
| +1 :green_heart: | compile | 1m 42s | the patch passed |
| +1 :green_heart: | javac | 1m 42s | the patch passed |
| +1 :green_heart: | shadedjars | 11m 14s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 148m 47s | hbase-server in the patch passed. |
| | | 191m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 08c64d147681 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9a2027bf71 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/testReport/ |
| Max. process+thread count | 4273 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851745942
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 23s | master passed |
| +1 :green_heart: | compile | 1m 3s | master passed |
| +1 :green_heart: | shadedjars | 8m 59s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 59s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 59s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 209m 53s | hbase-server in the patch passed. |
| | | 242m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux fd6e094938e7 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 06c6e06803 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/testReport/ |
| Max. process+thread count | 3086 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r641920706
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##########
@@ -103,6 +106,27 @@
private static final NavigableMap<byte[], Integer> scopes = getScopes();
private final String fakeWalGroupId = "fake-wal-group-id";
+ /**
+ * Test helper that waits until a non-null entry is available in the stream next or times out.
+ */
+ private static class WALEntryStreamWithRetries extends WALEntryStream {
+ // Class member to be able to set a non-final from within a lambda.
+ private Entry result;
+
+ public WALEntryStreamWithRetries(ReplicationSourceLogQueue logQueue, Configuration conf,
+ long startPosition, WALFileLengthProvider walFileLengthProvider, ServerName serverName,
+ MetricsSource metrics, String walGroupId) throws IOException {
+ super(logQueue, conf, startPosition, walFileLengthProvider, serverName, metrics, walGroupId);
+ }
+
+ @Override
+ public Entry next() {
+ Waiter.waitFor(CONF, TEST_TIMEOUT_MS, (Waiter.Predicate<Exception>) () -> (result
+ = WALEntryStreamWithRetries.super.next()) != null);
Review comment:
nit:
```
Waiter.waitFor(CONF, TEST_TIMEOUT_MS, () -> (result = WALEntryStreamWithRetries.super.next()) != null);
```
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##########
@@ -103,6 +106,27 @@
private static final NavigableMap<byte[], Integer> scopes = getScopes();
private final String fakeWalGroupId = "fake-wal-group-id";
+ /**
+ * Test helper that waits until a non-null entry is available in the stream next or times out.
+ */
+ private static class WALEntryStreamWithRetries extends WALEntryStream {
+ // Class member to be able to set a non-final from within a lambda.
+ private Entry result;
Review comment:
Nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851704613
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 6s | master passed |
| +1 :green_heart: | compile | 3m 54s | master passed |
| +1 :green_heart: | checkstyle | 1m 17s | master passed |
| +1 :green_heart: | spotbugs | 2m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 3s | the patch passed |
| +1 :green_heart: | compile | 3m 22s | the patch passed |
| +1 :green_heart: | javac | 3m 22s | the patch passed |
| -0 :warning: | checkstyle | 1m 9s | hbase-server: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 20m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 2m 22s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 53m 50s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 76d596f17c0d 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 06c6e06803 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642784660
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
Thanks for taking a look @anoopsjohn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642782022
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
Thanks for checking and confirming. Looks good then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850867187
Nice find
>only applicable for Async FSWAL implementation. Since this wal implementation closes the wal asynchronously,
FSHLog also been closed in async way in Master branch. Pls see FSHLog#doReplaceWriter. No applicable there? I have not seen the original issue description though!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642750975
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
Sorry for not making clear.
What we are in effect doing is, when the WAL file is not yet closed, we get the length from writer which tracks the synced length. Once the WAL file is closed, (we know that from the output of this method) there is different logic. (I have not seen that full area though). Now with the async close, it is possible that when this method execution is at Line 1208, the close is not over but we return from here, the WAL file is really closed. Still we will return a value and system thinks WAL file is still active one and undergo related logic. It may be very much fine in those places. But I just wanted to make we double check.
With sync close, this method and the writer close all were having a common lock. (rollWriterLock). The entire rollWriter(boolean force) is under that lock and cur writer close was within it.
Hope am making it clear now?
Again it may be very much fine only. But just can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
bharathv commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851151731
> FSHLog also been closed in async way in Master branch. Pls see FSHLog#doReplaceWriter. No applicable there?
@anoopsjohn Thanks for pointing it out, I incorrectly assumed that FSHLog behavior stayed the same in branch-2/master. This async closing of FSHLog was not ported to branch-1. On top of that branch-2/master tests have coverage only for AsyncWAL since that is the default and that didn't help. The latest patch fixes this issue for both the implementations along with test coverage. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #3332: HBASE-25932: Ensure replication reads trailer bytes with AsyncWAL imp…
Posted by GitBox <gi...@apache.org>.
bharathv commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-850602486
@shahrs87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851592364
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 15s | master passed |
| +1 :green_heart: | compile | 1m 10s | master passed |
| +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 17s | the patch passed |
| +1 :green_heart: | compile | 1m 11s | the patch passed |
| +1 :green_heart: | javac | 1m 11s | the patch passed |
| -1 :x: | shadedjars | 6m 36s | patch has 10 errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 41s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 136m 54s | hbase-server in the patch failed. |
| | | 166m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 05e8daa0e069 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 1ccba10847 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/testReport/ |
| Max. process+thread count | 4335 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
bharathv commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851811848
Oops, I thought I had a +1 from you @anoopsjohn and I merged the change. Let me open a follow up patch to address your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#issuecomment-851727851
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 32s | master passed |
| +1 :green_heart: | compile | 1m 12s | master passed |
| +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 16s | the patch passed |
| +1 :green_heart: | compile | 1m 11s | the patch passed |
| +1 :green_heart: | javac | 1m 11s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 18s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 41s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 136m 22s | hbase-server in the patch passed. |
| | | 168m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/3332 |
| JIRA Issue | HBASE-25932 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 1ef13b1b7d55 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 06c6e06803 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/testReport/ |
| Max. process+thread count | 4258 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642750975
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##########
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+ // Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) : OptionalLong.empty();
} else {
+ W temp = inflightWALClosures.get(path.getName());
+ if (temp != null) {
Review comment:
Sorry for not making clear.
What we are in effect doing is, when the WAL file is not yet closed, we get the length from writer which tracks the synced length. Once the WAL file is closed, (we know that from the output of this method) there is different logic. (I have not seen that full area though). Now with the async close, it is possible that when this method execution is at Line 1208, the close is not over but before we return from this method, the WAL file is really closed. Still we will return a value and system thinks WAL file is still active one and undergo related logic. It may be very much fine in those places. But I just wanted to make sure we double check.
With sync close, this method and the writer close all were having a common lock. (rollWriterLock). The entire rollWriter(boolean force) is under that lock and cur writer close was within it.
Hope am making it clear now (?)
Again it may be very much fine only. But just can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642797633
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##########
@@ -530,7 +557,8 @@ public void testReplicationSourceWALReaderWrongPosition() throws Exception {
@Override
public boolean evaluate() throws Exception {
- return fs.getFileStatus(walPath).getLen() > 0;
+ return fs.getFileStatus(walPath).getLen() > 0 &&
+ ((AbstractFSWAL) log).getInflightWALCloseCount() == 0;
Review comment:
We don't wait for 5s and check. We wait for the condition to pass within 5s or timeout, this is the javadoc, think the test is doing what you are saying (unless I misunderstood what you are saying)..
```
/**
* Waits up to the duration equal to the specified timeout multiplied by the
* {@link #getWaitForRatio(Configuration)} for the given {@link Predicate} to become
* <code>true</code>, failing the test if the timeout is reached and the Predicate is still
* <code>false</code>.
* <p/>
* @param conf the configuration
* @param timeout the timeout in milliseconds to wait for the predicate.
* @param predicate the predicate to evaluate.
* @return the effective wait, in milli-seconds until the predicate becomes <code>true</code> or
* wait is interrupted otherwise <code>-1</code> when times out
*/
```
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##########
@@ -103,6 +107,27 @@
private static final NavigableMap<byte[], Integer> scopes = getScopes();
private final String fakeWalGroupId = "fake-wal-group-id";
+ /**
+ * Test helper that waits until a non-null entry is available in the stream next or times out.
+ */
+ private static class WALEntryStreamWithRetries extends WALEntryStream {
Review comment:
Addressed in #3344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642784905
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##########
@@ -530,7 +557,8 @@ public void testReplicationSourceWALReaderWrongPosition() throws Exception {
@Override
public boolean evaluate() throws Exception {
- return fs.getFileStatus(walPath).getLen() > 0;
+ return fs.getFileStatus(walPath).getLen() > 0 &&
+ ((AbstractFSWAL) log).getInflightWALCloseCount() == 0;
Review comment:
We wait for 5 sec and check. Should be wait for getInflightWALCloseCount to be zero before coming here? Or will that be a better guarantee for WAL close has happened where we come here for FS.getLen() check?
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##########
@@ -103,6 +107,27 @@
private static final NavigableMap<byte[], Integer> scopes = getScopes();
private final String fakeWalGroupId = "fake-wal-group-id";
+ /**
+ * Test helper that waits until a non-null entry is available in the stream next or times out.
+ */
+ private static class WALEntryStreamWithRetries extends WALEntryStream {
Review comment:
Can u also pls explain why we use this special subclass here for the tests? (as comments) will be easy to read and digest later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org