You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/09/22 03:47:11 UTC
[GitHub] [hbase] nyl3532016 opened a new pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
nyl3532016 opened a new pull request #2437:
URL: https://github.com/apache/hbase/pull/2437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
nyl3532016 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-702163306
> Files that have not been closed show as 0-length. I'm concerned looking at length will have us miss edits. Can you address my concern? Thanks.
@saintstack as Duo Zhang comments in the above conversation? `Anyway, my point here is that, make sure that when we call this method, we have already finished renaming the wal directory and calling recover lease on the wal files. Calling recoverLease and then re-get the FileStatus will generate extra load to NameNode.` And I check the recoverFileLease is already invoked
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#discussion_r495850284
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -6423,7 +6427,11 @@ protected void restoreEdit(HStore s, Cell cell, MemStoreSizing memstoreAccountin
* @throws IOException
*/
private static boolean isZeroLengthThenDelete(final FileSystem fs, final FileStatus stat,
- final Path p) throws IOException {
+ final Path p, Configuration conf) throws IOException {
+ if (stat.getLen() > 0) {
+ return false;
+ }
+ RecoverLeaseFSUtils.recoverFileLease(fs, p, conf);
Review comment:
OK, then we do not need to add extra recover lease here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-699999788
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 44s | master passed |
| +1 :green_heart: | compile | 0m 59s | master passed |
| +1 :green_heart: | shadedjars | 7m 9s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 46s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 0s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 147m 10s | hbase-server in the patch passed. |
| | | 174m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 6d2d6ca3fb08 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 80ffac29c1 |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/testReport/ |
| Max. process+thread count | 3597 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
nyl3532016 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-706548263
@Apache9 @saintstack This PR can be merged and closed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-711474584
I‘m OK with merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-699638920
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 34s | master passed |
| +1 :green_heart: | checkstyle | 1m 20s | master passed |
| +1 :green_heart: | spotbugs | 2m 27s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 14s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 20s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 31s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 2m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 40m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 3acef72a33fa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b57bef5b7d |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
nyl3532016 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-711472666
> @Apache9 @saintstack This PR can be merged and closed?
@saintstack please have a look, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-696531355
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 12s | master passed |
| +1 :green_heart: | compile | 1m 19s | master passed |
| +1 :green_heart: | shadedjars | 7m 52s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 50s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 54s | the patch passed |
| +1 :green_heart: | compile | 1m 20s | the patch passed |
| +1 :green_heart: | javac | 1m 20s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 39s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 53s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 145m 43s | hbase-server in the patch passed. |
| | | 178m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 5d6374846d63 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 8a6299bb38 |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/testReport/ |
| Max. process+thread count | 3926 (vs. ulimit of 12500) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
saintstack merged pull request #2437:
URL: https://github.com/apache/hbase/pull/2437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-702529396
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 8s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 34s | master passed |
| +1 :green_heart: | compile | 0m 59s | master passed |
| +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 56s | the patch passed |
| +1 :green_heart: | javac | 0m 56s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 144m 41s | hbase-server in the patch passed. |
| | | 171m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 28815b2c9297 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3b91a15183 |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/testReport/ |
| Max. process+thread count | 3926 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-701590678
> Yes, we should invoke RecoverLeaseFSUtils.recoverFileLease in the method 'isZeroLengthThenDelete' ?
This has not been added?
Files that have not been closed show as 0-length. I'm concerned looking at length will have us miss edits. Can you address my concern? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#discussion_r495576703
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -6423,7 +6427,11 @@ protected void restoreEdit(HStore s, Cell cell, MemStoreSizing memstoreAccountin
* @throws IOException
*/
private static boolean isZeroLengthThenDelete(final FileSystem fs, final FileStatus stat,
- final Path p) throws IOException {
+ final Path p, Configuration conf) throws IOException {
+ if (stat.getLen() > 0) {
+ return false;
+ }
+ RecoverLeaseFSUtils.recoverFileLease(fs, p, conf);
Review comment:
But after this, you do not get the stat again?
Anyway, my point here is that, make sure that when we call this method, we have already finished renaming the wal directory and calling recover lease on the wal files. Calling recoverLease and then re-get the FileStatus will generate extra load to NameNode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-700004105
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 17s | master passed |
| +1 :green_heart: | compile | 1m 29s | master passed |
| +1 :green_heart: | shadedjars | 8m 46s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 49s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 7s | the patch passed |
| +1 :green_heart: | compile | 1m 29s | the patch passed |
| +1 :green_heart: | javac | 1m 29s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 43s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 147m 53s | hbase-server in the patch failed. |
| | | 182m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux f74f5bc2113c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 80ffac29c1 |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/testReport/ |
| Max. process+thread count | 3780 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-696496119
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 15s | master passed |
| +1 :green_heart: | checkstyle | 1m 16s | master passed |
| +1 :green_heart: | spotbugs | 2m 14s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 51s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 13s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 12m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 2m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 37m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux abc33ec778fa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 8a6299bb38 |
| Max. process+thread count | 84 (vs. ulimit of 12500) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-721830912
@nyl3532016 Sorry for delay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-696496119
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-702529905
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 44s | master passed |
| +1 :green_heart: | compile | 1m 14s | master passed |
| +1 :green_heart: | shadedjars | 7m 12s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 47s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 58s | the patch passed |
| +1 :green_heart: | compile | 1m 21s | the patch passed |
| +1 :green_heart: | javac | 1m 21s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 38s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 49s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 140m 18s | hbase-server in the patch passed. |
| | | 172m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 20b6853860ee 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3b91a15183 |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/testReport/ |
| Max. process+thread count | 4114 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-696734732
This is just a simple optimization or something related to correctness?
I'm not sure whether at this step we have finished recoverLease for all the WAL files? I mean whether the length in the FileStatus can be trusted?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#discussion_r498364347
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -6423,7 +6427,11 @@ protected void restoreEdit(HStore s, Cell cell, MemStoreSizing memstoreAccountin
* @throws IOException
*/
private static boolean isZeroLengthThenDelete(final FileSystem fs, final FileStatus stat,
- final Path p) throws IOException {
+ final Path p, Configuration conf) throws IOException {
+ if (stat.getLen() > 0) {
+ return false;
+ }
+ RecoverLeaseFSUtils.recoverFileLease(fs, p, conf);
Review comment:
I see now (was resolved so I missed it).
Then the patch looks good. A nit is that we could do w/ a comment here that says it is safe to trust the zero-length in this case because we've been through rename and lease recovery in the above. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-696532789
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 27s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 55s | master passed |
| +1 :green_heart: | compile | 0m 58s | master passed |
| +1 :green_heart: | shadedjars | 6m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 58s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 155m 22s | hbase-server in the patch passed. |
| | | 183m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9857379d9c72 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 8a6299bb38 |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/testReport/ |
| Max. process+thread count | 3723 (vs. ulimit of 12500) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-699942852
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 44s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 7m 56s | master passed |
| +1 :green_heart: | checkstyle | 2m 23s | master passed |
| +1 :green_heart: | spotbugs | 4m 45s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 54s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 15m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. |
| | | 52m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 0b9d8f7b5192 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 80ffac29c1 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on a change in pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
nyl3532016 commented on a change in pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#discussion_r498275885
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -6423,7 +6427,11 @@ protected void restoreEdit(HStore s, Cell cell, MemStoreSizing memstoreAccountin
* @throws IOException
*/
private static boolean isZeroLengthThenDelete(final FileSystem fs, final FileStatus stat,
- final Path p) throws IOException {
+ final Path p, Configuration conf) throws IOException {
+ if (stat.getLen() > 0) {
+ return false;
+ }
+ RecoverLeaseFSUtils.recoverFileLease(fs, p, conf);
Review comment:
@saintstack Here is the conversation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-702502601
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 41s | master passed |
| +1 :green_heart: | checkstyle | 1m 26s | master passed |
| +1 :green_heart: | spotbugs | 2m 42s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 10s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 14m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. |
| +1 :green_heart: | spotbugs | 2m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 43m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2437 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux c8bcd4e717a6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3b91a15183 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2437/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
Apache9 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-696734732
This is just a simple optimization or something related to correctness?
I'm not sure whether at this step we have finished recoverLease for all the WAL files? I mean whether the length in the FileStatus can be trusted?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on a change in pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
nyl3532016 commented on a change in pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#discussion_r495768332
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -6423,7 +6427,11 @@ protected void restoreEdit(HStore s, Cell cell, MemStoreSizing memstoreAccountin
* @throws IOException
*/
private static boolean isZeroLengthThenDelete(final FileSystem fs, final FileStatus stat,
- final Path p) throws IOException {
+ final Path p, Configuration conf) throws IOException {
+ if (stat.getLen() > 0) {
+ return false;
+ }
+ RecoverLeaseFSUtils.recoverFileLease(fs, p, conf);
Review comment:
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/region/MasterRegion.java#L255
I think we can make sure we finish rename and recover lease
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Posted by GitBox <gi...@apache.org>.
nyl3532016 commented on pull request #2437:
URL: https://github.com/apache/hbase/pull/2437#issuecomment-699634693
> This is just a simple optimization or something related to correctness?
>
> I'm not sure whether at this step we have finished recoverLease for all the WAL files? I mean whether the length in the FileStatus can be trusted?
>
> Thanks.
Yes, we should invoke `RecoverLeaseFSUtils.recoverFileLease` in the method 'isZeroLengthThenDelete' ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org