You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/10/20 08:54:55 UTC
[GitHub] [hbase] Joseph295 opened a new pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Joseph295 opened a new pull request #2568:
URL: https://github.com/apache/hbase/pull/2568
…s trying to open
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-716932347
> Can we change the jira title/description, as this really deals with recovered edits, not hfiles?
>
> And just giving a second thought on this issue: here we log only once the corruption is first detected, then mark it and never log it again, silently ignoring it. I'm wondering if we should rather stay verbose, in this case, given it may imply data loss.
I checked the code, and find it is not essential for recovered edits as every time we begin to deal with recover edits, we will clear the files first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-712796440
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 45s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 24s | master passed |
| +1 :green_heart: | compile | 1m 32s | master passed |
| +1 :green_heart: | shadedjars | 7m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 48s | the patch passed |
| +1 :green_heart: | compile | 1m 41s | the patch passed |
| +1 :green_heart: | javac | 1m 41s | the patch passed |
| +1 :green_heart: | shadedjars | 8m 34s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 13s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 50s | hbase-common in the patch passed. |
| -1 :x: | unit | 143m 9s | hbase-server in the patch failed. |
| | | 179m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 949cb680364e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 7d00e43dce |
| Default Java | 2020-01-14 |
| unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/testReport/ |
| Max. process+thread count | 3820 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509242775
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5301,7 +5301,11 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ if (isSkipErrorEnabled() && fileName.endsWith(HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
Review comment:
recovered.edits have same problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509244150
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5944,6 +5951,11 @@ private long loadRecoveredHFilesIfAny(Collection<HStore> stores) throws IOExcept
if (isZeroLengthThenDelete(fs.getFileSystem(), file, filePath)) {
continue;
}
+ if (isSkipErrorEnabled() && filePath.getName().endsWith(
+ HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
+ LOG.info("Skip corrupted HFile {}", filePath);
Review comment:
LOG.warn?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509243675
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5324,22 +5328,25 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
}
private void handleException(FileSystem fs, Path edits, IOException e) throws IOException {
- boolean skipErrors = conf.getBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS,
- conf.getBoolean("hbase.skip.errors", HConstants.DEFAULT_HREGION_EDITS_REPLAY_SKIP_ERRORS));
- if (conf.get("hbase.skip.errors") != null) {
- LOG.warn("The property 'hbase.skip.errors' has been deprecated. Please use "
- + HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + " instead.");
- }
- if (skipErrors) {
+ if (isSkipErrorEnabled()) {
Path p = WALSplitUtil.moveAsideBadEditsFile(fs, edits);
- LOG.error(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + "=true so continuing. Renamed "
- + edits + " as " + p,
- e);
+ LOG.info(
Review comment:
LOG not right? Seems hbase.skip.errors=true may got here too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-714519693
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 29s | master passed |
| +1 :green_heart: | compile | 1m 22s | master passed |
| +1 :green_heart: | shadedjars | 6m 43s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 29s | the patch passed |
| +1 :green_heart: | compile | 1m 19s | the patch passed |
| +1 :green_heart: | javac | 1m 19s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 25s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 155m 30s | hbase-server in the patch passed. |
| | | 186m 1s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 1b54cf6f8e6d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2cd5f914af |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/testReport/ |
| Max. process+thread count | 3811 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r512373857
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5301,7 +5301,11 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ if (isSkipErrorEnabled() && fileName.endsWith(HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
Review comment:
> recovered.edits have same problem? or problem only for SplitWALToHFile? Can we have UT for this?
I checked the code, recovered.edits will clear the files at the beginning, so we don't need UT now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509842639
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5324,22 +5328,25 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
}
private void handleException(FileSystem fs, Path edits, IOException e) throws IOException {
- boolean skipErrors = conf.getBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS,
- conf.getBoolean("hbase.skip.errors", HConstants.DEFAULT_HREGION_EDITS_REPLAY_SKIP_ERRORS));
- if (conf.get("hbase.skip.errors") != null) {
- LOG.warn("The property 'hbase.skip.errors' has been deprecated. Please use "
- + HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + " instead.");
- }
- if (skipErrors) {
+ if (isSkipErrorEnabled()) {
Path p = WALSplitUtil.moveAsideBadEditsFile(fs, edits);
- LOG.error(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + "=true so continuing. Renamed "
- + edits + " as " + p,
- e);
+ LOG.info(
Review comment:
This is a behavior as expected?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-716931810
> Can we change the jira title/description, as this really deals with recovered edits, not hfiles?
>
> And just giving a second thought on this issue: here we log only once the corruption is first detected, then mark it and never log it again, silently ignoring it. I'm wondering if we should rather stay verbose, in this case, given it may imply data loss.
Do you have other concerns?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-713318222
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 44s | master passed |
| +1 :green_heart: | compile | 1m 20s | master passed |
| +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 31s | the patch passed |
| +1 :green_heart: | compile | 1m 20s | the patch passed |
| +1 :green_heart: | javac | 1m 20s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 149m 40s | hbase-server in the patch passed. |
| | | 180m 20s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 18b5aab88c14 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 200343c59c |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/testReport/ |
| Max. process+thread count | 4032 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-713312633
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 14s | master passed |
| +1 :green_heart: | compile | 1m 34s | master passed |
| +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 3s | the patch passed |
| +1 :green_heart: | compile | 1m 33s | the patch passed |
| +1 :green_heart: | javac | 1m 33s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 132m 12s | hbase-server in the patch passed. |
| | | 164m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 845062de16c2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 200343c59c |
| Default Java | 2020-01-14 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/testReport/ |
| Max. process+thread count | 3937 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-714438695
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 31s | master passed |
| +1 :green_heart: | checkstyle | 1m 29s | master passed |
| +1 :green_heart: | spotbugs | 2m 38s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 28s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 18m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
| | | 46m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux b9b6bcc060dd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2cd5f914af |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509242775
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5301,7 +5301,11 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ if (isSkipErrorEnabled() && fileName.endsWith(HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
Review comment:
recovered.edits have same problem? or problem only for SplitWALToHFile? Can we have UT for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-713274238
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 40s | master passed |
| +1 :green_heart: | checkstyle | 1m 36s | master passed |
| +1 :green_heart: | spotbugs | 2m 46s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 18m 35s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 46m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux b6fda8ae0ff6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 200343c59c |
| Max. process+thread count | 94 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-712829053
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 0s | master passed |
| +1 :green_heart: | compile | 1m 19s | master passed |
| +1 :green_heart: | shadedjars | 7m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | the patch passed |
| +1 :green_heart: | compile | 1m 20s | the patch passed |
| +1 :green_heart: | javac | 1m 20s | the patch passed |
| +1 :green_heart: | shadedjars | 7m 11s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 54s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 208m 39s | hbase-server in the patch passed. |
| | | 241m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux f8dec23eac1b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 7d00e43dce |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/testReport/ |
| Max. process+thread count | 3476 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509847080
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5301,7 +5301,11 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ if (isSkipErrorEnabled() && fileName.endsWith(HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
Review comment:
OK, let me add a UT for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r508578218
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5301,7 +5301,7 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ maxSeqId = Math.abs(Long.parseLong(fileName.split("\\.")[0]));
Review comment:
For files already with the `corrupted` suffix, shouldn't we just skip the whole replay attempt from this point onwards? We already know it will not work, so we can save some computation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-822224859
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 55s | master passed |
| +1 :green_heart: | compile | 4m 8s | master passed |
| +1 :green_heart: | checkstyle | 1m 32s | master passed |
| +1 :green_heart: | spotbugs | 2m 49s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 32s | the patch passed |
| +1 :green_heart: | compile | 4m 0s | the patch passed |
| +1 :green_heart: | javac | 4m 0s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 17m 59s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. |
| +1 :green_heart: | spotbugs | 3m 7s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 51m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 655de0e1216d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b65890da1d |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 96 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r512373857
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5301,7 +5301,11 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ if (isSkipErrorEnabled() && fileName.endsWith(HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
Review comment:
> recovered.edits have same problem? or problem only for SplitWALToHFile? Can we have UT for this?
I checked the code, recovered.edits will be cleared at the beginning, so we don't need UT now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#issuecomment-714515576
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 21s | master passed |
| +1 :green_heart: | compile | 1m 45s | master passed |
| +1 :green_heart: | shadedjars | 8m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 5s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 2s | the patch passed |
| +1 :green_heart: | compile | 1m 29s | the patch passed |
| +1 :green_heart: | javac | 1m 29s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | hbase-common in the patch passed. |
| +1 :green_heart: | unit | 146m 19s | hbase-server in the patch passed. |
| | | 179m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2568 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 23e078489435 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2cd5f914af |
| Default Java | 2020-01-14 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/testReport/ |
| Max. process+thread count | 3924 (vs. ulimit of 30000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r510029108
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5324,22 +5328,25 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
}
private void handleException(FileSystem fs, Path edits, IOException e) throws IOException {
- boolean skipErrors = conf.getBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS,
- conf.getBoolean("hbase.skip.errors", HConstants.DEFAULT_HREGION_EDITS_REPLAY_SKIP_ERRORS));
- if (conf.get("hbase.skip.errors") != null) {
- LOG.warn("The property 'hbase.skip.errors' has been deprecated. Please use "
- + HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + " instead.");
- }
- if (skipErrors) {
+ if (isSkipErrorEnabled()) {
Review comment:
By default this skip error config is false. So similar case of WAL split to hfiles, we will have partial Hfiles from initial split and loading of that will fail always right? (loadRecoveredHFilesIfAny ())
cc @infraio
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Posted by GitBox <gi...@apache.org>.
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r512373549
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5324,22 +5328,25 @@ private long replayRecoveredEditsForPaths(long minSeqIdForTheRegion, FileSystem
}
private void handleException(FileSystem fs, Path edits, IOException e) throws IOException {
- boolean skipErrors = conf.getBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS,
- conf.getBoolean("hbase.skip.errors", HConstants.DEFAULT_HREGION_EDITS_REPLAY_SKIP_ERRORS));
- if (conf.get("hbase.skip.errors") != null) {
- LOG.warn("The property 'hbase.skip.errors' has been deprecated. Please use "
- + HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + " instead.");
- }
- if (skipErrors) {
+ if (isSkipErrorEnabled()) {
Review comment:
Sorry sir, I don't catch you. You mean set skip error config to true?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org