You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/03/22 10:36:53 UTC
[GitHub] [hbase] infraio opened a new pull request #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
infraio opened a new pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio merged pull request #1322: HBASE-24033 Add ut for
loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
infraio merged pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on a change in pull request #1322:
HBASE-24033 Add ut for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#discussion_r396086300
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitUtil.java
##########
@@ -345,16 +345,16 @@ public boolean accept(Path p) {
/**
* Move aside a bad edits file.
- * @param walFS WAL FileSystem used to rename bad edits file.
+ * @param fs the file system used to rename bad edits file.
* @param edits Edits file to move aside.
* @return The name of the moved aside file.
* @throws IOException
*/
- public static Path moveAsideBadEditsFile(final FileSystem walFS, final Path edits)
+ public static Path moveAsideBadEditsFile(final FileSystem fs, final Path edits)
Review comment:
Yes. For hfile, the passed fs is root file system.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#issuecomment-602189972
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 39s | master passed |
| +1 :green_heart: | compile | 1m 7s | master passed |
| -1 :x: | shadedjars | 0m 12s | branch has 7 errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 10s | the patch passed |
| +1 :green_heart: | compile | 1m 7s | the patch passed |
| +1 :green_heart: | javac | 1m 7s | the patch passed |
| -1 :x: | shadedjars | 0m 10s | patch has 7 errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| -0 :warning: | unit | 66m 50s | hbase-server in the patch failed. |
| | | 86m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1322 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux ccaaa7ec4b54 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ca533a77cb |
| Default Java | 2020-01-14 |
| shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk11-hadoop3-check/output/branch-shadedjars.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/testReport/ |
| Max. process+thread count | 5629 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1322:
HBASE-24033 Add ut for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#discussion_r396084244
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitToHFile.java
##########
@@ -163,24 +175,88 @@ private WAL createWAL(Configuration c, Path hbaseRootDir, String logName) throws
return wal;
}
- /**
- * Test writing edits into an HRegion, closing it, splitting logs, opening
- * Region again. Verify seqids.
- */
- @Test
- public void testReplayEditsWrittenViaHRegion()
- throws IOException, SecurityException, IllegalArgumentException, InterruptedException {
+ private Pair<TableDescriptor, RegionInfo> setupTableAndRegion() throws IOException {
final TableName tableName = TableName.valueOf(TEST_NAME.getMethodName());
final TableDescriptor td = createBasic3FamilyTD(tableName);
final RegionInfo ri = RegionInfoBuilder.newBuilder(tableName).build();
final Path tableDir = FSUtils.getTableDir(this.rootDir, tableName);
deleteDir(tableDir);
FSTableDescriptors.createTableDescriptorForTableDirectory(fs, tableDir, td, false);
- final byte[] rowName = tableName.getName();
- final int countPerFamily = 10;
+ HRegion region = HBaseTestingUtility.createRegionAndWAL(ri, rootDir, this.conf, td);
+ HBaseTestingUtility.closeRegionAndWAL(region);
+ return new Pair<>(td, ri);
+ }
+
+ @Test
+ public void testCorruptRecoveredHFile() throws Exception {
+ Pair<TableDescriptor, RegionInfo> pair = setupTableAndRegion();
+ TableDescriptor td = pair.getFirst();
+ RegionInfo ri = pair.getSecond();
+
+ WAL wal = createWAL(this.conf, rootDir, logName);
+ HRegion region = HRegion.openHRegion(this.conf, this.fs, rootDir, ri, td, wal);
+ final long timestamp = this.ee.currentTime();
+ // Write data and flush
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ region.put(new Put(ROW).addColumn(cfd.getName(), Bytes.toBytes("x"), timestamp, VALUE1));
+ }
+ region.flush(true);
+
+ // Now assert edits made it in.
+ Result result1 = region.get(new Get(ROW));
+ assertEquals(td.getColumnFamilies().length, result1.size());
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ assertTrue(Bytes.equals(VALUE1, result1.getValue(cfd.getName(), Bytes.toBytes("x"))));
+ }
+
+ // Now close the region
+ region.close(true);
+ wal.shutdown();
+ // split the log
+ WALSplitter.split(rootDir, logDir, oldLogDir, FileSystem.get(this.conf), this.conf, wals);
+
+ // Write a corrupt recovered hfile
+ Path regionDir =
+ new Path(CommonFSUtils.getTableDir(rootDir, td.getTableName()), ri.getEncodedName());
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ FileStatus[] files =
+ WALSplitUtil.getRecoveredHFiles(this.fs, regionDir, cfd.getNameAsString());
+ assertNotNull(files);
+ assertTrue(files.length > 0);
+ writeCorruptRecoveredHFile(files[0].getPath());
+ }
+
+ // Failed to reopen the region
+ WAL wal2 = createWAL(this.conf, rootDir, logName);
+ try {
+ HRegion.openHRegion(this.conf, this.fs, rootDir, ri, td, wal2);
+ fail("Should fail to open region");
+ } catch (CorruptHFileException che) {
+ // Expected
+ }
+
+ // Set skip errors to true and reopen the region
+ this.conf.setBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS, true);
+ HRegion region2 = HRegion.openHRegion(this.conf, this.fs, rootDir, ri, td, wal2);
+ Result result2 = region2.get(new Get(ROW));
+ assertEquals(td.getColumnFamilies().length, result2.size());
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ assertTrue(Bytes.equals(VALUE1, result2.getValue(cfd.getName(), Bytes.toBytes("x"))));
+ }
+ this.conf.setBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS, false);
+ }
Review comment:
Add an assert to confirm that we move the broken HFile to the expected place?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1322:
HBASE-24033 Add ut for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#discussion_r396091294
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5449,6 +5449,7 @@ private long loadRecoveredHFilesIfAny(Collection<HStore> stores) throws IOExcept
store.assertBulkLoadHFileOk(filePath);
} catch (IOException e) {
handleException(fs.getFileSystem(), filePath, e);
+ continue;
Review comment:
This is a bug?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#issuecomment-602190621
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 41s | master passed |
| +1 :green_heart: | compile | 0m 56s | master passed |
| +1 :green_heart: | shadedjars | 4m 59s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 17s | the patch passed |
| +1 :green_heart: | compile | 0m 57s | the patch passed |
| +1 :green_heart: | javac | 0m 57s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 59s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 64m 29s | hbase-server in the patch passed. |
| | | 91m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1322 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux bd5f88e0889e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ca533a77cb |
| Default Java | 1.8.0_232 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/testReport/ |
| Max. process+thread count | 5262 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#issuecomment-602198384
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 26s | master passed |
| +1 :green_heart: | checkstyle | 1m 15s | master passed |
| -1 :x: | spotbugs | 2m 3s | hbase-server in master has 1 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 14s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 19s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 29s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. |
| +1 :green_heart: | spotbugs | 2m 19s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 36m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1322 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux bac018bee683 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ca533a77cb |
| spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| Max. process+thread count | 93 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on a change in pull request #1322:
HBASE-24033 Add ut for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
infraio commented on a change in pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#discussion_r396092140
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##########
@@ -5449,6 +5449,7 @@ private long loadRecoveredHFilesIfAny(Collection<HStore> stores) throws IOExcept
store.assertBulkLoadHFileOk(filePath);
} catch (IOException e) {
handleException(fs.getFileSystem(), filePath, e);
+ continue;
Review comment:
Yes. Found it when add ut.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#issuecomment-602207720
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 10m 31s | master passed |
| +1 :green_heart: | compile | 1m 51s | master passed |
| -1 :x: | shadedjars | 0m 19s | branch has 7 errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 1m 7s | hbase-server in master failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 10m 22s | the patch passed |
| +1 :green_heart: | compile | 1m 53s | the patch passed |
| +1 :green_heart: | javac | 1m 53s | the patch passed |
| -1 :x: | shadedjars | 0m 14s | patch has 7 errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 1m 1s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 73m 33s | hbase-server in the patch passed. |
| | | 104m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1322 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux bb5e86c67ec3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ca533a77cb |
| Default Java | 2020-01-14 |
| shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-jdk11-hadoop3-check/output/branch-shadedjars.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/testReport/ |
| Max. process+thread count | 5861 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1322:
HBASE-24033 Add ut for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#discussion_r396084038
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitUtil.java
##########
@@ -345,16 +345,16 @@ public boolean accept(Path p) {
/**
* Move aside a bad edits file.
- * @param walFS WAL FileSystem used to rename bad edits file.
+ * @param fs the file system used to rename bad edits file.
* @param edits Edits file to move aside.
* @return The name of the moved aside file.
* @throws IOException
*/
- public static Path moveAsideBadEditsFile(final FileSystem walFS, final Path edits)
+ public static Path moveAsideBadEditsFile(final FileSystem fs, final Path edits)
Review comment:
Why this name changing? Because for HFile based split we will write HFile to the root fs instead of wal fs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1322:
HBASE-24033 Add ut for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache9 commented on a change in pull request #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#discussion_r396084197
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitToHFile.java
##########
@@ -163,24 +175,88 @@ private WAL createWAL(Configuration c, Path hbaseRootDir, String logName) throws
return wal;
}
- /**
- * Test writing edits into an HRegion, closing it, splitting logs, opening
- * Region again. Verify seqids.
- */
- @Test
- public void testReplayEditsWrittenViaHRegion()
- throws IOException, SecurityException, IllegalArgumentException, InterruptedException {
+ private Pair<TableDescriptor, RegionInfo> setupTableAndRegion() throws IOException {
final TableName tableName = TableName.valueOf(TEST_NAME.getMethodName());
final TableDescriptor td = createBasic3FamilyTD(tableName);
final RegionInfo ri = RegionInfoBuilder.newBuilder(tableName).build();
final Path tableDir = FSUtils.getTableDir(this.rootDir, tableName);
deleteDir(tableDir);
FSTableDescriptors.createTableDescriptorForTableDirectory(fs, tableDir, td, false);
- final byte[] rowName = tableName.getName();
- final int countPerFamily = 10;
+ HRegion region = HBaseTestingUtility.createRegionAndWAL(ri, rootDir, this.conf, td);
+ HBaseTestingUtility.closeRegionAndWAL(region);
+ return new Pair<>(td, ri);
+ }
+
+ @Test
+ public void testCorruptRecoveredHFile() throws Exception {
+ Pair<TableDescriptor, RegionInfo> pair = setupTableAndRegion();
+ TableDescriptor td = pair.getFirst();
+ RegionInfo ri = pair.getSecond();
+
+ WAL wal = createWAL(this.conf, rootDir, logName);
+ HRegion region = HRegion.openHRegion(this.conf, this.fs, rootDir, ri, td, wal);
+ final long timestamp = this.ee.currentTime();
+ // Write data and flush
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ region.put(new Put(ROW).addColumn(cfd.getName(), Bytes.toBytes("x"), timestamp, VALUE1));
+ }
+ region.flush(true);
+
+ // Now assert edits made it in.
+ Result result1 = region.get(new Get(ROW));
+ assertEquals(td.getColumnFamilies().length, result1.size());
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ assertTrue(Bytes.equals(VALUE1, result1.getValue(cfd.getName(), Bytes.toBytes("x"))));
+ }
+
+ // Now close the region
+ region.close(true);
+ wal.shutdown();
+ // split the log
+ WALSplitter.split(rootDir, logDir, oldLogDir, FileSystem.get(this.conf), this.conf, wals);
+
+ // Write a corrupt recovered hfile
+ Path regionDir =
+ new Path(CommonFSUtils.getTableDir(rootDir, td.getTableName()), ri.getEncodedName());
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ FileStatus[] files =
+ WALSplitUtil.getRecoveredHFiles(this.fs, regionDir, cfd.getNameAsString());
+ assertNotNull(files);
+ assertTrue(files.length > 0);
+ writeCorruptRecoveredHFile(files[0].getPath());
+ }
+
+ // Failed to reopen the region
+ WAL wal2 = createWAL(this.conf, rootDir, logName);
+ try {
+ HRegion.openHRegion(this.conf, this.fs, rootDir, ri, td, wal2);
+ fail("Should fail to open region");
+ } catch (CorruptHFileException che) {
+ // Expected
+ }
+
+ // Set skip errors to true and reopen the region
+ this.conf.setBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS, true);
+ HRegion region2 = HRegion.openHRegion(this.conf, this.fs, rootDir, ri, td, wal2);
+ Result result2 = region2.get(new Get(ROW));
+ assertEquals(td.getColumnFamilies().length, result2.size());
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+ assertTrue(Bytes.equals(VALUE1, result2.getValue(cfd.getName(), Bytes.toBytes("x"))));
+ }
+ this.conf.setBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS, false);
Review comment:
Better move this line into setUp or tearDown if necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#issuecomment-602206144
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 56s | master passed |
| +1 :green_heart: | compile | 1m 0s | master passed |
| +1 :green_heart: | shadedjars | 5m 4s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 27s | the patch passed |
| +1 :green_heart: | compile | 1m 0s | the patch passed |
| +1 :green_heart: | javac | 1m 0s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 58s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 65m 34s | hbase-server in the patch passed. |
| | | 92m 48s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1322 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 75be5e5cadf9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ca533a77cb |
| Default Java | 1.8.0_232 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/testReport/ |
| Max. process+thread count | 5921 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/2/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1322: HBASE-24033 Add ut
for loading the corrupt recovered hfiles
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1322: HBASE-24033 Add ut for loading the corrupt recovered hfiles
URL: https://github.com/apache/hbase/pull/1322#issuecomment-602184041
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 6s | master passed |
| +1 :green_heart: | checkstyle | 1m 15s | master passed |
| -1 :x: | spotbugs | 2m 12s | hbase-server in master has 1 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 36s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 10s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 16s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. |
| +1 :green_heart: | spotbugs | 2m 14s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 37m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1322 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux f9fe52a809a2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / ca533a77cb |
| spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html |
| Max. process+thread count | 93 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1322/1/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services