You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/09/07 23:17:08 UTC

[GitHub] [hbase-operator-tools] z-york opened a new pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

z-york opened a new pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92


   …ables
   
   On clusters with a large number of tables, listing all table directories
   from the filesystem can take a lot of time (especially on S3).
   We should avoid listing all the tables if we pass in a subset of the tables


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] taklwu commented on a change in pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#discussion_r705690436



##########
File path: hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HBaseFsck.java
##########
@@ -2168,13 +2169,18 @@ public void loadHdfsRegionDirs() throws IOException, InterruptedException {
     // List all tables from HDFS
     List<FileStatus> tableDirs = Lists.newArrayList();
 
-    List<Path> paths = FSUtils.getTableDirs(fs, rootDir);
-    for (Path path : paths) {
-      TableName tableName = CommonFSUtils.getTableName(path);
-      if ((!checkMetaOnly && isTableIncluded(tableName)) ||
-          tableName.equals(TableName.META_TABLE_NAME)) {
-        tableDirs.add(fs.getFileStatus(path));
+    if (!checkMetaOnly) {
+      for (Path tableDir : this.tableDirs) {
+        try {
+          fs.getFileStatus(tableDir);
+        } catch (IOException ioe) {
+          LOG.warn("Failed to get Table directory for included table: {}",
+              CommonFSUtils.getTableName(tableDir), ioe);
+        }

Review comment:
       let me try to clarify and confirm, before this change if any give table name of a table directory does not exist, will hbck continue to run ? 
   
   but I see here , we have a `LOG.warn` instead of throwing out the IOE to the driver, doesn't it ?  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] Apache-HBase commented on pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#issuecomment-914697230


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   0m 38s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +0 :ok: |  spotbugs  |   0m  0s |  spotbugs executables are not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 53s |  master passed  |
   | +1 :green_heart: |  compile  |   0m  9s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   0m  6s |  master passed  |
   | +1 :green_heart: |  javadoc  |   0m  8s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 11s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 10s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 10s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   0m  4s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  javadoc  |   0m  6s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   3m  0s |  hbase-hbck2 in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m  5s |  The patch does not generate ASF License warnings.  |
   |  |   |   5m 39s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/2/artifact/yetus-precommit-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase-operator-tools/pull/92 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
   | uname | Linux 3c3460df7639 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux |
   | Build tool | maven |
   | git revision | master / 90e0f45 |
   | Default Java | Oracle Corporation-1.8.0_282-b08 |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/2/testReport/ |
   | Max. process+thread count | 1190 (vs. ulimit of 5000) |
   | modules | C: hbase-hbck2 U: hbase-hbck2 |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/2/console |
   | versions | git=2.20.1 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] z-york merged pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
z-york merged pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] taklwu commented on a change in pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#discussion_r705690436



##########
File path: hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HBaseFsck.java
##########
@@ -2168,13 +2169,18 @@ public void loadHdfsRegionDirs() throws IOException, InterruptedException {
     // List all tables from HDFS
     List<FileStatus> tableDirs = Lists.newArrayList();
 
-    List<Path> paths = FSUtils.getTableDirs(fs, rootDir);
-    for (Path path : paths) {
-      TableName tableName = CommonFSUtils.getTableName(path);
-      if ((!checkMetaOnly && isTableIncluded(tableName)) ||
-          tableName.equals(TableName.META_TABLE_NAME)) {
-        tableDirs.add(fs.getFileStatus(path));
+    if (!checkMetaOnly) {
+      for (Path tableDir : this.tableDirs) {
+        try {
+          fs.getFileStatus(tableDir);
+        } catch (IOException ioe) {
+          LOG.warn("Failed to get Table directory for included table: {}",
+              CommonFSUtils.getTableName(tableDir), ioe);
+        }

Review comment:
       let me try to clarify and confirm, before this change if any give table name of a table directory does not exist on HDFS, will hbck continue to run ? 
   
   but I see here , we have a `LOG.warn` instead of throwing out the IOE to the driver, doesn't it ?  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] taklwu commented on a change in pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#discussion_r705695953



##########
File path: hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HBaseFsck.java
##########
@@ -2168,13 +2169,18 @@ public void loadHdfsRegionDirs() throws IOException, InterruptedException {
     // List all tables from HDFS
     List<FileStatus> tableDirs = Lists.newArrayList();
 
-    List<Path> paths = FSUtils.getTableDirs(fs, rootDir);
-    for (Path path : paths) {
-      TableName tableName = CommonFSUtils.getTableName(path);
-      if ((!checkMetaOnly && isTableIncluded(tableName)) ||
-          tableName.equals(TableName.META_TABLE_NAME)) {
-        tableDirs.add(fs.getFileStatus(path));
+    if (!checkMetaOnly) {
+      for (Path tableDir : this.tableDirs) {
+        try {
+          fs.getFileStatus(tableDir);
+        } catch (IOException ioe) {
+          LOG.warn("Failed to get Table directory for included table: {}",
+              CommonFSUtils.getTableName(tableDir), ioe);
+        }

Review comment:
       sorry I missed this line `List<Path> paths = FSUtils.getTableDirs(fs, rootDir);` you're right, no behavior change, my bad




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] z-york commented on a change in pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
z-york commented on a change in pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#discussion_r705670536



##########
File path: hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HBaseFsck.java
##########
@@ -2168,13 +2169,18 @@ public void loadHdfsRegionDirs() throws IOException, InterruptedException {
     // List all tables from HDFS
     List<FileStatus> tableDirs = Lists.newArrayList();
 
-    List<Path> paths = FSUtils.getTableDirs(fs, rootDir);
-    for (Path path : paths) {
-      TableName tableName = CommonFSUtils.getTableName(path);
-      if ((!checkMetaOnly && isTableIncluded(tableName)) ||
-          tableName.equals(TableName.META_TABLE_NAME)) {
-        tableDirs.add(fs.getFileStatus(path));
+    if (!checkMetaOnly) {
+      for (Path tableDir : this.tableDirs) {
+        try {
+          fs.getFileStatus(tableDir);
+        } catch (IOException ioe) {
+          LOG.warn("Failed to get Table directory for included table: {}",
+              CommonFSUtils.getTableName(tableDir), ioe);
+        }

Review comment:
       This isn't really a behavior change since FSUtils.getTableDirs will do a list and only include those tables that exist on the FS (even if a table is passed that doesn't exist within the table dirs). With this change there isn't any checking that the directory exists before this getFileStatus so this is essentially doing the same thing as before.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] z-york commented on pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
z-york commented on pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#issuecomment-916425883


   > [nit] although I think `FileSystemFsck` has covered your change in the unit test of hbck2, can you show successful console results of run for hbck/hbck1?
   
   I also manually tested by forcing all the unit tests to specify the affected tables instead of just running a full hbck and all tests passed as well. I think there is sufficient testing with the unit tests, but I can run it on a cluster if you prefer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] taklwu commented on a change in pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
taklwu commented on a change in pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#discussion_r705539475



##########
File path: hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HBaseFsck.java
##########
@@ -2168,13 +2169,18 @@ public void loadHdfsRegionDirs() throws IOException, InterruptedException {
     // List all tables from HDFS
     List<FileStatus> tableDirs = Lists.newArrayList();
 
-    List<Path> paths = FSUtils.getTableDirs(fs, rootDir);
-    for (Path path : paths) {
-      TableName tableName = CommonFSUtils.getTableName(path);
-      if ((!checkMetaOnly && isTableIncluded(tableName)) ||
-          tableName.equals(TableName.META_TABLE_NAME)) {
-        tableDirs.add(fs.getFileStatus(path));
+    if (!checkMetaOnly) {
+      for (Path tableDir : this.tableDirs) {
+        try {
+          fs.getFileStatus(tableDir);
+        } catch (IOException ioe) {
+          LOG.warn("Failed to get Table directory for included table: {}",
+              CommonFSUtils.getTableName(tableDir), ioe);
+        }

Review comment:
       [nit] this seems like a behavior change, before this change, it will fail whenever the `fs.getFileStatus` failed for any given table. after this change, we still continue to work on it if any partial `fs.getFileStatus` failed. 
   
   I'm okie with this change, but want to have some note down on this PR. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hbase-operator-tools] Apache-HBase commented on pull request #92: HBASE-26257 Improve Performance of HBCK when specifying a subset of t…

Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #92:
URL: https://github.com/apache/hbase-operator-tools/pull/92#issuecomment-914689086


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  3s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +0 :ok: |  spotbugs  |   0m  0s |  spotbugs executables are not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 53s |  master passed  |
   | +1 :green_heart: |  compile  |   0m 10s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   0m  7s |  master passed  |
   | +1 :green_heart: |  javadoc  |   0m  7s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 10s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m  9s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m  9s |  the patch passed  |
   | -1 :x: |  checkstyle  |   0m  4s |  hbase-hbck2: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   | +1 :green_heart: |  javadoc  |   0m  5s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   3m  3s |  hbase-hbck2 in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m  6s |  The patch does not generate ASF License warnings.  |
   |  |   |   6m  5s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/1/artifact/yetus-precommit-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase-operator-tools/pull/92 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
   | uname | Linux a698eac114c8 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux |
   | Build tool | maven |
   | git revision | master / 90e0f45 |
   | Default Java | Oracle Corporation-1.8.0_282-b08 |
   | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/1/artifact/yetus-precommit-check/output/diff-checkstyle-hbase-hbck2.txt |
   |  Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/1/testReport/ |
   | Max. process+thread count | 1175 (vs. ulimit of 5000) |
   | modules | C: hbase-hbck2 U: hbase-hbck2 |
   | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-92/1/console |
   | versions | git=2.20.1 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org