You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ankitsol (via GitHub)" <gi...@apache.org> on 2025/08/26 11:14:22 UTC

[PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

ankitsol opened a new pull request, #7246:
URL: https://github.com/apache/hbase/pull/7246

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2331452611


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+      if (backupInfo.isContinuousBackupEnabled() && !Strings.isNullOrEmpty(backupRootDir)) {
+        String dayDirectoryName = BackupUtils.formatToDateString(bulkLoad.getTimestamp());
+        Path bulkLoadBackupPath =
+          new Path(backupRootDir, BULKLOAD_FILES_DIR + Path.SEPARATOR + dayDirectoryName);
+        Path bulkLoadDir = new Path(bulkLoadBackupPath,
+          srcTable.getNamespaceAsString() + Path.SEPARATOR + srcTable.getNameAsString());
+        FileSystem backupFs = FileSystem.get(bulkLoadDir.toUri(), conf);
+        Path fullBulkLoadBackupPath =
+          new Path(bulkLoadDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
+        if (backupFs.exists(fullBulkLoadBackupPath)) {
+          LOG.debug("Backup bulkload file found {}", fullBulkLoadBackupPath);
+          p = fullBulkLoadBackupPath;
+        } else {
+          LOG.warn("Backup bulkload file not found {}", fullBulkLoadBackupPath);

Review Comment:
   > In continuous backup approach, these files are instead copied from the bulkload backup location. I’ve added a warning here because if the required bulkload backup files are missing, they would be copied from the source cluster (as a fallback mechanism). 
   
   [nit] right, my concerns are the action after the WARN message, and current message does not clearly tell what the user should do when he see this, do you think we should add some suggestion in this message ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu merged PR #7246:
URL: https://github.com/apache/hbase/pull/7246


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3232029644

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 39s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  |  Patch does not have any anti-patterns.  |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  7s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 54s |  |  HBASE-28957 passed  |
   | -0 :warning: |  checkstyle  |   0m 17s | [/buildtool-branch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 44s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  spotless  |   1m  4s |  |  branch has no errors when running spotless:check.  |
   | -0 :warning: |  patch  |   1m 12s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  0s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 42s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 42s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 11s | [/buildtool-patch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/artifact/yetus-general-check/output/buildtool-patch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 54s |  |  the patch passed  |
   | +1 :green_heart: |  hadoopcheck  |  14m 41s |  |  Patch does not cause any errors with Hadoop 3.3.6 3.4.0.  |
   | +1 :green_heart: |  spotless  |   0m 57s |  |  patch has no errors when running spotless:check.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 13s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  38m 51s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
   | uname | Linux 07cd308c0697 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 9d52dc652d623cd9323a254d66506a58eda52adf |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   | Max. process+thread count | 83 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/console |
   | versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3229575752

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  6s |  |  https://github.com/apache/hbase/pull/7246 does not apply to HBASE-28957. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/2/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3231921816

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  7s |  |  https://github.com/apache/hbase/pull/7246 does not apply to HBASE-28957. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/4/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3234039992

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 31s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  |  Patch does not have any anti-patterns.  |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 19s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  HBASE-28957 passed  |
   | -0 :warning: |  checkstyle  |   0m 10s | [/buildtool-branch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 31s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  spotless  |   0m 49s |  |  branch has no errors when running spotless:check.  |
   | -0 :warning: |  patch  |   0m 56s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  7s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m  9s | [/buildtool-patch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/artifact/yetus-general-check/output/buildtool-patch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 36s |  |  the patch passed  |
   | +1 :green_heart: |  hadoopcheck  |  12m  9s |  |  Patch does not cause any errors with Hadoop 3.3.6 3.4.0.  |
   | +1 :green_heart: |  spotless  |   0m 45s |  |  patch has no errors when running spotless:check.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 10s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  30m 55s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
   | uname | Linux b9d1a659da4c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / d489d23f0eb544f67189be8f03f44b228923a912 |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   | Max. process+thread count | 83 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/console |
   | versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3234069994

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 32s |  |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck  |
   |||| _ Prechecks _ |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 18s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 19s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  javadoc  |   0m 15s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  shadedjars  |   6m  7s |  |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  patch  |   6m 15s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  7s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 19s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 19s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  2s |  |  patch has no errors when building our shaded downstream artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  19m 18s | [/patch-unit-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-backup.txt) |  hbase-backup in the patch failed.  |
   |  |   |  40m 32s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | javac javadoc unit compile shadedjars |
   | uname | Linux b6122eb6e945 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / d489d23f0eb544f67189be8f03f44b228923a912 |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   |  Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/testReport/ |
   | Max. process+thread count | 3756 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/7/console |
   | versions | git=2.34.1 maven=3.9.8 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3253332462

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 33s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  |  Patch does not have any anti-patterns.  |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 22s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  HBASE-28957 passed  |
   | -0 :warning: |  checkstyle  |   0m  9s | [/buildtool-branch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 31s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  spotless  |   0m 51s |  |  branch has no errors when running spotless:check.  |
   | -0 :warning: |  patch  |   0m 58s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  4s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m  9s | [/buildtool-patch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/artifact/yetus-general-check/output/buildtool-patch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 35s |  |  the patch passed  |
   | +1 :green_heart: |  hadoopcheck  |  12m 10s |  |  Patch does not cause any errors with Hadoop 3.3.6 3.4.0.  |
   | +1 :green_heart: |  spotless  |   0m 46s |  |  patch has no errors when running spotless:check.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 10s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  31m  7s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
   | uname | Linux fa5c0a4b126b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 12940d5babb0fe64488431fd7ce78cf52253941c |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   | Max. process+thread count | 84 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/console |
   | versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2322753266


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+      if (backupInfo.isContinuousBackupEnabled() && !Strings.isNullOrEmpty(backupRootDir)) {
+        String dayDirectoryName = BackupUtils.formatToDateString(bulkLoad.getTimestamp());
+        Path bulkLoadBackupPath =
+          new Path(backupRootDir, BULKLOAD_FILES_DIR + Path.SEPARATOR + dayDirectoryName);
+        Path bulkLoadDir = new Path(bulkLoadBackupPath,
+          srcTable.getNamespaceAsString() + Path.SEPARATOR + srcTable.getNameAsString());
+        FileSystem backupFs = FileSystem.get(bulkLoadDir.toUri(), conf);
+        Path fullBulkLoadBackupPath =
+          new Path(bulkLoadDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
+        if (backupFs.exists(fullBulkLoadBackupPath)) {
+          LOG.debug("Backup bulkload file found {}", fullBulkLoadBackupPath);
+          p = fullBulkLoadBackupPath;
+        } else {
+          LOG.warn("Backup bulkload file not found {}", fullBulkLoadBackupPath);

Review Comment:
   I assumed we're hitting the following
    1/ it does not have any bulkload previously, 
    2/ something is wrong?
    
    so, should we ask user to check if this is expected in the warning message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3244711906

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   2m 56s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  |  Patch does not have any anti-patterns.  |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 47s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 42s |  |  HBASE-28957 passed  |
   | -0 :warning: |  checkstyle  |   0m 11s | [/buildtool-branch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 34s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  spotless  |   0m 49s |  |  branch has no errors when running spotless:check.  |
   | -0 :warning: |  patch  |   0m 56s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m  9s | [/buildtool-patch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/artifact/yetus-general-check/output/buildtool-patch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 36s |  |  the patch passed  |
   | +1 :green_heart: |  hadoopcheck  |  12m 13s |  |  Patch does not cause any errors with Hadoop 3.3.6 3.4.0.  |
   | +1 :green_heart: |  spotless  |   0m 45s |  |  patch has no errors when running spotless:check.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 10s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  34m 14s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
   | uname | Linux d9b6ab7de5c9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / d9f3c950000016e395f24817ca80db95b33704e5 |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   | Max. process+thread count | 84 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/console |
   | versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3231921526

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  7s |  |  https://github.com/apache/hbase/pull/7246 does not apply to HBASE-28957. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/4/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "ankitsol (via GitHub)" <gi...@apache.org>.
ankitsol commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2331204207


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+      if (backupInfo.isContinuousBackupEnabled() && !Strings.isNullOrEmpty(backupRootDir)) {
+        String dayDirectoryName = BackupUtils.formatToDateString(bulkLoad.getTimestamp());
+        Path bulkLoadBackupPath =
+          new Path(backupRootDir, BULKLOAD_FILES_DIR + Path.SEPARATOR + dayDirectoryName);
+        Path bulkLoadDir = new Path(bulkLoadBackupPath,
+          srcTable.getNamespaceAsString() + Path.SEPARATOR + srcTable.getNameAsString());
+        FileSystem backupFs = FileSystem.get(bulkLoadDir.toUri(), conf);
+        Path fullBulkLoadBackupPath =
+          new Path(bulkLoadDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
+        if (backupFs.exists(fullBulkLoadBackupPath)) {
+          LOG.debug("Backup bulkload file found {}", fullBulkLoadBackupPath);
+          p = fullBulkLoadBackupPath;
+        } else {
+          LOG.warn("Backup bulkload file not found {}", fullBulkLoadBackupPath);

Review Comment:
   In non-continuous incremental backup approach, bulkload files are copied directly from the source cluster to the backup location.
   
   In continuous backup approach, these files are instead copied from the bulkload backup location. I’ve added a warning here because if the required bulkload backup files are missing, they would be copied from the source cluster (as a fallback mechanism).
   
   Source cluster files are only deleted after a successful full or incremental backup (in both non-continuous and continuous)
   
   Ideally, with continuous backups, once a bulkload file has been backed up by the replication endpoint, it could be safely deleted from the source cluster. However, doing so would significantly complicate the checkpoint logic, since it would then depend on both WAL flushes and bulkload backups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3229696785

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  7s |  |  https://github.com/apache/hbase/pull/7246 does not apply to HBASE-28957. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/3/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3229698387

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  8s |  |  https://github.com/apache/hbase/pull/7246 does not apply to HBASE-28957. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/3/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3233463429

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 36s |  |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck  |
   |||| _ Prechecks _ |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  5s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  javadoc  |   0m 14s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  shadedjars  |   6m  5s |  |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  patch  |   6m 13s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  7s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  4s |  |  patch has no errors when building our shaded downstream artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  22m 39s | [/patch-unit-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/6/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-backup.txt) |  hbase-backup in the patch failed.  |
   |  |   |  43m 43s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | javac javadoc unit compile shadedjars |
   | uname | Linux 4adff2155c7a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 4d89d3846bc05b1edda3ca430d456f0c2901cc9b |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   |  Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/6/testReport/ |
   | Max. process+thread count | 3784 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/6/console |
   | versions | git=2.34.1 maven=3.9.8 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2331452611


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+      if (backupInfo.isContinuousBackupEnabled() && !Strings.isNullOrEmpty(backupRootDir)) {
+        String dayDirectoryName = BackupUtils.formatToDateString(bulkLoad.getTimestamp());
+        Path bulkLoadBackupPath =
+          new Path(backupRootDir, BULKLOAD_FILES_DIR + Path.SEPARATOR + dayDirectoryName);
+        Path bulkLoadDir = new Path(bulkLoadBackupPath,
+          srcTable.getNamespaceAsString() + Path.SEPARATOR + srcTable.getNameAsString());
+        FileSystem backupFs = FileSystem.get(bulkLoadDir.toUri(), conf);
+        Path fullBulkLoadBackupPath =
+          new Path(bulkLoadDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
+        if (backupFs.exists(fullBulkLoadBackupPath)) {
+          LOG.debug("Backup bulkload file found {}", fullBulkLoadBackupPath);
+          p = fullBulkLoadBackupPath;
+        } else {
+          LOG.warn("Backup bulkload file not found {}", fullBulkLoadBackupPath);

Review Comment:
   > In continuous backup approach, these files are instead copied from the bulkload backup location. I’ve added a warning here because if the required bulkload backup files are missing, they would be copied from the source cluster (as a fallback mechanism). 
   
   [nit] right, my concerns are the action after the WARN message, and current message does not clearly tell what the user should do when he see this, do you think we should add some suggestion in this message ?
   
   we can improve it as a follow up JIRA or in the documentation, if user see this message , should they care about it? if so, what should they fix it ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3223907086

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 32s |  |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck  |
   |||| _ Prechecks _ |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 21s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  javadoc  |   0m 15s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  shadedjars  |   6m  7s |  |  branch has no errors when building our shaded downstream artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  4s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 14s |  |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  4s |  |  patch has no errors when building our shaded downstream artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  19m 10s |  |  hbase-backup in the patch passed.  |
   |  |   |  40m 34s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | javac javadoc unit compile shadedjars |
   | uname | Linux c70fffc003b3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 0802f3a98cede10799f05c16ac619ffbfe91ef82 |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   |  Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/testReport/ |
   | Max. process+thread count | 3696 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/console |
   | versions | git=2.34.1 maven=3.9.8 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3229578211

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  patch  |   0m  8s |  |  https://github.com/apache/hbase/pull/7246 does not apply to HBASE-28957. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/2/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3244752838

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   3m 43s |  |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck  |
   |||| _ Prechecks _ |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 23s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 28s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  javadoc  |   0m 16s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  shadedjars  |   6m 16s |  |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  patch  |   6m 25s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  7s |  |  patch has no errors when building our shaded downstream artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  22m 32s | [/patch-unit-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-backup.txt) |  hbase-backup in the patch failed.  |
   |  |   |  47m 35s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | javac javadoc unit compile shadedjars |
   | uname | Linux 83cef2807c9f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / d9f3c950000016e395f24817ca80db95b33704e5 |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   |  Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/testReport/ |
   | Max. process+thread count | 3774 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/8/console |
   | versions | git=2.34.1 maven=3.9.8 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3232034320

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 32s |  |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck  |
   |||| _ Prechecks _ |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 22s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 24s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  javadoc  |   0m 16s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  shadedjars  |   6m 11s |  |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  patch  |   6m 20s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 10s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 20s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  8s |  |  patch has no errors when building our shaded downstream artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  19m 18s | [/patch-unit-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-backup.txt) |  hbase-backup in the patch failed.  |
   |  |   |  41m  7s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | Optional Tests | javac javadoc unit compile shadedjars |
   | uname | Linux ccbbde135246 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 9d52dc652d623cd9323a254d66506a58eda52adf |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   |  Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/testReport/ |
   | Max. process+thread count | 3761 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/5/console |
   | versions | git=2.34.1 maven=3.9.8 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3223874212

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 29s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  |  Patch does not have any anti-patterns.  |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  5s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 32s |  |  HBASE-28957 passed  |
   | -0 :warning: |  checkstyle  |   0m 10s | [/buildtool-branch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 31s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  spotless  |   0m 49s |  |  branch has no errors when running spotless:check.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   2m 55s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 10s | [/buildtool-patch-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/artifact/yetus-general-check/output/buildtool-patch-checkstyle-hbase-backup.txt) |  The patch fails to run checkstyle in hbase-backup  |
   | +1 :green_heart: |  spotbugs  |   0m 37s |  |  the patch passed  |
   | +1 :green_heart: |  hadoopcheck  |  11m 27s |  |  Patch does not cause any errors with Hadoop 3.3.6 3.4.0.  |
   | +1 :green_heart: |  spotless  |   0m 45s |  |  patch has no errors when running spotless:check.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 10s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  29m 38s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
   | uname | Linux 1675e703e237 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 0802f3a98cede10799f05c16ac619ffbfe91ef82 |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   | Max. process+thread count | 88 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/1/console |
   | versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3233315300

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m  0s |  |  Docker mode activated.  |
   | -1 :x: |  docker  |   0m  3s |  |  Docker failed to build run-specific yetus/hbase:tp-3407}.  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/6/console |
   | versions | git=2.17.1 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2331452611


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+      if (backupInfo.isContinuousBackupEnabled() && !Strings.isNullOrEmpty(backupRootDir)) {
+        String dayDirectoryName = BackupUtils.formatToDateString(bulkLoad.getTimestamp());
+        Path bulkLoadBackupPath =
+          new Path(backupRootDir, BULKLOAD_FILES_DIR + Path.SEPARATOR + dayDirectoryName);
+        Path bulkLoadDir = new Path(bulkLoadBackupPath,
+          srcTable.getNamespaceAsString() + Path.SEPARATOR + srcTable.getNameAsString());
+        FileSystem backupFs = FileSystem.get(bulkLoadDir.toUri(), conf);
+        Path fullBulkLoadBackupPath =
+          new Path(bulkLoadDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
+        if (backupFs.exists(fullBulkLoadBackupPath)) {
+          LOG.debug("Backup bulkload file found {}", fullBulkLoadBackupPath);
+          p = fullBulkLoadBackupPath;
+        } else {
+          LOG.warn("Backup bulkload file not found {}", fullBulkLoadBackupPath);

Review Comment:
   > In continuous backup approach, these files are instead copied from the bulkload backup location. I’ve added a warning here because if the required bulkload backup files are missing, they would be copied from the source cluster (as a fallback mechanism). 
   
   [nit] right, my concerns are the action after the WARN message, and current message does not clearly tell what the user should do when he see this, do you think we should add some suggestion in this message ?
   
   we can improve it as a follow up JIRA or in the documentation, if user see this message , should they care about it? if so, what manual operation should they perform to get rid of it ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "ankitsol (via GitHub)" <gi...@apache.org>.
ankitsol commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2331206418


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);

Review Comment:
   Yes, makes sense. I will create a new JIRA to update this config from everywhere it is referenced



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #7246:
URL: https://github.com/apache/hbase/pull/7246#issuecomment-3253378540

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 36s |  |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  |  Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck  |
   |||| _ Prechecks _ |
   |||| _ HBASE-28957 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 17s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  compile  |   0m 20s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  javadoc  |   0m 15s |  |  HBASE-28957 passed  |
   | +1 :green_heart: |  shadedjars  |   6m  8s |  |  branch has no errors when building our shaded downstream artifacts.  |
   | -0 :warning: |  patch  |   6m 16s |  |  Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 19s |  |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 19s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  3s |  |  patch has no errors when building our shaded downstream artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  18m 35s |  |  hbase-backup in the patch passed.  |
   |  |   |  39m 54s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/hbase/pull/7246 |
   | JIRA Issue | HBASE-29520 |
   | Optional Tests | javac javadoc unit compile shadedjars |
   | uname | Linux b0e61d5ca550 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-28957 / 12940d5babb0fe64488431fd7ce78cf52253941c |
   | Default Java | Eclipse Adoptium-17.0.11+9 |
   |  Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/testReport/ |
   | Max. process+thread count | 3759 (vs. ulimit of 30000) |
   | modules | C: hbase-backup U: hbase-backup |
   | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/console |
   | versions | git=2.34.1 maven=3.9.8 |
   | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup [hbase]

Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu commented on code in PR #7246:
URL: https://github.com/apache/hbase/pull/7246#discussion_r2322784211


##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java:
##########
@@ -166,6 +167,26 @@ protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup) throws I
       Path tblDir = CommonFSUtils.getTableDir(rootdir, srcTable);
       Path p = new Path(tblDir, regionName + Path.SEPARATOR + fam + Path.SEPARATOR + filename);
 
+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);

Review Comment:
   [nit] somehow, `hbase.backup.continuous.wal.dir` is become the root now for both WALs and Bulkloads, shouldn't we call it as `hbase.backup.continuous.dir`/`hbase.backup.continuous.root.dir`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org