You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "vinayakphegde (via GitHub)" <gi...@apache.org> on 2025/02/24 16:54:52 UTC
[PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
vinayakphegde opened a new pull request, #6717:
URL: https://github.com/apache/hbase/pull/6717
This PR introduces the `pitr` command to restore tables from the most recent backup before `--to-datetime` and replay WAL logs for changes made after the last backup.
#### **Command Usage:**
```bash
hbase pitr <backup_path>
[-t <table_name[,table_name]>]
[-s <backup_set_name>]
[-q <name>]
[-c]
[-m <target_tables>]
[-o]
[--to-datetime <end_time>]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2922342728
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 36s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 9m 30s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 19s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 9m 18s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 0m 51s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 44s | | the patch passed |
| +1 :green_heart: | compile | 9m 59s | | the patch passed |
| -0 :warning: | javac | 9m 59s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1273 unchanged - 0 fixed = 1274 total (was 1273) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 26s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | spotbugs | 10m 28s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 13m 7s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 53s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 76m 24s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux c8bcb4e6215d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 04afea2a728acfad28eeb7f8e7d45c1b1a1b2f17 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 189 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2680270812
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 10s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 8s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 2m 7s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 5m 49s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 13s | | the patch passed |
| +1 :green_heart: | javac | 2m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed |
| +1 :green_heart: | shadedjars | 7m 7s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 277m 49s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 312m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 3b9e992d0499 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 606ff9eb584d65c2d751fa34732f2bbec2c34f45 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/testReport/ |
| Max. process+thread count | 6055 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2770384020
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 11s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 11s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 2m 26s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 6m 2s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 2m 7s | | the patch passed |
| +1 :green_heart: | javac | 2m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 10s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 53s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 350m 3s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 384m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux eaa86a45a5b8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / cf0f97ebd4c6c7b3ced7f97b7c11bc4a1a9b7828 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/testReport/ |
| Max. process+thread count | 5150 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2793429101
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 11s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 57s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 3m 23s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 7m 40s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 31s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 33s | | the patch passed |
| +1 :green_heart: | shadedjars | 7m 50s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 300m 23s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 346m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 921d9702e677 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / f0c0a7f41b623487cdfe3b12ebf351e2f2585ddf |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/testReport/ |
| Max. process+thread count | 5436 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2922245786
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 17s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 12s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 2m 10s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 5m 59s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 12s | | the patch passed |
| +1 :green_heart: | javac | 2m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 58s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 3m 9s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 32m 24s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux cfa9f4152a3f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 04afea2a728acfad28eeb7f8e7d45c1b1a1b2f17 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/testReport/ |
| Max. process+thread count | 315 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/11/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2114793555
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
+ LOG.debug(
+ "Found -check option in restore command, " + "will check and verify the dependencies");
+ }
+
+ if (cmd.hasOption(OPTION_SET) && cmd.hasOption(OPTION_TABLE)) {
+ System.err.println(
+ "Options -s and -t are mutually exclusive," + " you can not specify both of them.");
Review Comment:
Maybe add a little info to quickly remind the user what `-s` and `-t` do?
```suggestion
System.err.printf(
"Set name (-%s) and table list (-%s) are mutually exclusive, you can not specify both "
+ "of them.%n", OPTION_SET, OPTION_TABLE);
```
If you decide this isn't necessary, then please remove the unnecessary `+`:
```suggestion
System.err.println(
"Options -s and -t are mutually exclusive, you can not specify both of them.");
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
Review Comment:
nit - I see in Line 78 you're just using the `hasOption()` method and not assigning its output to a variable. Same with Lines 94, 101, and 107. Maybe this should be like that for consistency?
```suggestion
if (cmd.hasOption(OPTION_OVERWRITE)) {
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/replication/ContinuousBackupReplicationEndpoint.java:
##########
@@ -218,6 +230,12 @@ public ReplicationResult replicate(ReplicateContext replicateContext) {
backupWalEntries(entry.getKey(), entry.getValue());
}
+ // Capture the timestamp of the last WAL entry processed. This is used as the replication
+ // checkpoint
+ // so that point-in-time restores know the latest consistent time up to which replication has
+ // occurred.
Review Comment:
nit - It looks like there is an accidental newline after `checkpoint`.
```suggestion
// Capture the timestamp of the last WAL entry processed. This is used as the replication
// checkpoint so that point-in-time restores know the latest consistent time up to which
// replication has occurred.
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/RestoreDriver.java:
##########
@@ -17,117 +17,34 @@
*/
package org.apache.hadoop.hbase.backup;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
-
-import java.io.IOException;
import java.net.URI;
-import java.util.List;
-import java.util.Objects;
-import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
-import org.apache.hadoop.hbase.backup.impl.BackupManager;
-import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
import org.apache.hadoop.hbase.backup.util.BackupUtils;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
-import org.apache.hadoop.hbase.logging.Log4jUtils;
-import org.apache.hadoop.hbase.util.AbstractHBaseTool;
import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.hadoop.util.ToolRunner;
import org.apache.yetus.audience.InterfaceAudience;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
-import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
/**
* Command-line entry point for restore operation
*/
@InterfaceAudience.Private
-public class RestoreDriver extends AbstractHBaseTool {
- private static final Logger LOG = LoggerFactory.getLogger(RestoreDriver.class);
- private CommandLine cmd;
-
- private static final String USAGE_STRING =
- "Usage: hbase restore <backup_path> <backup_id> [options]\n"
- + " backup_path Path to a backup destination root\n"
- + " backup_id Backup image ID to restore\n"
- + " table(s) Comma-separated list of tables to restore\n";
-
- private static final String USAGE_FOOTER = "";
-
- protected RestoreDriver() throws IOException {
- init();
- }
-
- protected void init() {
- // disable irrelevant loggers to avoid it mess up command output
- Log4jUtils.disableZkAndClientLoggers();
- }
-
- private int parseAndRun() throws IOException {
- // Check if backup is enabled
- if (!BackupManager.isBackupEnabled(getConf())) {
- System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
- return -1;
- }
-
- // enable debug logging
- if (cmd.hasOption(OPTION_DEBUG)) {
- Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
- }
-
- // whether to overwrite to existing table if any, false by default
- boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
- if (overwrite) {
- LOG.debug("Found -overwrite option in restore command, "
- + "will overwrite to existing table if any in the restore target");
- }
-
- // whether to only check the dependencies, false by default
- boolean check = cmd.hasOption(OPTION_CHECK);
- if (check) {
- LOG.debug(
- "Found -check option in restore command, " + "will check and verify the dependencies");
- }
-
- if (cmd.hasOption(OPTION_SET) && cmd.hasOption(OPTION_TABLE)) {
- System.err.println(
- "Options -s and -t are mutaully exclusive," + " you can not specify both of them.");
- printToolUsage();
- return -1;
- }
-
- if (!cmd.hasOption(OPTION_SET) && !cmd.hasOption(OPTION_TABLE)) {
- System.err.println("You have to specify either set name or table list to restore");
- printToolUsage();
- return -1;
- }
-
- if (cmd.hasOption(OPTION_YARN_QUEUE_NAME)) {
- String queueName = cmd.getOptionValue(OPTION_YARN_QUEUE_NAME);
- // Set MR job queuename to configuration
- getConf().set("mapreduce.job.queuename", queueName);
- }
+public class RestoreDriver extends AbstractRestoreDriver {
+ private static final String USAGE_STRING = """
+ Usage: hbase restore <backup_path> <backup_id> [options]
Review Comment:
Should `<table(s)>` be in this usage line as well?
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
+ LOG.debug(
+ "Found -check option in restore command, " + "will check and verify the dependencies");
Review Comment:
nit - Same as the previous comment
```suggestion
LOG.debug(
"Found check option (-{}) in restore command, will check and verify the dependencies",
OPTION_CHECK);
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
Review Comment:
Same as the previous comment
```suggestion
if (cmd.hasOption(OPTION_CHECK)) {
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/PointInTimeRestoreDriver.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.CONF_CONTINUOUS_BACKUP_WAL_DIR;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME_DESC;
+
+import java.net.URI;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Command-line entry point for restore operation
+ */
+@InterfaceAudience.Private
+public class PointInTimeRestoreDriver extends AbstractRestoreDriver {
+ private static final String USAGE_STRING = """
+ Usage: hbase pitr [options]
+ <backup_path> Backup Path to use for Point in Time Restore
+ table(s) Comma-separated list of tables to restore
+ """;
+
+ @Override
+ protected int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite) {
+ String walBackupDir = getConf().get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ if (walBackupDir == null || walBackupDir.isEmpty()) {
Review Comment:
nit - You can use `Strings.isNullOrEmpty()` from `org.apache.hbase.thirdparty.com.google.common.base`:
```suggestion
if (Strings.isNullOrEmpty(walBackupDir)) {
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
Review Comment:
nit - I see you have `-o`, but from what I can tell `-overwrite` is not an available alternative option.
```suggestion
LOG.debug("Found overwrite option (-{}) in restore command, "
+ "will overwrite to existing table if any in the restore target", OPTION_OVERWRITE);
```
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
+ LOG.debug(
+ "Found -check option in restore command, " + "will check and verify the dependencies");
+ }
+
+ if (cmd.hasOption(OPTION_SET) && cmd.hasOption(OPTION_TABLE)) {
+ System.err.println(
+ "Options -s and -t are mutually exclusive," + " you can not specify both of them.");
+ printToolUsage();
+ return -1;
+ }
+
+ if (!cmd.hasOption(OPTION_SET) && !cmd.hasOption(OPTION_TABLE)) {
+ System.err.println("You have to specify either set name or table list to restore");
+ printToolUsage();
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_YARN_QUEUE_NAME)) {
+ String queueName = cmd.getOptionValue(OPTION_YARN_QUEUE_NAME);
+ // Set MR job queuename to configuration
+ getConf().set("mapreduce.job.queuename", queueName);
+ }
+
+ String tables;
+ TableName[] sTableArray;
+ TableName[] tTableArray;
+
+ String tableMapping =
+ cmd.hasOption(OPTION_TABLE_MAPPING) ? cmd.getOptionValue(OPTION_TABLE_MAPPING) : null;
Review Comment:
I believe `cmd.getOptionValue(OPTION_TABLE_MAPPING)` will just automatically return `null` if needed.
https://commons.apache.org/proper/commons-cli/apidocs/org/apache/commons/cli/CommandLine.html#getOptionValue(java.lang.String)
##########
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestPointInTimeRestore.java:
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.CONF_CONTINUOUS_BACKUP_WAL_DIR;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_ENABLE_CONTINUOUS_BACKUP;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.replication.ContinuousBackupReplicationEndpoint.ONE_DAY_IN_MILLISECONDS;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.util.ToolRunner;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category(LargeTests.class)
+public class TestPointInTimeRestore extends TestBackupBase {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestPointInTimeRestore.class);
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestPointInTimeRestore.class);
+
+ private static final String backupWalDirName = "TestPointInTimeRestoreWalDir";
+ private static final int WAIT_FOR_REPLICATION_MS = 30_000;
+ static Path backupWalDir;
+ static FileSystem fs;
+
+ @BeforeClass
+ public static void setupBeforeClass() throws Exception {
+ Path root = TEST_UTIL.getDataTestDirOnTestFS();
+ backupWalDir = new Path(root, backupWalDirName);
+ fs = FileSystem.get(conf1);
+ fs.mkdirs(backupWalDir);
+ conf1.set(CONF_CONTINUOUS_BACKUP_WAL_DIR, backupWalDir.toString());
+
+ setUpBackupUps();
+ }
+
+ /**
+ * Sets up multiple backups at different timestamps by: 1. Adjusting the system time to simulate
+ * past backup points. 2. Loading data into tables to create meaningful snapshots. 3. Running full
+ * backups with or without continuous backup enabled. 4. Ensuring replication is complete before
+ * proceeding.
+ */
+ private static void setUpBackupUps() throws Exception {
Review Comment:
Is this method name a typo? Should it say `setUpBackups()`?
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/PointInTimeRestoreDriver.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.CONF_CONTINUOUS_BACKUP_WAL_DIR;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME_DESC;
+
+import java.net.URI;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Command-line entry point for restore operation
+ */
+@InterfaceAudience.Private
+public class PointInTimeRestoreDriver extends AbstractRestoreDriver {
+ private static final String USAGE_STRING = """
+ Usage: hbase pitr [options]
+ <backup_path> Backup Path to use for Point in Time Restore
+ table(s) Comma-separated list of tables to restore
+ """;
+
+ @Override
+ protected int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite) {
+ String walBackupDir = getConf().get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ if (walBackupDir == null || walBackupDir.isEmpty()) {
+ System.err.printf(
+ "Point-in-Time Restore requires the WAL backup directory (%s) to replay logs after full and incremental backups. "
+ + "Set this property if you need Point-in-Time Restore. Otherwise, use the normal restore process with the appropriate backup ID.%n",
+ CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ return -1;
+ }
+
+ String[] remainArgs = cmd.getArgs();
+ if (remainArgs.length != 0) {
+ printToolUsage();
+ return -1;
+ }
+
+ String backupRootDir =
+ cmd.hasOption(OPTION_PITR_BACKUP_PATH) ? cmd.getOptionValue(OPTION_PITR_BACKUP_PATH) : null;
Review Comment:
`cmd.getOptionValue()` will automatically return `null` if the option is not set.
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
+ LOG.debug(
+ "Found -check option in restore command, " + "will check and verify the dependencies");
+ }
+
+ if (cmd.hasOption(OPTION_SET) && cmd.hasOption(OPTION_TABLE)) {
+ System.err.println(
+ "Options -s and -t are mutually exclusive," + " you can not specify both of them.");
+ printToolUsage();
+ return -1;
+ }
+
+ if (!cmd.hasOption(OPTION_SET) && !cmd.hasOption(OPTION_TABLE)) {
+ System.err.println("You have to specify either set name or table list to restore");
Review Comment:
Similar to the previous comment
```suggestion
System.err.printf("You have to specify either set name (-%s) or table list (-%s) to "
+ "restore\n", OPTION_SET, OPTION_TABLE);
```
##########
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestPointInTimeRestore.java:
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.CONF_CONTINUOUS_BACKUP_WAL_DIR;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_ENABLE_CONTINUOUS_BACKUP;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.replication.ContinuousBackupReplicationEndpoint.ONE_DAY_IN_MILLISECONDS;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.util.ToolRunner;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category(LargeTests.class)
+public class TestPointInTimeRestore extends TestBackupBase {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestPointInTimeRestore.class);
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestPointInTimeRestore.class);
+
+ private static final String backupWalDirName = "TestPointInTimeRestoreWalDir";
+ private static final int WAIT_FOR_REPLICATION_MS = 30_000;
+ static Path backupWalDir;
+ static FileSystem fs;
+
+ @BeforeClass
+ public static void setupBeforeClass() throws Exception {
+ Path root = TEST_UTIL.getDataTestDirOnTestFS();
+ backupWalDir = new Path(root, backupWalDirName);
+ fs = FileSystem.get(conf1);
+ fs.mkdirs(backupWalDir);
+ conf1.set(CONF_CONTINUOUS_BACKUP_WAL_DIR, backupWalDir.toString());
+
+ setUpBackupUps();
+ }
+
+ /**
+ * Sets up multiple backups at different timestamps by: 1. Adjusting the system time to simulate
+ * past backup points. 2. Loading data into tables to create meaningful snapshots. 3. Running full
+ * backups with or without continuous backup enabled. 4. Ensuring replication is complete before
+ * proceeding.
+ */
+ private static void setUpBackupUps() throws Exception {
+ // Simulate a backup taken 20 days ago
+ EnvironmentEdgeManager
+ .injectEdge(() -> System.currentTimeMillis() - 20 * ONE_DAY_IN_MILLISECONDS);
+ loadRandomData(table1, 1000); // Insert initial data into table1
+
+ // Perform a full backup for table1 with continuous backup enabled
+ String[] args = buildBackupArgs("full", new TableName[] { table1 }, true);
+ int ret = ToolRunner.run(conf1, new BackupDriver(), args);
+ assertEquals("Backup should succeed", 0, ret);
+
+ // Move time forward to simulate 15 days ago
+ EnvironmentEdgeManager
+ .injectEdge(() -> System.currentTimeMillis() - 15 * ONE_DAY_IN_MILLISECONDS);
+ loadRandomData(table1, 1000); // Add more data to table1
+ loadRandomData(table2, 500); // Insert data into table2
+
+ waitForReplication(); // Ensure replication is complete
+
+ // Perform a full backup for table2 with continuous backup enabled
+ args = buildBackupArgs("full", new TableName[] { table2 }, true);
+ ret = ToolRunner.run(conf1, new BackupDriver(), args);
+ assertEquals("Backup should succeed", 0, ret);
+
+ // Move time forward to simulate 10 days ago
+ EnvironmentEdgeManager
+ .injectEdge(() -> System.currentTimeMillis() - 10 * ONE_DAY_IN_MILLISECONDS);
+ loadRandomData(table2, 500); // Add more data to table2
+ loadRandomData(table3, 500); // Insert data into table3
+
+ // Perform a full backup for table3 and table4 (without continuous backup)
+ args = buildBackupArgs("full", new TableName[] { table3, table4 }, false);
+ ret = ToolRunner.run(conf1, new BackupDriver(), args);
+ assertEquals("Backup should succeed", 0, ret);
+
+ waitForReplication(); // Ensure replication is complete before concluding setup
+
+ // Reset time mocking to avoid affecting other tests
+ EnvironmentEdgeManager.reset();
+ }
+
+ @AfterClass
+ public static void setupAfterClass() throws IOException {
+ Path root = TEST_UTIL.getDataTestDirOnTestFS();
+ Path backupWalDir = new Path(root, backupWalDirName);
+ FileSystem fs = FileSystem.get(conf1);
+
+ if (fs.exists(backupWalDir)) {
+ fs.delete(backupWalDir, true);
+ }
+
+ conf1.unset(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ }
+
+ /**
+ * Verifies that PITR (Point-in-Time Restore) fails when the requested restore time is either in
+ * the future or outside the allowed retention window.
+ */
+ @Test
+ public void testPITR_FailsOutsideWindow() throws Exception {
+ // Case 1: Requested restore time is in the future (should fail)
+ String[] args = buildPITRArgs(new TableName[] { table1 },
+ new TableName[] { TableName.valueOf("restoredTable1") },
+ EnvironmentEdgeManager.currentTime() + ONE_DAY_IN_MILLISECONDS);
+
+ int ret = ToolRunner.run(conf1, new PointInTimeRestoreDriver(), args);
+ assertNotEquals("Restore should fail since the requested restore time is in the future", 0,
+ ret);
+
+ // Case 2: Requested restore time is too old (beyond the retention window, should fail)
+ args = buildPITRArgs(new TableName[] { table1 },
+ new TableName[] { TableName.valueOf("restoredTable1") },
+ EnvironmentEdgeManager.currentTime() - 40 * ONE_DAY_IN_MILLISECONDS);
+
+ ret = ToolRunner.run(conf1, new PointInTimeRestoreDriver(), args);
+ assertNotEquals(
+ "Restore should fail since the requested restore time is outside the retention window", 0,
+ ret);
+ }
+
+ /**
+ * Ensures that PITR fails when attempting to restore tables where continuous backup was not
+ * enabled.
+ */
+ @Test
+ public void testPointInTimeRestore_ContinuousBackupNotEnabledTables() throws Exception {
+ String[] args = buildPITRArgs(new TableName[] { table3 },
+ new TableName[] { TableName.valueOf("restoredTable1") },
+ EnvironmentEdgeManager.currentTime() - 10 * ONE_DAY_IN_MILLISECONDS);
+
+ int ret = ToolRunner.run(conf1, new PointInTimeRestoreDriver(), args);
+ assertNotEquals("Restore should fail since continuous backup is not enabled for the table", 0,
+ ret);
+ }
+
+ /**
+ * Ensures that PITR fails when trying to restore from a point before continuous backup started.
+ */
+ @Test
+ public void testPointInTimeRestore_TablesWithNoProperBackupOrWals() throws Exception {
+ String[] args = buildPITRArgs(new TableName[] { table2 },
+ new TableName[] { TableName.valueOf("restoredTable1") },
+ EnvironmentEdgeManager.currentTime() - 16 * ONE_DAY_IN_MILLISECONDS);
+
+ int ret = ToolRunner.run(conf1, new PointInTimeRestoreDriver(), args);
+ assertNotEquals(
+ "Restore should fail since the requested restore point is before the start of continuous backup",
+ 0, ret);
+ }
+
+ /**
+ * Verifies that PITR successfully restores data for a single table.
+ */
+ @Test
+ public void testPointInTimeRestore_SuccessfulRestoreForOneTable() throws Exception {
+ TableName restoredTable = TableName.valueOf("restoredTable");
+
+ // Perform restore operation
+ String[] args = buildPITRArgs(new TableName[] { table1 }, new TableName[] { restoredTable },
+ EnvironmentEdgeManager.currentTime() - 5 * ONE_DAY_IN_MILLISECONDS);
+
+ int ret = ToolRunner.run(conf1, new PointInTimeRestoreDriver(), args);
+ assertEquals("Restore should succeed", 0, ret);
+
+ // Validate that the restored table contains the same number of rows as the original table
+ assertEquals("Restored table should have the same row count as the original",
+ getRowCount(table1), getRowCount(restoredTable));
+ }
+
+ /**
+ * Verifies that PITR successfully restores multiple tables at once.
+ */
+ @Test
+ public void testPointInTimeRestore_SuccessfulRestoreForMultipleTables() throws Exception {
+ TableName restoredTable1 = TableName.valueOf("restoredTable1");
+ TableName restoredTable2 = TableName.valueOf("restoredTable2");
+
+ // Perform restore operation for multiple tables
+ String[] args = buildPITRArgs(new TableName[] { table1, table2 },
+ new TableName[] { restoredTable1, restoredTable2 },
+ EnvironmentEdgeManager.currentTime() - 5 * ONE_DAY_IN_MILLISECONDS);
+
+ int ret = ToolRunner.run(conf1, new PointInTimeRestoreDriver(), args);
+ assertEquals("Restore should succeed", 0, ret);
+
+ // Validate that the restored tables contain the same number of rows as the originals
+ assertEquals("Restored table1 should have the same row count as the original",
+ getRowCount(table1), getRowCount(restoredTable1));
+ assertEquals("Restored table2 should have the same row count as the original",
+ getRowCount(table2), getRowCount(restoredTable2));
+ }
+
+ private String[] buildPITRArgs(TableName[] sourceTables, TableName[] targetTables, long endTime) {
+ String sourceTableNames =
+ Arrays.stream(sourceTables).map(TableName::getNameAsString).collect(Collectors.joining(","));
+
+ String targetTableNames =
+ Arrays.stream(targetTables).map(TableName::getNameAsString).collect(Collectors.joining(","));
+
+ return new String[] { "-t", sourceTableNames, "-m", targetTableNames, "-" + OPTION_TO_DATETIME,
Review Comment:
To be safe, you can use `"-" + OPTION_TABLE` instead of `-t` in case the option is ever changed. Similar for `-m` and the args in `buildBackupArgs()`.
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java:
##########
@@ -509,6 +520,338 @@ public void restore(RestoreRequest request) throws IOException {
new RestoreTablesClient(conn, request).execute();
}
+ private boolean validateRequest(RestoreRequest request) throws IOException {
+ // check and load backup image manifest for the tables
+ Path rootPath = new Path(request.getBackupRootDir());
+ String backupId = request.getBackupId();
+ TableName[] sTableArray = request.getFromTables();
+ BackupManifest manifest =
+ HBackupFileSystem.getManifest(conn.getConfiguration(), rootPath, backupId);
+
+ // Validate the backup image and its dependencies
+ return BackupUtils.validate(Arrays.asList(sTableArray), manifest, conn.getConfiguration());
+ }
+
+ @Override
+ public void pointInTimeRestore(PointInTimeRestoreRequest request) throws IOException {
+ if (request.getBackupRootDir() == null) {
+ defaultPointInTimeRestore(request);
+ } else {
+ // TODO: special case, not supported at the moment
+ throw new IOException("Custom backup location for Point-In-Time Recovery Not supported!");
+ }
+ LOG.info("Successfully completed Point In Time Restore for all tables.");
+ }
+
+ /**
+ * Performs a default Point-In-Time Restore (PITR) by restoring the latest valid backup and
+ * replaying the WALs to bring the table to the desired state. PITR requires: 1. A valid backup
+ * available before the end time. 2. Write-Ahead Logs (WALs) covering the remaining duration up to
+ * the end time.
+ * @param request PointInTimeRestoreRequest containing restore parameters.
+ * @throws IOException If no valid backup or WALs are found, or if an error occurs during
+ * restoration.
+ */
+ private void defaultPointInTimeRestore(PointInTimeRestoreRequest request) throws IOException {
+ long endTime = request.getToDateTime();
+ validateRequestToTime(endTime);
+
+ TableName[] sTableArray = request.getFromTables();
+ TableName[] tTableArray = resolveTargetTables(sTableArray, request.getToTables());
+
+ // Validate PITR requirements
+ validatePitr(endTime, sTableArray, tTableArray);
+
+ // If only validation is required, log and return
+ if (request.isCheck()) {
+ LOG.info("PITR can be successfully executed");
+ return;
+ }
+
+ // Execute PITR process
+ try (BackupSystemTable table = new BackupSystemTable(conn)) {
+ Map<TableName, Long> continuousBackupTables = table.getContinuousBackupTableSet();
+ List<BackupInfo> backupInfos = table.getBackupInfos(BackupState.COMPLETE);
+
+ for (int i = 0; i < sTableArray.length; i++) {
+ restoreTableWithWalReplay(sTableArray[i], tTableArray[i], endTime, continuousBackupTables,
+ backupInfos, request);
+ }
+ }
+ }
+
+ /**
+ * Validates whether the requested end time falls within the allowed PITR recovery window.
+ * @param endTime The target recovery time.
+ * @throws IOException If the requested recovery time is outside the allowed window.
+ */
+ private void validateRequestToTime(long endTime) throws IOException {
+ long pitrWindowDays = conn.getConfiguration().getLong(CONF_CONTINUOUS_BACKUP_PITR_WINDOW_DAYS,
+ DEFAULT_CONTINUOUS_BACKUP_PITR_WINDOW_DAYS);
+ long currentTime = EnvironmentEdgeManager.getDelegate().currentTime();
+ long pitrMaxStartTime = currentTime - TimeUnit.DAYS.toMillis(pitrWindowDays);
+
+ if (endTime < pitrMaxStartTime) {
+ String errorMsg = String.format(
+ "Requested recovery time (%d) is out of the allowed PITR window (last %d days).", endTime,
+ pitrWindowDays);
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+
+ if (endTime > currentTime) {
+ String errorMsg = String.format(
+ "Requested recovery time (%d) is in the future. Current time: %d.", endTime, currentTime);
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+ }
+
+ /**
+ * Resolves the target table array. If null or empty, defaults to the source table array.
+ */
+ private TableName[] resolveTargetTables(TableName[] sourceTables, TableName[] targetTables) {
+ return (targetTables == null || targetTables.length == 0) ? sourceTables : targetTables;
+ }
+
+ /**
+ * Validates whether Point-In-Time Recovery (PITR) is possible for the given tables at the
+ * specified time.
+ * <p>
+ * PITR requires:
+ * <ul>
+ * <li>Continuous backup to be enabled for the source tables.</li>
+ * <li>A valid backup image and corresponding WALs to be available.</li>
+ * </ul>
+ * @param endTime The target recovery time.
+ * @param sTableArray The source tables to restore.
+ * @param tTableArray The target tables where the restore will be performed.
+ * @throws IOException If PITR is not possible due to missing continuous backup or backup images.
+ */
+ private void validatePitr(long endTime, TableName[] sTableArray, TableName[] tTableArray)
+ throws IOException {
+ try (BackupSystemTable table = new BackupSystemTable(conn)) {
+ // Retrieve the set of tables with continuous backup enabled
+ Map<TableName, Long> continuousBackupTables = table.getContinuousBackupTableSet();
+
+ // Ensure all source tables have continuous backup enabled
+ validateContinuousBackup(sTableArray, continuousBackupTables);
+
+ // Fetch completed backup information
+ List<BackupInfo> backupInfos = table.getBackupInfos(BackupState.COMPLETE);
+
+ // Ensure a valid backup and WALs exist for PITR
+ validateBackupAvailability(sTableArray, tTableArray, endTime, continuousBackupTables,
+ backupInfos);
+ }
+ }
+
+ /**
+ * Ensures that all source tables have continuous backup enabled.
+ */
+ private void validateContinuousBackup(TableName[] tables,
+ Map<TableName, Long> continuousBackupTables) throws IOException {
+ List<TableName> missingTables =
+ Arrays.stream(tables).filter(table -> !continuousBackupTables.containsKey(table)).toList();
+
+ if (!missingTables.isEmpty()) {
+ String errorMsg = "Continuous Backup is not enabled for the following tables: "
+ + missingTables.stream().map(TableName::getNameAsString).collect(Collectors.joining(", "));
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+ }
+
+ /**
+ * Ensures that a valid backup and corresponding WALs exist for PITR for each source table. PITR
+ * requires: 1. A valid backup available before the end time. 2. Write-Ahead Logs (WALs) covering
+ * the remaining duration up to the end time.
+ */
+ private void validateBackupAvailability(TableName[] sTableArray, TableName[] tTableArray,
+ long endTime, Map<TableName, Long> continuousBackupTables, List<BackupInfo> backupInfos)
+ throws IOException {
+ for (int i = 0; i < sTableArray.length; i++) {
+ if (
+ !canPerformPitr(sTableArray[i], tTableArray[i], endTime, continuousBackupTables,
+ backupInfos)
+ ) {
+ String errorMsg = "Could not find a valid backup and WALs for PITR for table: "
+ + sTableArray[i].getNameAsString();
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+ }
+ }
+
+ /**
+ * Checks whether PITR can be performed for a given source-target table pair.
+ */
+ private boolean canPerformPitr(TableName stableName, TableName tTableName, long endTime,
+ Map<TableName, Long> continuousBackupTables, List<BackupInfo> backupInfos) {
+ return getValidBackupInfo(stableName, tTableName, endTime, continuousBackupTables, backupInfos)
+ != null;
+ }
+
+ /**
+ * Finds a valid backup for PITR that meets the required conditions.
+ */
+ private BackupInfo getValidBackupInfo(TableName sTableName, TableName tTablename, long endTime,
+ Map<TableName, Long> continuousBackupTables, List<BackupInfo> backupInfos) {
+ for (BackupInfo info : backupInfos) {
+ if (isValidBackupForPitr(info, sTableName, endTime, continuousBackupTables)) {
+
+ RestoreRequest restoreRequest =
+ BackupUtils.createRestoreRequest(info.getBackupRootDir(), info.getBackupId(), true,
+ new TableName[] { sTableName }, new TableName[] { tTablename }, false);
+
+ try {
+ if (validateRequest(restoreRequest)) {
+ return info;
+ }
+ } catch (IOException e) {
+ LOG.warn("Exception occurred while testing the backup : {} for restore ", info, e);
+ }
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Determines if the given backup is valid for PITR.
+ * <p>
+ * A backup is valid if:
+ * <ul>
+ * <li>It contains the source table.</li>
+ * <li>It was completed before the end time.</li>
+ * <li>The start timestamp of the backup is after the continuous backup start time for the
+ * table.</li>
+ * </ul>
+ * @param info Backup information object.
+ * @param tableName Table to check.
+ * @param endTime The target recovery time.
+ * @param continuousBackupTables Map of tables with continuous backup enabled.
+ * @return true if the backup is valid for PITR, false otherwise.
+ */
+ private boolean isValidBackupForPitr(BackupInfo info, TableName tableName, long endTime,
+ Map<TableName, Long> continuousBackupTables) {
+ return info.getTableNames().contains(tableName) && info.getCompleteTs() <= endTime
+ && continuousBackupTables.getOrDefault(tableName, 0L) <= info.getStartTs();
+ }
+
+ /**
+ * Restores a table from a valid backup and replays WALs to reach the desired PITR state.
+ */
+ private void restoreTableWithWalReplay(TableName sourceTable, TableName targetTable, long endTime,
+ Map<TableName, Long> continuousBackupTables, List<BackupInfo> backupInfos,
+ PointInTimeRestoreRequest request) throws IOException {
+ BackupInfo backupInfo =
+ getValidBackupInfo(sourceTable, targetTable, endTime, continuousBackupTables, backupInfos);
+ if (backupInfo == null) {
+ String errorMsg = "Could not find a valid backup and WALs for PITR for table: "
+ + sourceTable.getNameAsString();
+ LOG.error(errorMsg);
+ throw new IOException(errorMsg);
+ }
+
+ RestoreRequest restoreRequest = BackupUtils.createRestoreRequest(backupInfo.getBackupRootDir(),
+ backupInfo.getBackupId(), false, new TableName[] { sourceTable },
+ new TableName[] { targetTable }, request.isOverwrite());
+
+ restore(restoreRequest);
+ replayWal(sourceTable, targetTable, backupInfo.getStartTs(), endTime);
+ }
+
+ /**
+ * Replays WALs to bring the table to the desired state.
+ */
+ private void replayWal(TableName sourceTable, TableName targetTable, long startTime, long endTime)
+ throws IOException {
+ String walBackupDir = conn.getConfiguration().get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ Path walDirPath = new Path(walBackupDir);
+ LOG.info(
+ "Starting WAL replay for source: {}, target: {}, time range: {} - {}, WAL backup dir: {}",
+ sourceTable, targetTable, startTime, endTime, walDirPath);
+
+ List<String> validDirs =
+ getValidWalDirs(conn.getConfiguration(), walDirPath, startTime, endTime);
+ if (validDirs.isEmpty()) {
+ LOG.warn("No valid WAL directories found for range {} - {}. Skipping WAL replay.", startTime,
+ endTime);
+ return;
+ }
+
+ executeWalReplay(validDirs, sourceTable, targetTable, startTime, endTime);
+ }
+
+ /**
+ * Fetches valid WAL directories based on the given time range.
+ */
+ private List<String> getValidWalDirs(Configuration conf, Path walBackupDir, long startTime,
+ long endTime) throws IOException {
+ FileSystem backupFs = FileSystem.get(walBackupDir.toUri(), conf);
+ FileStatus[] dayDirs = backupFs.listStatus(new Path(walBackupDir, WALS_DIR));
+
+ List<String> validDirs = new ArrayList<>();
+ SimpleDateFormat dateFormat = new SimpleDateFormat(DATE_FORMAT);
+
+ for (FileStatus dayDir : dayDirs) {
+ if (!dayDir.isDirectory()) {
+ continue; // Skip files, only process directories
+ }
+
+ String dirName = dayDir.getPath().getName();
+ try {
+ Date dirDate = dateFormat.parse(dirName);
+ long dirStartTime = dirDate.getTime(); // Start of that day (00:00:00)
+ long dirEndTime = dirStartTime + ONE_DAY_IN_MILLISECONDS - 1; // End time of the day
+ // (23:59:59)
Review Comment:
It feels a little odd to have an inline comment on two lines. It's probably because of `mvn spotless:apply`. You can probably get it all on one line if you trim the comment down:
```suggestion
long dirEndTime = dirStartTime + ONE_DAY_IN_MILLISECONDS - 1; // End time of day (23:59:59)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2754134768
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 12s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 7m 51s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 10s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 7m 50s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 0m 44s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 0s | | the patch passed |
| +1 :green_heart: | compile | 7m 50s | | the patch passed |
| -0 :warning: | javac | 7m 50s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1273 unchanged - 0 fixed = 1274 total (was 1273) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 10s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | spotbugs | 8m 2s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 48s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 20s | | The patch does not generate ASF License warnings. |
| | | 62m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux b2c23608dc02 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / dc20fbe957936fc024e211c8c4cab85edace5097 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 189 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2114744849
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
Review Comment:
nit - I see you have `-o`, but from what I can tell `-overwrite` is not an available alternative option. It may be more clear to have the actual option in parenthesis.
```suggestion
LOG.debug("Found overwrite option (-{}) in restore command, "
+ "will overwrite to existing table if any in the restore target", OPTION_OVERWRITE);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2790625668
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 1s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 30s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 7m 58s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 11s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 7m 57s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 0m 45s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 1s | | the patch passed |
| +1 :green_heart: | compile | 9m 39s | | the patch passed |
| -0 :warning: | javac | 9m 39s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1273 unchanged - 0 fixed = 1274 total (was 1273) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 27s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | spotbugs | 10m 59s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 59s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 46s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 21s | | The patch does not generate ASF License warnings. |
| | | 69m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux 0623615afd49 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 4fa1c3b31e3266c2f177480a92af433d2ddc50cd |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 191 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2679833313
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 3s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 12m 56s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 51s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 11m 43s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 0m 52s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 12m 28s | | the patch passed |
| -0 :warning: | javac | 12m 28s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1281 unchanged - 0 fixed = 1282 total (was 1281) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 34s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | spotbugs | 12m 5s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 15m 53s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 8s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 91m 20s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux 35ff8a72c6c1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 606ff9eb584d65c2d751fa34732f2bbec2c34f45 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 188 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2116259352
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
Review Comment:
Got it, I missed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2914556397
@vinayakphegde You also need to rebase the patch. Some flaky tests have already been fixed on the feature branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2769353514
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 12s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 8m 3s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 11s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 7m 55s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 0m 46s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 4s | | the patch passed |
| +1 :green_heart: | compile | 8m 0s | | the patch passed |
| -0 :warning: | javac | 8m 0s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1273 unchanged - 0 fixed = 1274 total (was 1273) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 9s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | spotbugs | 8m 17s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 55s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 20s | | The patch does not generate ASF License warnings. |
| | | 63m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux ea6bd2b72cf5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / cf0f97ebd4c6c7b3ced7f97b7c11bc4a1a9b7828 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 190 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/7/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "taklwu (via GitHub)" <gi...@apache.org>.
taklwu merged PR #6717:
URL: https://github.com/apache/hbase/pull/6717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2917985019
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 11s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 13s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 2m 11s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 5m 59s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 9s | | the patch passed |
| +1 :green_heart: | javac | 2m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 12s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 56s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 247m 12s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 280m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 43e28404db18 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 8063bbe39ea421ae2df1980f20d235270c76f366 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/testReport/ |
| Max. process+thread count | 6046 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2065030821
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java:
##########
@@ -770,4 +789,156 @@ public static String findMostRecentBackupId(String[] backupIds) {
return BackupRestoreConstants.BACKUPID_PREFIX + recentTimestamp;
}
+ /**
+ * Calculates the replication checkpoint timestamp used for continuous backup.
+ * <p>
+ * A replication checkpoint is the earliest timestamp across all region servers such that every
+ * WAL entry before that point is known to be replicated to the target system. This is essential
+ * for features like Point-in-Time Restore (PITR) and incremental backups, where we want to
+ * confidently restore data to a consistent state without missing updates.
+ * <p>
+ * The checkpoint is calculated using a combination of:
+ * <ul>
+ * <li>The start timestamps of WAL files currently being replicated for each server.</li>
+ * <li>The latest successfully replicated timestamp recorded by the replication marker chore.</li>
+ * </ul>
+ * <p>
+ * We combine these two sources to handle the following challenges:
+ * <ul>
+ * <li><b>Stale WAL start times:</b> If replication traffic is low or WALs are long-lived, the
+ * replication offset may point to the same WAL for a long time, resulting in stale timestamps
+ * that underestimate progress. This could delay PITR unnecessarily.</li>
+ * <li><b>Limitations of marker-only tracking:</b> The replication marker chore stores the last
+ * successfully replicated timestamp per region server in a system table. However, this data may
+ * become stale if the server goes offline or region ownership changes. For example, if a region
+ * initially belonged to rs1 and was later moved to rs4 due to re-balancing, rs1’s marker would
+ * persist even though it no longer holds any regions. Relying solely on these stale markers could
+ * lead to incorrect or outdated checkpoints.</li>
+ * </ul>
+ * <p>
+ * To handle these limitations, the method:
+ * <ol>
+ * <li>Verifies that the continuous backup peer exists to ensure replication is enabled.</li>
+ * <li>Retrieves WAL replication queue information for the peer, collecting WAL start times per
+ * region server. This gives us a lower bound for replication progress.</li>
+ * <li>Reads the marker chore's replicated timestamps from the backup system table.</li>
+ * <li>For servers found in both sources, if the marker timestamp is more recent than the WAL's
+ * start timestamp, we use the marker (since replication has progressed beyond the WAL).</li>
+ * <li>We discard marker entries for region servers that are not present in WAL queues, assuming
+ * those servers are no longer relevant (e.g., decommissioned or reassigned).</li>
+ * <li>The checkpoint is the minimum of all chosen timestamps — i.e., the slowest replicating
+ * region server.</li>
+ * <li>Finally, we persist the updated marker information to include any newly participating
+ * region servers.</li>
+ * </ol>
+ * <p>
+ * Note: If the replication marker chore is disabled, we fall back to using only the WAL start
+ * times. This ensures correctness but may lead to conservative checkpoint estimates during idle
+ * periods.
+ * @param conn the HBase connection
+ * @return the calculated replication checkpoint timestamp
+ * @throws IOException if reading replication queues or updating the backup system table fails
+ */
+ public static long getReplicationCheckpoint(Connection conn) throws IOException {
+ Configuration conf = conn.getConfiguration();
+ long checkpoint = EnvironmentEdgeManager.getDelegate().currentTime();
+
+ // Step 1: Ensure the continuous backup replication peer exists
+ if (!continuousBackupReplicationPeerExists(conn.getAdmin())) {
+ String msg = "Replication peer '" + CONTINUOUS_BACKUP_REPLICATION_PEER
+ + "' not found. Continuous backup not enabled.";
+ LOG.error(msg);
+ throw new IOException(msg);
+ }
+
+ // Step 2: Get all replication queues for the continuous backup peer
+ ReplicationQueueStorage queueStorage =
+ ReplicationStorageFactory.getReplicationQueueStorage(conn, conf);
+
+ List<ReplicationQueueId> queueIds;
+ try {
+ queueIds = queueStorage.listAllQueueIds(CONTINUOUS_BACKUP_REPLICATION_PEER);
+ } catch (ReplicationException e) {
+ String msg = "Failed to retrieve replication queue IDs for peer '"
+ + CONTINUOUS_BACKUP_REPLICATION_PEER + "'";
+ LOG.error(msg, e);
+ throw new IOException(msg, e);
+ }
+
+ if (queueIds.isEmpty()) {
+ String msg = "Replication peer '" + CONTINUOUS_BACKUP_REPLICATION_PEER + "' has no queues. "
+ + "This may indicate that continuous backup replication is not initialized correctly.";
+ LOG.error(msg);
+ throw new IOException(msg);
+ }
+
+ // Step 3: Build a map of ServerName -> WAL start timestamp (lowest seen per server)
+ Map<ServerName, Long> serverToCheckpoint = new HashMap<>();
+ for (ReplicationQueueId queueId : queueIds) {
+ Map<String, ReplicationGroupOffset> offsets;
+ try {
+ offsets = queueStorage.getOffsets(queueId);
+ } catch (ReplicationException e) {
+ String msg = "Failed to fetch WAL offsets for replication queue: " + queueId;
+ LOG.error(msg, e);
+ throw new IOException(msg, e);
+ }
+
+ for (ReplicationGroupOffset offset : offsets.values()) {
+ String walFile = offset.getWal();
+ long ts = AbstractFSWALProvider.getTimestamp(walFile); // WAL creation time
+ ServerName server = queueId.getServerName();
+ // Store the minimum timestamp per server (ts - 1 to avoid edge boundary issues)
+ serverToCheckpoint.merge(server, ts - 1, Math::min);
+ }
+ }
+
+ // Step 4: If replication markers are enabled, overlay fresher timestamps from backup system
+ // table
+ boolean replicationMarkerEnabled =
+ conf.getBoolean(REPLICATION_MARKER_ENABLED_KEY, REPLICATION_MARKER_ENABLED_DEFAULT);
+ if (replicationMarkerEnabled) {
+ try (BackupSystemTable backupSystemTable = new BackupSystemTable(conn)) {
+ Map<ServerName, Long> markerTimestamps = backupSystemTable.getBackupCheckpointTimestamps();
+
+ for (Map.Entry<ServerName, Long> entry : markerTimestamps.entrySet()) {
+ ServerName server = entry.getKey();
+ long markerTs = entry.getValue();
+
+ // If marker timestamp is newer, override
+ if (serverToCheckpoint.containsKey(server)) {
+ long current = serverToCheckpoint.get(server);
+ if (markerTs > current) {
+ serverToCheckpoint.put(server, markerTs);
+ }
+ } else {
+ // This server is no longer active (e.g., RS moved or removed); skip
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Skipping replication marker timestamp for invalid server: {}", server);
Review Comment:
Is the`LOG.isDebugEnabled()` check necessary here? Shouldn't log4j just not log this message if the log level is set to INFO or higher?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2792183358
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 38s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 10m 52s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 19s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 11m 11s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 1m 12s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 45s | | the patch passed |
| +1 :green_heart: | compile | 12m 4s | | the patch passed |
| -0 :warning: | javac | 12m 4s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1273 unchanged - 0 fixed = 1274 total (was 1273) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 2m 6s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | spotbugs | 11m 1s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 18m 7s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 10s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not generate ASF License warnings. |
| | | 90m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux 21eda626d1f6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / f0c0a7f41b623487cdfe3b12ebf351e2f2585ddf |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 188 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/9/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2115679885
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/RestoreDriver.java:
##########
@@ -17,117 +17,34 @@
*/
package org.apache.hadoop.hbase.backup;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
-import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
-
-import java.io.IOException;
import java.net.URI;
-import java.util.List;
-import java.util.Objects;
-import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
-import org.apache.hadoop.hbase.backup.impl.BackupManager;
-import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
import org.apache.hadoop.hbase.backup.util.BackupUtils;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
-import org.apache.hadoop.hbase.logging.Log4jUtils;
-import org.apache.hadoop.hbase.util.AbstractHBaseTool;
import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.hadoop.util.ToolRunner;
import org.apache.yetus.audience.InterfaceAudience;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
-import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
/**
* Command-line entry point for restore operation
*/
@InterfaceAudience.Private
-public class RestoreDriver extends AbstractHBaseTool {
- private static final Logger LOG = LoggerFactory.getLogger(RestoreDriver.class);
- private CommandLine cmd;
-
- private static final String USAGE_STRING =
- "Usage: hbase restore <backup_path> <backup_id> [options]\n"
- + " backup_path Path to a backup destination root\n"
- + " backup_id Backup image ID to restore\n"
- + " table(s) Comma-separated list of tables to restore\n";
-
- private static final String USAGE_FOOTER = "";
-
- protected RestoreDriver() throws IOException {
- init();
- }
-
- protected void init() {
- // disable irrelevant loggers to avoid it mess up command output
- Log4jUtils.disableZkAndClientLoggers();
- }
-
- private int parseAndRun() throws IOException {
- // Check if backup is enabled
- if (!BackupManager.isBackupEnabled(getConf())) {
- System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
- return -1;
- }
-
- // enable debug logging
- if (cmd.hasOption(OPTION_DEBUG)) {
- Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
- }
-
- // whether to overwrite to existing table if any, false by default
- boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
- if (overwrite) {
- LOG.debug("Found -overwrite option in restore command, "
- + "will overwrite to existing table if any in the restore target");
- }
-
- // whether to only check the dependencies, false by default
- boolean check = cmd.hasOption(OPTION_CHECK);
- if (check) {
- LOG.debug(
- "Found -check option in restore command, " + "will check and verify the dependencies");
- }
-
- if (cmd.hasOption(OPTION_SET) && cmd.hasOption(OPTION_TABLE)) {
- System.err.println(
- "Options -s and -t are mutaully exclusive," + " you can not specify both of them.");
- printToolUsage();
- return -1;
- }
-
- if (!cmd.hasOption(OPTION_SET) && !cmd.hasOption(OPTION_TABLE)) {
- System.err.println("You have to specify either set name or table list to restore");
- printToolUsage();
- return -1;
- }
-
- if (cmd.hasOption(OPTION_YARN_QUEUE_NAME)) {
- String queueName = cmd.getOptionValue(OPTION_YARN_QUEUE_NAME);
- // Set MR job queuename to configuration
- getConf().set("mapreduce.job.queuename", queueName);
- }
+public class RestoreDriver extends AbstractRestoreDriver {
+ private static final String USAGE_STRING = """
+ Usage: hbase restore <backup_path> <backup_id> [options]
Review Comment:
it is already included in the options list. 3 line below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2116260761
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
Review Comment:
I missed that you're using this at the end of the function as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2919423263
> > > > Overall lgtm, but surprising how low the level of unit tests' granularity is: for instance `BackupAdminImpl` is such a big class and there's no corresponding test class. It has a lot of private methods which could be tested as "units" if they were package private and we had a `BackupAdminTest` class. Please consider that.
> > >
> > >
> > > True, there isn’t a separate test class specifically for `BackupAdminImpl`. However, if you look at the overall structure, the core implementation resides in `BackupAdminImpl`, while the other classes primarily handle input preparation (including validation) and delegate the calls to `BackupAdminImpl`. We have a lot of tests for those classes, which indirectly cover almost all parts of `BackupAdminImpl`.
> > > That said, I’ll still take a closer look at the class and add tests if I find any gaps.
> >
> >
> > Could you please provide examples of that? Are they really just proxy calls?
>
> Yes, for example, mergeBackups() is already covered by tests in TestBackupMerge, TestIncrementalBackupMergeWithBulkLoad, and IntegrationTestBackupRestore. The checkIfValidForMerge() method is private and is only called from mergeBackups(), which is the case for most of the methods in this class.
>
> That said, we can definitely create a dedicated test class for this and add the remaining/uncovered test cases. I’ll open a separate Jira for that. Does that sound good?
Yeah, these examples are all integration tests as the same suggests in some cases: they create "real"/mini HBase cluster, sets up a connection, get reference to Admin and call methods which are publicly accessible. Looks like this is historically the main approach for HBase testing, but this is not unit testing.
What I'm talking about is instantiating BackupAdminImpl class directly with a mocked connection and test the methods individually. In order to do this you have to make methods accessible from unit tests, for instance, by changing visibility to package private. I don't insist doing this strictly, just leaving here as something to think about. Let me approve this patch regardless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2115697664
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
Review Comment:
But I am using the value of `overwrite` in another place (line 152).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2755113867
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 59s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 28s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 2m 36s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 6m 30s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | | the patch passed |
| +1 :green_heart: | compile | 2m 54s | | the patch passed |
| +1 :green_heart: | javac | 2m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 37s | | the patch passed |
| +1 :green_heart: | shadedjars | 6m 36s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 324m 51s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 364m 15s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 0321603f87fe 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / dc20fbe957936fc024e211c8c4cab85edace5097 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/testReport/ |
| Max. process+thread count | 5063 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/6/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2917738689
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 4m 7s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 10m 48s | | HBASE-28957 passed |
| +1 :green_heart: | checkstyle | 1m 29s | | HBASE-28957 passed |
| +1 :green_heart: | spotbugs | 10m 12s | | HBASE-28957 passed |
| +1 :green_heart: | spotless | 0m 59s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | | the patch passed |
| +1 :green_heart: | compile | 10m 36s | | the patch passed |
| -0 :warning: | javac | 10m 36s | [/results-compile-javac-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/artifact/yetus-general-check/output/results-compile-javac-root.txt) | root generated 1 new + 1273 unchanged - 0 fixed = 1274 total (was 1273) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 29s | [/buildtool-patch-checkstyle-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/artifact/yetus-general-check/output/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | spotbugs | 10m 53s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 14m 49s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 1m 0s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not generate ASF License warnings. |
| | | 81m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti |
| uname | Linux 099ca9c0447e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 8063bbe39ea421ae2df1980f20d235270c76f366 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 189 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/10/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2110477468
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java:
##########
@@ -671,6 +682,14 @@ public static RestoreRequest createRestoreRequest(String backupRootDir, String b
return request;
}
+ public static PointInTimeRestoreRequest createPointInTimeRestoreRequest(String backupRootDir,
Review Comment:
nit: Since you have Builder implemented for PointInTimeRestoreRequest, this method is redundant. Creating a new request is already a one-liner.
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java:
##########
@@ -770,4 +789,156 @@ public static String findMostRecentBackupId(String[] backupIds) {
return BackupRestoreConstants.BACKUPID_PREFIX + recentTimestamp;
}
+ /**
+ * Calculates the replication checkpoint timestamp used for continuous backup.
+ * <p>
+ * A replication checkpoint is the earliest timestamp across all region servers such that every
+ * WAL entry before that point is known to be replicated to the target system. This is essential
+ * for features like Point-in-Time Restore (PITR) and incremental backups, where we want to
+ * confidently restore data to a consistent state without missing updates.
+ * <p>
+ * The checkpoint is calculated using a combination of:
+ * <ul>
+ * <li>The start timestamps of WAL files currently being replicated for each server.</li>
+ * <li>The latest successfully replicated timestamp recorded by the replication marker chore.</li>
+ * </ul>
+ * <p>
+ * We combine these two sources to handle the following challenges:
+ * <ul>
+ * <li><b>Stale WAL start times:</b> If replication traffic is low or WALs are long-lived, the
+ * replication offset may point to the same WAL for a long time, resulting in stale timestamps
+ * that underestimate progress. This could delay PITR unnecessarily.</li>
+ * <li><b>Limitations of marker-only tracking:</b> The replication marker chore stores the last
+ * successfully replicated timestamp per region server in a system table. However, this data may
+ * become stale if the server goes offline or region ownership changes. For example, if a region
+ * initially belonged to rs1 and was later moved to rs4 due to re-balancing, rs1’s marker would
+ * persist even though it no longer holds any regions. Relying solely on these stale markers could
+ * lead to incorrect or outdated checkpoints.</li>
+ * </ul>
+ * <p>
+ * To handle these limitations, the method:
+ * <ol>
+ * <li>Verifies that the continuous backup peer exists to ensure replication is enabled.</li>
+ * <li>Retrieves WAL replication queue information for the peer, collecting WAL start times per
+ * region server. This gives us a lower bound for replication progress.</li>
+ * <li>Reads the marker chore's replicated timestamps from the backup system table.</li>
+ * <li>For servers found in both sources, if the marker timestamp is more recent than the WAL's
+ * start timestamp, we use the marker (since replication has progressed beyond the WAL).</li>
+ * <li>We discard marker entries for region servers that are not present in WAL queues, assuming
+ * those servers are no longer relevant (e.g., decommissioned or reassigned).</li>
+ * <li>The checkpoint is the minimum of all chosen timestamps — i.e., the slowest replicating
+ * region server.</li>
+ * <li>Finally, we persist the updated marker information to include any newly participating
+ * region servers.</li>
+ * </ol>
+ * <p>
+ * Note: If the replication marker chore is disabled, we fall back to using only the WAL start
+ * times. This ensures correctness but may lead to conservative checkpoint estimates during idle
+ * periods.
+ * @param conn the HBase connection
+ * @return the calculated replication checkpoint timestamp
+ * @throws IOException if reading replication queues or updating the backup system table fails
+ */
+ public static long getReplicationCheckpoint(Connection conn) throws IOException {
+ Configuration conf = conn.getConfiguration();
+ long checkpoint = EnvironmentEdgeManager.getDelegate().currentTime();
+
+ // Step 1: Ensure the continuous backup replication peer exists
+ if (!continuousBackupReplicationPeerExists(conn.getAdmin())) {
+ String msg = "Replication peer '" + CONTINUOUS_BACKUP_REPLICATION_PEER
+ + "' not found. Continuous backup not enabled.";
+ LOG.error(msg);
+ throw new IOException(msg);
+ }
+
+ // Step 2: Get all replication queues for the continuous backup peer
+ ReplicationQueueStorage queueStorage =
+ ReplicationStorageFactory.getReplicationQueueStorage(conn, conf);
+
+ List<ReplicationQueueId> queueIds;
+ try {
+ queueIds = queueStorage.listAllQueueIds(CONTINUOUS_BACKUP_REPLICATION_PEER);
+ } catch (ReplicationException e) {
+ String msg = "Failed to retrieve replication queue IDs for peer '"
+ + CONTINUOUS_BACKUP_REPLICATION_PEER + "'";
+ LOG.error(msg, e);
+ throw new IOException(msg, e);
+ }
+
+ if (queueIds.isEmpty()) {
+ String msg = "Replication peer '" + CONTINUOUS_BACKUP_REPLICATION_PEER + "' has no queues. "
+ + "This may indicate that continuous backup replication is not initialized correctly.";
+ LOG.error(msg);
+ throw new IOException(msg);
+ }
+
+ // Step 3: Build a map of ServerName -> WAL start timestamp (lowest seen per server)
+ Map<ServerName, Long> serverToCheckpoint = new HashMap<>();
+ for (ReplicationQueueId queueId : queueIds) {
+ Map<String, ReplicationGroupOffset> offsets;
+ try {
+ offsets = queueStorage.getOffsets(queueId);
+ } catch (ReplicationException e) {
+ String msg = "Failed to fetch WAL offsets for replication queue: " + queueId;
+ LOG.error(msg, e);
+ throw new IOException(msg, e);
+ }
+
+ for (ReplicationGroupOffset offset : offsets.values()) {
+ String walFile = offset.getWal();
+ long ts = AbstractFSWALProvider.getTimestamp(walFile); // WAL creation time
+ ServerName server = queueId.getServerName();
+ // Store the minimum timestamp per server (ts - 1 to avoid edge boundary issues)
+ serverToCheckpoint.merge(server, ts - 1, Math::min);
+ }
+ }
+
+ // Step 4: If replication markers are enabled, overlay fresher timestamps from backup system
+ // table
+ boolean replicationMarkerEnabled =
+ conf.getBoolean(REPLICATION_MARKER_ENABLED_KEY, REPLICATION_MARKER_ENABLED_DEFAULT);
+ if (replicationMarkerEnabled) {
+ try (BackupSystemTable backupSystemTable = new BackupSystemTable(conn)) {
+ Map<ServerName, Long> markerTimestamps = backupSystemTable.getBackupCheckpointTimestamps();
+
+ for (Map.Entry<ServerName, Long> entry : markerTimestamps.entrySet()) {
+ ServerName server = entry.getKey();
+ long markerTs = entry.getValue();
+
+ // If marker timestamp is newer, override
+ if (serverToCheckpoint.containsKey(server)) {
+ long current = serverToCheckpoint.get(server);
+ if (markerTs > current) {
+ serverToCheckpoint.put(server, markerTs);
+ }
+ } else {
+ // This server is no longer active (e.g., RS moved or removed); skip
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Skipping replication marker timestamp for invalid server: {}", server);
Review Comment:
"inactive" maybe instead of "invalid" - could be slightly less misleading, these servers are probably not invalid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2917299012
> > > Overall lgtm, but surprising how low the level of unit tests' granularity is: for instance `BackupAdminImpl` is such a big class and there's no corresponding test class. It has a lot of private methods which could be tested as "units" if they were package private and we had a `BackupAdminTest` class. Please consider that.
> >
> >
> > True, there isn’t a separate test class specifically for `BackupAdminImpl`. However, if you look at the overall structure, the core implementation resides in `BackupAdminImpl`, while the other classes primarily handle input preparation (including validation) and delegate the calls to `BackupAdminImpl`. We have a lot of tests for those classes, which indirectly cover almost all parts of `BackupAdminImpl`.
> > That said, I’ll still take a closer look at the class and add tests if I find any gaps.
>
> Could you please provide examples of that? Are they really just proxy calls?
Yes, for example, mergeBackups() is already covered by tests in TestBackupMerge, TestIncrementalBackupMergeWithBulkLoad, and IntegrationTestBackupRestore. The checkIfValidForMerge() method is private and is only called from mergeBackups(), which is the case for most of the methods in this class.
That said, we can definitely create a dedicated test class for this and add the remaining/uncovered test cases. I’ll open a separate Jira for that. Does that sound good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2115689818
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/PointInTimeRestoreDriver.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.CONF_CONTINUOUS_BACKUP_WAL_DIR;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME_DESC;
+
+import java.net.URI;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Command-line entry point for restore operation
+ */
+@InterfaceAudience.Private
+public class PointInTimeRestoreDriver extends AbstractRestoreDriver {
+ private static final String USAGE_STRING = """
+ Usage: hbase pitr [options]
+ <backup_path> Backup Path to use for Point in Time Restore
+ table(s) Comma-separated list of tables to restore
+ """;
+
+ @Override
+ protected int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite) {
+ String walBackupDir = getConf().get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ if (walBackupDir == null || walBackupDir.isEmpty()) {
+ System.err.printf(
+ "Point-in-Time Restore requires the WAL backup directory (%s) to replay logs after full and incremental backups. "
+ + "Set this property if you need Point-in-Time Restore. Otherwise, use the normal restore process with the appropriate backup ID.%n",
+ CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ return -1;
+ }
+
+ String[] remainArgs = cmd.getArgs();
+ if (remainArgs.length != 0) {
+ printToolUsage();
+ return -1;
+ }
+
+ String backupRootDir =
+ cmd.hasOption(OPTION_PITR_BACKUP_PATH) ? cmd.getOptionValue(OPTION_PITR_BACKUP_PATH) : null;
Review Comment:
But I will modify it. user will know that backupRootDir could be null since we are using cmd.getOptionValue(), which could return null;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2115684437
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/PointInTimeRestoreDriver.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.CONF_CONTINUOUS_BACKUP_WAL_DIR;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.LONG_OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_PITR_BACKUP_PATH_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TO_DATETIME_DESC;
+
+import java.net.URI;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Command-line entry point for restore operation
+ */
+@InterfaceAudience.Private
+public class PointInTimeRestoreDriver extends AbstractRestoreDriver {
+ private static final String USAGE_STRING = """
+ Usage: hbase pitr [options]
+ <backup_path> Backup Path to use for Point in Time Restore
+ table(s) Comma-separated list of tables to restore
+ """;
+
+ @Override
+ protected int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite) {
+ String walBackupDir = getConf().get(CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ if (walBackupDir == null || walBackupDir.isEmpty()) {
+ System.err.printf(
+ "Point-in-Time Restore requires the WAL backup directory (%s) to replay logs after full and incremental backups. "
+ + "Set this property if you need Point-in-Time Restore. Otherwise, use the normal restore process with the appropriate backup ID.%n",
+ CONF_CONTINUOUS_BACKUP_WAL_DIR);
+ return -1;
+ }
+
+ String[] remainArgs = cmd.getArgs();
+ if (remainArgs.length != 0) {
+ printToolUsage();
+ return -1;
+ }
+
+ String backupRootDir =
+ cmd.hasOption(OPTION_PITR_BACKUP_PATH) ? cmd.getOptionValue(OPTION_PITR_BACKUP_PATH) : null;
Review Comment:
True. but can we keep for readability purpose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2114795970
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
+ LOG.debug(
+ "Found -check option in restore command, " + "will check and verify the dependencies");
+ }
+
+ if (cmd.hasOption(OPTION_SET) && cmd.hasOption(OPTION_TABLE)) {
+ System.err.println(
+ "Options -s and -t are mutually exclusive," + " you can not specify both of them.");
+ printToolUsage();
+ return -1;
+ }
+
+ if (!cmd.hasOption(OPTION_SET) && !cmd.hasOption(OPTION_TABLE)) {
+ System.err.println("You have to specify either set name or table list to restore");
Review Comment:
Similar to the previous comment
```suggestion
System.err.printf("You have to specify either set name (-%s) or table list (-%s) to "
+ "restore%n", OPTION_SET, OPTION_TABLE);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "kgeisz (via GitHub)" <gi...@apache.org>.
kgeisz commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2116260761
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/AbstractRestoreDriver.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_CHECK_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_DEBUG_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_OVERWRITE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_SET_RESTORE_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_LIST_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_TABLE_MAPPING_DESC;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME;
+import static org.apache.hadoop.hbase.backup.BackupRestoreConstants.OPTION_YARN_QUEUE_NAME_RESTORE_DESC;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.backup.impl.BackupSystemTable;
+import org.apache.hadoop.hbase.backup.util.BackupUtils;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.logging.Log4jUtils;
+import org.apache.hadoop.hbase.util.AbstractHBaseTool;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
+import org.apache.hbase.thirdparty.org.apache.commons.cli.HelpFormatter;
+
+@InterfaceAudience.Private
+public abstract class AbstractRestoreDriver extends AbstractHBaseTool {
+ protected static final Logger LOG = LoggerFactory.getLogger(AbstractRestoreDriver.class);
+ protected CommandLine cmd;
+
+ protected static final String USAGE_FOOTER = "";
+
+ protected AbstractRestoreDriver() {
+ init();
+ }
+
+ protected void init() {
+ Log4jUtils.disableZkAndClientLoggers();
+ }
+
+ protected abstract int executeRestore(boolean check, TableName[] fromTables, TableName[] toTables,
+ boolean isOverwrite);
+
+ private int parseAndRun() throws IOException {
+ if (!BackupManager.isBackupEnabled(getConf())) {
+ System.err.println(BackupRestoreConstants.ENABLE_BACKUP);
+ return -1;
+ }
+
+ if (cmd.hasOption(OPTION_DEBUG)) {
+ Log4jUtils.setLogLevel("org.apache.hadoop.hbase.backup", "DEBUG");
+ }
+
+ boolean overwrite = cmd.hasOption(OPTION_OVERWRITE);
+ if (overwrite) {
+ LOG.debug("Found -overwrite option in restore command, "
+ + "will overwrite to existing table if any in the restore target");
+ }
+
+ boolean check = cmd.hasOption(OPTION_CHECK);
+ if (check) {
Review Comment:
I missed that you're using this at the end of the function as well, so no change is needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on code in PR #6717:
URL: https://github.com/apache/hbase/pull/6717#discussion_r2074738766
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java:
##########
@@ -770,4 +789,156 @@ public static String findMostRecentBackupId(String[] backupIds) {
return BackupRestoreConstants.BACKUPID_PREFIX + recentTimestamp;
}
+ /**
+ * Calculates the replication checkpoint timestamp used for continuous backup.
+ * <p>
+ * A replication checkpoint is the earliest timestamp across all region servers such that every
+ * WAL entry before that point is known to be replicated to the target system. This is essential
+ * for features like Point-in-Time Restore (PITR) and incremental backups, where we want to
+ * confidently restore data to a consistent state without missing updates.
+ * <p>
+ * The checkpoint is calculated using a combination of:
+ * <ul>
+ * <li>The start timestamps of WAL files currently being replicated for each server.</li>
+ * <li>The latest successfully replicated timestamp recorded by the replication marker chore.</li>
+ * </ul>
+ * <p>
+ * We combine these two sources to handle the following challenges:
+ * <ul>
+ * <li><b>Stale WAL start times:</b> If replication traffic is low or WALs are long-lived, the
+ * replication offset may point to the same WAL for a long time, resulting in stale timestamps
+ * that underestimate progress. This could delay PITR unnecessarily.</li>
+ * <li><b>Limitations of marker-only tracking:</b> The replication marker chore stores the last
+ * successfully replicated timestamp per region server in a system table. However, this data may
+ * become stale if the server goes offline or region ownership changes. For example, if a region
+ * initially belonged to rs1 and was later moved to rs4 due to re-balancing, rs1’s marker would
+ * persist even though it no longer holds any regions. Relying solely on these stale markers could
+ * lead to incorrect or outdated checkpoints.</li>
+ * </ul>
+ * <p>
+ * To handle these limitations, the method:
+ * <ol>
+ * <li>Verifies that the continuous backup peer exists to ensure replication is enabled.</li>
+ * <li>Retrieves WAL replication queue information for the peer, collecting WAL start times per
+ * region server. This gives us a lower bound for replication progress.</li>
+ * <li>Reads the marker chore's replicated timestamps from the backup system table.</li>
+ * <li>For servers found in both sources, if the marker timestamp is more recent than the WAL's
+ * start timestamp, we use the marker (since replication has progressed beyond the WAL).</li>
+ * <li>We discard marker entries for region servers that are not present in WAL queues, assuming
+ * those servers are no longer relevant (e.g., decommissioned or reassigned).</li>
+ * <li>The checkpoint is the minimum of all chosen timestamps — i.e., the slowest replicating
+ * region server.</li>
+ * <li>Finally, we persist the updated marker information to include any newly participating
+ * region servers.</li>
+ * </ol>
+ * <p>
+ * Note: If the replication marker chore is disabled, we fall back to using only the WAL start
+ * times. This ensures correctness but may lead to conservative checkpoint estimates during idle
+ * periods.
+ * @param conn the HBase connection
+ * @return the calculated replication checkpoint timestamp
+ * @throws IOException if reading replication queues or updating the backup system table fails
+ */
+ public static long getReplicationCheckpoint(Connection conn) throws IOException {
+ Configuration conf = conn.getConfiguration();
+ long checkpoint = EnvironmentEdgeManager.getDelegate().currentTime();
+
+ // Step 1: Ensure the continuous backup replication peer exists
+ if (!continuousBackupReplicationPeerExists(conn.getAdmin())) {
+ String msg = "Replication peer '" + CONTINUOUS_BACKUP_REPLICATION_PEER
+ + "' not found. Continuous backup not enabled.";
+ LOG.error(msg);
+ throw new IOException(msg);
+ }
+
+ // Step 2: Get all replication queues for the continuous backup peer
+ ReplicationQueueStorage queueStorage =
+ ReplicationStorageFactory.getReplicationQueueStorage(conn, conf);
+
+ List<ReplicationQueueId> queueIds;
+ try {
+ queueIds = queueStorage.listAllQueueIds(CONTINUOUS_BACKUP_REPLICATION_PEER);
+ } catch (ReplicationException e) {
+ String msg = "Failed to retrieve replication queue IDs for peer '"
+ + CONTINUOUS_BACKUP_REPLICATION_PEER + "'";
+ LOG.error(msg, e);
+ throw new IOException(msg, e);
+ }
+
+ if (queueIds.isEmpty()) {
+ String msg = "Replication peer '" + CONTINUOUS_BACKUP_REPLICATION_PEER + "' has no queues. "
+ + "This may indicate that continuous backup replication is not initialized correctly.";
+ LOG.error(msg);
+ throw new IOException(msg);
+ }
+
+ // Step 3: Build a map of ServerName -> WAL start timestamp (lowest seen per server)
+ Map<ServerName, Long> serverToCheckpoint = new HashMap<>();
+ for (ReplicationQueueId queueId : queueIds) {
+ Map<String, ReplicationGroupOffset> offsets;
+ try {
+ offsets = queueStorage.getOffsets(queueId);
+ } catch (ReplicationException e) {
+ String msg = "Failed to fetch WAL offsets for replication queue: " + queueId;
+ LOG.error(msg, e);
+ throw new IOException(msg, e);
+ }
+
+ for (ReplicationGroupOffset offset : offsets.values()) {
+ String walFile = offset.getWal();
+ long ts = AbstractFSWALProvider.getTimestamp(walFile); // WAL creation time
+ ServerName server = queueId.getServerName();
+ // Store the minimum timestamp per server (ts - 1 to avoid edge boundary issues)
+ serverToCheckpoint.merge(server, ts - 1, Math::min);
+ }
+ }
+
+ // Step 4: If replication markers are enabled, overlay fresher timestamps from backup system
+ // table
+ boolean replicationMarkerEnabled =
+ conf.getBoolean(REPLICATION_MARKER_ENABLED_KEY, REPLICATION_MARKER_ENABLED_DEFAULT);
+ if (replicationMarkerEnabled) {
+ try (BackupSystemTable backupSystemTable = new BackupSystemTable(conn)) {
+ Map<ServerName, Long> markerTimestamps = backupSystemTable.getBackupCheckpointTimestamps();
+
+ for (Map.Entry<ServerName, Long> entry : markerTimestamps.entrySet()) {
+ ServerName server = entry.getKey();
+ long markerTs = entry.getValue();
+
+ // If marker timestamp is newer, override
+ if (serverToCheckpoint.containsKey(server)) {
+ long current = serverToCheckpoint.get(server);
+ if (markerTs > current) {
+ serverToCheckpoint.put(server, markerTs);
+ }
+ } else {
+ // This server is no longer active (e.g., RS moved or removed); skip
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Skipping replication marker timestamp for invalid server: {}", server);
Review Comment:
Correct, it's unnecessary here. We typically use the conditional check when building a complex string or performing expensive operations, which isn't the case in this line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "anmolnar (via GitHub)" <gi...@apache.org>.
anmolnar commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2916411766
> > Overall lgtm, but surprising how low the level of unit tests' granularity is: for instance `BackupAdminImpl` is such a big class and there's no corresponding test class. It has a lot of private methods which could be tested as "units" if they were package private and we had a `BackupAdminTest` class. Please consider that.
>
> True, there isn’t a separate test class specifically for `BackupAdminImpl`. However, if you look at the overall structure, the core implementation resides in `BackupAdminImpl`, while the other classes primarily handle input preparation (including validation) and delegate the calls to `BackupAdminImpl`. We have a lot of tests for those classes, which indirectly cover almost all parts of `BackupAdminImpl`.
>
> That said, I’ll still take a closer look at the class and add tests if I find any gaps.
Could you please provide examples of that?
Are they really just proxy calls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "vinayakphegde (via GitHub)" <gi...@apache.org>.
vinayakphegde commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2915955005
> Overall lgtm, but surprising how low the level of unit tests' granularity is: for instance `BackupAdminImpl` is such a big class and there's no corresponding test class. It has a lot of private methods which could be tested as "units" if they were package private and we had a `BackupAdminTest` class. Please consider that.
True, there isn’t a separate test class specifically for `BackupAdminImpl`. However, if you look at the overall structure, the core implementation resides in `BackupAdminImpl`, while the other classes primarily handle input preparation (including validation) and delegate the calls to `BackupAdminImpl`. We have a lot of tests for those classes, which indirectly cover almost all parts of `BackupAdminImpl`.
That said, I’ll still take a closer look at the class and add tests if I find any gaps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29133: Implement "pitr" Command for Point-in-Time Restore [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6717:
URL: https://github.com/apache/hbase/pull/6717#issuecomment-2791215313
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ HBASE-28957 Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 43s | | HBASE-28957 passed |
| +1 :green_heart: | compile | 2m 40s | | HBASE-28957 passed |
| +1 :green_heart: | javadoc | 3m 12s | | HBASE-28957 passed |
| +1 :green_heart: | shadedjars | 7m 17s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 41s | | the patch passed |
| +1 :green_heart: | compile | 2m 25s | | the patch passed |
| +1 :green_heart: | javac | 2m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 48s | | the patch passed |
| +1 :green_heart: | shadedjars | 7m 4s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 341m 10s | [/patch-unit-root.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-root.txt) | root in the patch failed. |
| | | 381m 51s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6717 |
| JIRA Issue | HBASE-29133 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 226dc3cbdea3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | HBASE-28957 / 4fa1c3b31e3266c2f177480a92af433d2ddc50cd |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/testReport/ |
| Max. process+thread count | 5128 (vs. ulimit of 30000) |
| modules | C: hbase-backup . U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6717/8/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org