You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "hgromer (via GitHub)" <gi...@apache.org> on 2025/04/17 19:28:16 UTC
[PR] HBASE-29268: Restoring to original splits will fail for remote file systems not on the RS classpath [hbase]
hgromer opened a new pull request, #6915:
URL: https://github.com/apache/hbase/pull/6915
cc @rmdmattingly @ndimiduk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29268: Restoring to original splits will fail for remote file systems not on the RS classpath [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6915:
URL: https://github.com/apache/hbase/pull/6915#issuecomment-2813926507
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 20s | | master passed |
| +1 :green_heart: | compile | 0m 18s | | master passed |
| +1 :green_heart: | javadoc | 0m 14s | | master passed |
| +1 :green_heart: | shadedjars | 5m 53s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 52s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 10s | | hbase-backup in the patch passed. |
| | | 36m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6915/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6915 |
| JIRA Issue | HBASE-29268 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 7949dfc03dff 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c35d43086cd14af9d379ae34cd963319f505e1dc |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6915/1/testReport/ |
| Max. process+thread count | 3690 (vs. ulimit of 30000) |
| modules | C: hbase-backup U: hbase-backup |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6915/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29268: Restoring to original splits will fail for remote file systems not on the RS classpath [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #6915:
URL: https://github.com/apache/hbase/pull/6915#discussion_r2049530542
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/InputFileListingGenerator.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup.mapreduce;
Review Comment:
this is all c/p from MapReduceBackupCopyJob so we can share the logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29268: Restoring to original splits will fail for remote file systems not on the RS classpath [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on code in PR #6915:
URL: https://github.com/apache/hbase/pull/6915#discussion_r2049529944
##########
hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceRestoreToOriginalSplitsJob.java:
##########
@@ -66,41 +73,23 @@ public Configuration getConf() {
return conf;
}
- private static Map<byte[], List<Path>> buildFamily2Files(FileSystem fs, Path[] dirs,
- boolean isFullBackup) throws IOException {
- if (isFullBackup) {
- return buildFullBackupFamily2Files(fs, dirs);
+ private static DistCp getDistCp(Configuration conf, boolean fullBackupRestore) throws Exception {
+ if (fullBackupRestore) {
+ return new DistCp(conf, null);
}
- Map<byte[], List<Path>> family2Files = new HashMap<>();
+ return new IncrementalBackupDistCp(conf);
+ }
- for (Path dir : dirs) {
- byte[] familyName = Bytes.toBytes(dir.getParent().getName());
- if (family2Files.containsKey(familyName)) {
- family2Files.get(familyName).add(dir);
- } else {
- family2Files.put(familyName, Lists.newArrayList(dir));
- }
+ private static class IncrementalBackupDistCp extends DistCp {
+ public IncrementalBackupDistCp(Configuration conf) throws Exception {
+ super(conf, null);
}
- return family2Files;
- }
-
- private static Map<byte[], List<Path>> buildFullBackupFamily2Files(FileSystem fs, Path[] dirs)
- throws IOException {
- Map<byte[], List<Path>> family2Files = new HashMap<>();
- for (Path regionPath : dirs) {
- FSVisitor.visitRegionStoreFiles(fs, regionPath, (region, family, name) -> {
- Path path = new Path(regionPath, new Path(family, name));
- byte[] familyName = Bytes.toBytes(family);
- if (family2Files.containsKey(familyName)) {
- family2Files.get(familyName).add(path);
- } else {
- family2Files.put(familyName, Lists.newArrayList(path));
- }
- });
+ @Override
+ protected Path createInputFileListing(Job job) throws IOException {
Review Comment:
We need to do the same thing we're currently doing [here](https://github.com/apache/hbase/blob/e56e14bcc9f15ceae3f7961a21d4eef94ce1c159/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java#L435C52-L435C57) or else the subsequent bulkload doesn't work correctly, specifically for incremental backups
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29268: Restoring to original splits will fail for remote file systems not on the RS classpath [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6915:
URL: https://github.com/apache/hbase/pull/6915#issuecomment-2813907180
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 21s | | master passed |
| +1 :green_heart: | compile | 0m 30s | | master passed |
| +1 :green_heart: | checkstyle | 0m 10s | | master passed |
| +1 :green_heart: | spotbugs | 0m 31s | | master passed |
| +1 :green_heart: | spotless | 0m 46s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed |
| +1 :green_heart: | javac | 0m 29s | | hbase-backup generated 0 new + 102 unchanged - 2 fixed = 102 total (was 104) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 9s | [/results-checkstyle-hbase-backup.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6915/1/artifact/yetus-general-check/output/results-checkstyle-hbase-backup.txt) | hbase-backup: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 0m 35s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 57s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 46s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 30m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6915/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6915 |
| JIRA Issue | HBASE-29268 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux e336daa4d3eb 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / c35d43086cd14af9d379ae34cd963319f505e1dc |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 81 (vs. ulimit of 30000) |
| modules | C: hbase-backup U: hbase-backup |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6915/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29268: Restoring to original splits will fail for remote file systems not on the RS classpath [hbase]
Posted by "hgromer (via GitHub)" <gi...@apache.org>.
hgromer commented on PR #6915:
URL: https://github.com/apache/hbase/pull/6915#issuecomment-2813841061
This should be covered by existing unit tests, as the underlying behavior shouldn't have changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org