You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mnpoonia (via GitHub)" <gi...@apache.org> on 2024/11/20 09:59:11 UTC
[PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
mnpoonia opened a new pull request, #6483:
URL: https://github.com/apache/hbase/pull/6483
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "mnpoonia (via GitHub)" <gi...@apache.org>.
mnpoonia commented on code in PR #6483:
URL: https://github.com/apache/hbase/pull/6483#discussion_r1855835848
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java:
##########
@@ -437,33 +440,54 @@ private static List<File> resolveAndArchive(FileSystem fs, Path baseArchiveDir,
LOG.trace("Created archive directory {}", baseArchiveDir);
}
- List<File> failures = new ArrayList<>();
+ List<File> failures = Collections.synchronizedList(new ArrayList<>());
String startTime = Long.toString(start);
+ List<File> filesOnly = new ArrayList<>();
for (File file : toArchive) {
- // if its a file archive it
try {
- LOG.trace("Archiving {}", file);
- if (file.isFile()) {
- // attempt to archive the file
- if (!resolveAndArchiveFile(baseArchiveDir, file, startTime)) {
- LOG.warn("Couldn't archive " + file + " into backup directory: " + baseArchiveDir);
- failures.add(file);
- }
- } else {
- // otherwise its a directory and we need to archive all files
+ if (!file.isFile()) {
+ // if its a directory and we need to archive all files
LOG.trace("{} is a directory, archiving children files", file);
// so we add the directory name to the one base archive
Path parentArchiveDir = new Path(baseArchiveDir, file.getName());
// and then get all the files from that directory and attempt to
// archive those too
Collection<File> children = file.getChildren();
- failures.addAll(resolveAndArchive(fs, parentArchiveDir, children, start));
+ failures.addAll(resolveAndArchive(conf, fs, parentArchiveDir, children, start));
+ } else {
+ filesOnly.add(file);
}
} catch (IOException e) {
LOG.warn("Failed to archive {}", file, e);
failures.add(file);
}
}
+ Map<File, Future<Boolean>> futures = new HashMap<>();
+ // In current baseDir all files will be processed concurrently
+ for (File file : filesOnly) {
+ LOG.trace("Archiving {}", file);
+ Future<Boolean> archiveTask = getArchiveExecutor(conf)
+ .submit(() -> resolveAndArchiveFile(baseArchiveDir, file, startTime));
+ futures.put(file, archiveTask);
+ }
+
+ for (Map.Entry<File, Future<Boolean>> fileFutureEntry : futures.entrySet()) {
+ try {
+ boolean fileCleaned = fileFutureEntry.getValue().get();
+ if (!fileCleaned) {
+ LOG.warn("Couldn't archive %s into backup directory: %s"
+ .formatted(fileFutureEntry.getKey(), baseArchiveDir));
Review Comment:
Thanks for pointing to this. Yes i was using jdk 17 so missed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "d-c-manning (via GitHub)" <gi...@apache.org>.
d-c-manning commented on code in PR #6483:
URL: https://github.com/apache/hbase/pull/6483#discussion_r1854777979
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java:
##########
@@ -437,33 +440,54 @@ private static List<File> resolveAndArchive(FileSystem fs, Path baseArchiveDir,
LOG.trace("Created archive directory {}", baseArchiveDir);
}
- List<File> failures = new ArrayList<>();
+ List<File> failures = Collections.synchronizedList(new ArrayList<>());
String startTime = Long.toString(start);
+ List<File> filesOnly = new ArrayList<>();
for (File file : toArchive) {
- // if its a file archive it
try {
- LOG.trace("Archiving {}", file);
- if (file.isFile()) {
- // attempt to archive the file
- if (!resolveAndArchiveFile(baseArchiveDir, file, startTime)) {
- LOG.warn("Couldn't archive " + file + " into backup directory: " + baseArchiveDir);
- failures.add(file);
- }
- } else {
- // otherwise its a directory and we need to archive all files
+ if (!file.isFile()) {
+ // if its a directory and we need to archive all files
LOG.trace("{} is a directory, archiving children files", file);
// so we add the directory name to the one base archive
Path parentArchiveDir = new Path(baseArchiveDir, file.getName());
// and then get all the files from that directory and attempt to
// archive those too
Collection<File> children = file.getChildren();
- failures.addAll(resolveAndArchive(fs, parentArchiveDir, children, start));
+ failures.addAll(resolveAndArchive(conf, fs, parentArchiveDir, children, start));
+ } else {
+ filesOnly.add(file);
}
} catch (IOException e) {
LOG.warn("Failed to archive {}", file, e);
failures.add(file);
}
}
+ Map<File, Future<Boolean>> futures = new HashMap<>();
+ // In current baseDir all files will be processed concurrently
+ for (File file : filesOnly) {
+ LOG.trace("Archiving {}", file);
+ Future<Boolean> archiveTask = getArchiveExecutor(conf)
+ .submit(() -> resolveAndArchiveFile(baseArchiveDir, file, startTime));
+ futures.put(file, archiveTask);
+ }
+
+ for (Map.Entry<File, Future<Boolean>> fileFutureEntry : futures.entrySet()) {
+ try {
+ boolean fileCleaned = fileFutureEntry.getValue().get();
+ if (!fileCleaned) {
+ LOG.warn("Couldn't archive %s into backup directory: %s"
+ .formatted(fileFutureEntry.getKey(), baseArchiveDir));
Review Comment:
I don't know all the jdk requirements, but looks like `.formatted` was added only in JDK 15. If this is intended for backport, there may be earlier JDK versions to support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2513648896
@Apache9 @apurtell this looks good, any reviews from your side before we merge it?
Keeping this open for at least 24 hr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2488242795
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 3s | | master passed |
| +1 :green_heart: | compile | 0m 52s | | master passed |
| +1 :green_heart: | javadoc | 0m 28s | | master passed |
| +1 :green_heart: | shadedjars | 5m 40s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 37s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 16s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 43m 17s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6483 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a252be1ba51b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d3703bbeda2f139aa1894508d6af12de60bc046d |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/1/testReport/ |
| Max. process+thread count | 2203 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "mnpoonia (via GitHub)" <gi...@apache.org>.
mnpoonia commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2488130275
@d-c-manning @virajjasani @Umeshkumar9414 @Apache9 @apurtell FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2490143936
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 19s | | master passed |
| +1 :green_heart: | compile | 3m 3s | | master passed |
| +1 :green_heart: | checkstyle | 0m 42s | | master passed |
| +1 :green_heart: | spotbugs | 1m 46s | | master passed |
| +1 :green_heart: | spotless | 0m 58s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 46s | | the patch passed |
| +1 :green_heart: | compile | 3m 36s | | the patch passed |
| +1 :green_heart: | javac | 3m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 37s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 10m 13s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 43s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | | The patch does not generate ASF License warnings. |
| | | 38m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6483 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 52f31024ca0a 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f86aa9bac714ba4205d07335bcc89d82d0ec5883 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/2/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "mnpoonia (via GitHub)" <gi...@apache.org>.
mnpoonia commented on code in PR #6483:
URL: https://github.com/apache/hbase/pull/6483#discussion_r1851367892
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java:
##########
@@ -437,33 +440,53 @@ private static List<File> resolveAndArchive(FileSystem fs, Path baseArchiveDir,
LOG.trace("Created archive directory {}", baseArchiveDir);
}
- List<File> failures = new ArrayList<>();
+ List<File> failures = Collections.synchronizedList(new ArrayList<>());
String startTime = Long.toString(start);
+ List<File> filesOnly = new ArrayList<>();
for (File file : toArchive) {
- // if its a file archive it
try {
- LOG.trace("Archiving {}", file);
- if (file.isFile()) {
- // attempt to archive the file
- if (!resolveAndArchiveFile(baseArchiveDir, file, startTime)) {
- LOG.warn("Couldn't archive " + file + " into backup directory: " + baseArchiveDir);
- failures.add(file);
- }
- } else {
- // otherwise its a directory and we need to archive all files
+ if (!file.isFile()) {
+ // if its a directory and we need to archive all files
LOG.trace("{} is a directory, archiving children files", file);
// so we add the directory name to the one base archive
Path parentArchiveDir = new Path(baseArchiveDir, file.getName());
// and then get all the files from that directory and attempt to
// archive those too
Collection<File> children = file.getChildren();
- failures.addAll(resolveAndArchive(fs, parentArchiveDir, children, start));
+ failures.addAll(resolveAndArchive(conf, fs, parentArchiveDir, children, start));
+ } else {
+ filesOnly.add(file);
}
} catch (IOException e) {
LOG.warn("Failed to archive {}", file, e);
failures.add(file);
}
}
+ Map<File, Future<Boolean>> futures = new HashMap<>();
+ // In current baseDir all files will be processed concurrently
+ for (File file : filesOnly) {
+ LOG.trace("Archiving {}", file);
+ Future<Boolean> archiveTask = getArchiveExecutor(conf)
+ .submit(() -> resolveAndArchiveFile(baseArchiveDir, file, startTime));
+ futures.put(file, archiveTask);
+ }
+
+ for (Map.Entry<File, Future<Boolean>> fileFutureEntry : futures.entrySet()) {
+ try {
+ boolean fileCleaned = fileFutureEntry.getValue().get();
+ if (!fileCleaned) {
+ LOG.warn("Couldn't archive %s into backup directory: %s"
+ .formatted(fileFutureEntry.getKey(), baseArchiveDir));
+ failures.add(fileFutureEntry.getKey());
+ }
+ } catch (InterruptedException e) {
+ LOG.warn("HFileArchive Cleanup thread was interrupted");
Review Comment:
Thanks for pointing. Added it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #6483:
URL: https://github.com/apache/hbase/pull/6483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Umeshkumar9414 (via GitHub)" <gi...@apache.org>.
Umeshkumar9414 commented on code in PR #6483:
URL: https://github.com/apache/hbase/pull/6483#discussion_r1850143652
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java:
##########
@@ -437,33 +440,53 @@ private static List<File> resolveAndArchive(FileSystem fs, Path baseArchiveDir,
LOG.trace("Created archive directory {}", baseArchiveDir);
}
- List<File> failures = new ArrayList<>();
+ List<File> failures = Collections.synchronizedList(new ArrayList<>());
String startTime = Long.toString(start);
+ List<File> filesOnly = new ArrayList<>();
for (File file : toArchive) {
- // if its a file archive it
try {
- LOG.trace("Archiving {}", file);
- if (file.isFile()) {
- // attempt to archive the file
- if (!resolveAndArchiveFile(baseArchiveDir, file, startTime)) {
- LOG.warn("Couldn't archive " + file + " into backup directory: " + baseArchiveDir);
- failures.add(file);
- }
- } else {
- // otherwise its a directory and we need to archive all files
+ if (!file.isFile()) {
+ // if its a directory and we need to archive all files
LOG.trace("{} is a directory, archiving children files", file);
// so we add the directory name to the one base archive
Path parentArchiveDir = new Path(baseArchiveDir, file.getName());
// and then get all the files from that directory and attempt to
// archive those too
Collection<File> children = file.getChildren();
- failures.addAll(resolveAndArchive(fs, parentArchiveDir, children, start));
+ failures.addAll(resolveAndArchive(conf, fs, parentArchiveDir, children, start));
+ } else {
+ filesOnly.add(file);
}
} catch (IOException e) {
LOG.warn("Failed to archive {}", file, e);
failures.add(file);
}
}
+ Map<File, Future<Boolean>> futures = new HashMap<>();
+ // In current baseDir all files will be processed concurrently
+ for (File file : filesOnly) {
+ LOG.trace("Archiving {}", file);
+ Future<Boolean> archiveTask = getArchiveExecutor(conf)
+ .submit(() -> resolveAndArchiveFile(baseArchiveDir, file, startTime));
+ futures.put(file, archiveTask);
+ }
+
+ for (Map.Entry<File, Future<Boolean>> fileFutureEntry : futures.entrySet()) {
+ try {
+ boolean fileCleaned = fileFutureEntry.getValue().get();
+ if (!fileCleaned) {
+ LOG.warn("Couldn't archive %s into backup directory: %s"
+ .formatted(fileFutureEntry.getKey(), baseArchiveDir));
+ failures.add(fileFutureEntry.getKey());
+ }
+ } catch (InterruptedException e) {
+ LOG.warn("HFileArchive Cleanup thread was interrupted");
Review Comment:
not adding into failures list ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "mnpoonia (via GitHub)" <gi...@apache.org>.
mnpoonia commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2528558055
@virajjasani Sure. It can be applied to all active branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "mnpoonia (via GitHub)" <gi...@apache.org>.
mnpoonia commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2530265137
Thank you @virajjasani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2496959350
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 26s | | master passed |
| +1 :green_heart: | compile | 0m 53s | | master passed |
| +1 :green_heart: | javadoc | 0m 27s | | master passed |
| +1 :green_heart: | shadedjars | 5m 36s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 50s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 21m 39s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 44m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6483 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 045cbd2cb87c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 925ff4a14c966304533e0934d4a8912b60e2154a |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/testReport/ |
| Max. process+thread count | 2201 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2488221945
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 59s | | master passed |
| +1 :green_heart: | compile | 2m 55s | | master passed |
| +1 :green_heart: | checkstyle | 0m 37s | | master passed |
| +1 :green_heart: | spotbugs | 1m 33s | | master passed |
| +1 :green_heart: | spotless | 0m 45s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 49s | | the patch passed |
| +1 :green_heart: | compile | 3m 0s | | the patch passed |
| +1 :green_heart: | javac | 3m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 36s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 38s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 10m 15s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 35m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6483 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux e6baf66a966f 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / d3703bbeda2f139aa1894508d6af12de60bc046d |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2532604623
Thank you @mnpoonia for this nice improvement!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2496951021
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any anti-patterns. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 18s | | master passed |
| +1 :green_heart: | compile | 3m 10s | | master passed |
| +1 :green_heart: | checkstyle | 0m 37s | | master passed |
| +1 :green_heart: | spotbugs | 1m 35s | | master passed |
| +1 :green_heart: | spotless | 0m 45s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 0s | | the patch passed |
| +1 :green_heart: | compile | 3m 14s | | the patch passed |
| +1 :green_heart: | javac | 3m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 36s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 38s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 11m 37s | | Patch does not cause any errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | spotless | 0m 43s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | | The patch does not generate ASF License warnings. |
| | | 38m 15s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6483 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux c2fb7253f3e4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 925ff4a14c966304533e0934d4a8912b60e2154a |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/3/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2528600252
Sounds good, let me verify on branch-2 and then will merge it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2525914625
@mnpoonia could you also create branch-2 backport PR so that after clean build results, the changes can be ported to all active branches?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-28836 Parallize the file archival to improve the split times [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6483:
URL: https://github.com/apache/hbase/pull/6483#issuecomment-2490152023
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 3s | | master passed |
| +1 :green_heart: | compile | 0m 54s | | master passed |
| +1 :green_heart: | javadoc | 0m 28s | | master passed |
| +1 :green_heart: | shadedjars | 5m 41s | | branch has no errors when building our shaded downstream artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 51s | | patch has no errors when building our shaded downstream artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 22m 50s | [/patch-unit-hbase-server.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt) | hbase-server in the patch failed. |
| | | 45m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6483 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux a97c42504ae2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / f86aa9bac714ba4205d07335bcc89d82d0ec5883 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/2/testReport/ |
| Max. process+thread count | 2198 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6483/2/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org